Extensibility: The “100% Lisp” Fallacy

For example, the Lem: An Alternative To Emacs? article from Irreal claims: (emphasis preserved)

One thing I like about it is that it’s 100% Common Lisp. There’s no C core; just Lisp all the way down. That makes it easier to customize or extend any part of the editor.

This argument sounds good: Looking at the repository, Lem is 100% 90% Lisp code; since the editor code and user customization live in the Common Lisp runtime, we should be able to extend any part of the editor on the fly, right?

Or, does it really?

Does it offer composition-function-table so that you can program your font ligatures from the editor’s scripting language?

Does it provide an API to define an arbitrary encoding system and the corresponding charset, beyond what is supported by Unicode or any existing standard?

Does it allow you to “override” its newline character, so that a file is displayed all on a single line?

…

These examples are taken from some of the more obscure features in Emacs, and I can go on and on. I don’t think many of the editors out there could possibly support them, as they are probably the 10% non-pure part of a “100% Lisp” system.

To be honest, I hate these features as they’ve haunted me forever since I started designing an IPC protocol for my Emacs clone. But, dang it, I suffered and suffer from it exactly because of Emacs’s extensibility — not the lack of it.

For example, Steel Bank Common Lisp, a common Common Lisp runtime, is only mostly written in Lisp because it has to provide threading primitives, interface with the OS, or leverage assembly code. And obviously you won’t be able to customize those bits.

Going back to (graphical) editors this is no less true. Usually², as is with any GUI program, you will want to support font fallback, input methods and screen readers, all of which require interacting with platform specific APIs and are thus much less customizable. FFI helps to some degree by keeping your niche “pure”, but it can’t extend your customizability beyond that boundary: webviews don’t expose font ligature internals, and CSS is the only way to control that, albeit quite limited; input methods are trickier, as they interfere with keyboard events and are platform-specific; screen readers are … I don’t know, you should really use a library for it if you want portability.

Anyway, it’s just impossible these days to have a “pure” Lisp program with all those platform specific things to deal with, or with all those convenient library bindings to use. It’s not inherently bad, but it certainly limits how you’re allowed to extend things.

However, even with the non-pure parts, provided with suitable building blocks, it’s usually quite surprising to what degree people can work around all those “non-extensible” parts.

Let’s first have a look at some approaches people adopt to extend their editors:

• Neovim and many TUI editors don’t have native scrollbar support since they are bounded by what ANSI provides. But guess what? By coloring the rightmost column with extmark and virt_text, it’s totally doable to display a pretty scrollbar for Neovim.
• Stock Emacs does not support cursor animations. And yet, you’ve guessed it, people:
  • Come up with patches for this,
  • Or extend Lisp code with Python, which in turn calls PyQt or compositor commands to control overlayed windows, so as to display things like moving cursors.

• Similarly, Emacs Application Framework allows you to write GUI programs for Emacs by … first programming them in PyQt and then sticking them onto the Emacs window, so that the PyQt window “fills” the corresponding buffer window.

  Not all Wayland compositors provide a way to programmatically position user windows, and that can be a problem for EAF.

• And, have you heard of EXWM (Emacs X Window Manager)?

The moral here is that a lot of things are more extensible than one might think. It’s just amazing how people keep coming up with all kinds of workarounds all the time. And yes, I do think those are all extensibility, regardless of “purity”.

Compared to extending via workarounds, extending in “pure Lisp” can be both easier and harder, as we are still bounded by coding conventions and existing code, and one cannot possibly extend everything without breaking some of them.

Let’s start by overriding a single function. For example, when exporting Org-mode files to HTML, Org-mode defaults to generating random HTML ID anchors. To change that, you just override the org-export-get-reference function that generates the IDs, right?

(advice-add #'org-export-get-reference :around #'org-html-stable-ids--get-reference)

Oh no! It turns out that, sometimes Org-mode directly calls org-html--reference, bypassing our override. That means we also need to redirect org-html--reference:

(advice-add #'org-html--reference :override #'org-html-stable-ids--reference)

Problem solved? No. Conventionally, Emacs Lisp code uses double dashes to tell the users “this function is internal”, as is in the org-html--reference name. Yes, by being free to extend any part of the editor, you are free to modify any internal functions or states, in a way that may or may not be problematic under specific circumstances, with code that can be broken in any future updates.

And it’s not the end of it. The el-patch package allows you to apply “patches” on most any Lisp code to modify behaviours nested deep inside a function:

;; Original function
(defun company-statistics--load ()
  "Restore statistics."
  (load company-statistics-file 'noerror nil 'nosuffix))

;; Patching
(el-patch-feature company-statistics)
(with-eval-after-load 'company-statistics
  (el-patch-defun company-statistics--load ()
                  "Restore statistics."
                  (load company-statistics-file 'noerror
                        ;; The patch
                        (el-patch-swap nil 'nomessage)
                        'nosuffix)))

;; Patched version
(defun company-statistics--load ()
  "Restore statistics."
  (load company-statistics-file 'noerror 'nomessage 'nosuffix))

Luckily, el-patch provides el-patch-validate so that you can worry less about your patches going ineffective or unexpectedly destructive. But you still need to maintain all your patches if anything goes wrong.

Any extensible system is not void of these problems. If you impose strong enough encapsulation, then eventually something can’t get customized; if you expose everything, well, good luck keeping backward compatibility (as the system maintainer) or forward compatibility (as the user doing your modifications). By making it possible to “extend any part of the editor,” you are literally making any part of your code unextensible, and now “every change breaks someone’s workflow.”

xkcd 1172 workflow, aka, spacebar heating

Emacs’s cross-language isolation/API might not be perfect, but I’m very grateful for it. If Emacs were written in pure Lisp code and anything is extensible, my work-in-progress Emacs clone couldn’t be remotely possible (because we do want to rid some of the spacebar heating problems while keeping some compatibility).

Allow me to be blunt: the “100% Lisp” argument is lazy marketing. Writing a Lisp-extended editor in Lisp won’t immediately make your editor more extensible. Extensibility comes from careful designing of your API interfaces, it comes from learning from your history, listening to user needs, and, after all these, taking the effort and time actually write the interfacing code.

This blog entry was adapted from a rant thread I posted to vent my shock about Emacs’s composition-function-table. Emacs is just never short of wonders and surprises, especially when you’re creating your own Emacsen by replicating Emacs.

By the way, this post is not against Lem, from which I’ve got a lot of inspirations and seen quite some good designs. But, it’s still the most convenient example I can think of. So, pardon me!