ANSI escaping is a mess (or, why Emacs adds [] to my file?)

Kana

So I’ve long been puzzled by this behavior of my Emacs config: sometimes when I open a file in the terminal with Emacs, it adds a pair of square brackets around the current line/word/expression. After some digging, I’ve come to the conclusion that, I absolutely don’t want to work with ANSI escape codes.

A quick demonstration:

1. Why?¶

Let’s launch a vanilla Emacs session with emacs -q -nw and see what happens under the hood with C-h l (view-lossage):

ESC [ I C-h l ;; view-lossage

Great, so ESC [ I is the culprit. Emacs interprets ESC [ (that is, first pressing the Escape key, releasing it, and then pressing the bracket ([) key) as M-[ (or Alt+[). In my setup, this key binds to evil-cp-wrap-next-square, which wraps the current S-expression with brackets. Bracket mystery solved!

Now, why does my terminal send this ANSI sequence to Emacs? It seems XTerm (and compatible terminals) uses ESC [ I and ESC [ O for focus and unfocus events: (xterm/button.c)

void
SendFocusButton(XtermWidget xw, XFocusChangeEvent *event)
{
  if (okSendFocusPos(xw)) {
    ANSI reply;

    memset(&reply, 0, sizeof(reply));
    reply.a_type = ANSI_CSI;

    reply.a_final = CharOf((event->type == FocusIn) ? 'I' : 'O');
    unparseseq(xw, &reply);
  }
  return;
}

Then, why are other programs unaffected? Well, it turns out these events are opt-in and Emacs explicitly requests them, except that it is somehow not handling them correctly when the M-[ key binding is present: (xterm.el)

(defun xterm--init-focus-tracking ()
  "Terminal initialization for focus tracking mode."
  (send-string-to-terminal "\e[?1004h")
  (push "\e[?1004l" (terminal-parameter nil 'tty-mode-reset-strings))
  (push "\e[?1004h" (terminal-parameter nil 'tty-mode-set-strings)))

So this is an Emacs bug then? And why am I complaining about ANSI? Because ESC [ I is an Escape key press followed by a [ key press followed by I. And how are programs expected to handle them without using all kinds of quirks? Yes, all terminal UI programs have been handling these just fine ¹, but it really seems off to me that an escape sequence is not escaping its escape marker (ESC).

Or, maybe not, considering “ESC[31m”?! ANSI Terminal security in 2023 and finding 10 CVEs.

Is ESC an Escape key press or the start of an escape sequence? Wait a millisecond and check the buffer to see if it is followed by something else. This “time escaped” protocol might see some use for hardwares speaking baudrates (and might have actually come from the physical terminal age?), but I really wish we had better choices.

2. A Tale of Two Takes¶

This section is in part an overdue effort to turn a rant-only post into an Emacs Carnival 2025-06: Take Two submission. Instead of ranting, let’s actually do some coding, shall we?

ANSI sequence handling is a well-known issue for terminal programs. Not every TUI binds to M-O (ESC O) or M-[ (ESC [), but many of them do bind to ESC which needs some special treatment:

ESCDELAY

For curses to distinguish the ESC character resulting from a user’s press of the “Escape” key on the input device from one beginning an escape sequence (as commonly produced by function keys), it waits after the escape character to see if further characters are available on the input stream within a short interval. ESCDELAY stores this interval in milliseconds. ²

Most famously for Emacs users, the Evil mode and viper mode also bind to the Escape key and handle it perfectly fine in a terminal. So let’s take a look at how they work around the ANSI issue.

If you use vim in a screen/tmux session over a remote SSH connection, you might be able to feel those latencies and ESCDELAY adding up.

2.1. Emacs Keymaps¶

Before digging into Evil mode, let’s first try to understand how an <up> key press in a terminal is turned into a previous-line command in Emacs. This will involve several keymaps (trie structures mapping from key sequences to their key bindings):

(let ((keymap (make-sparse-keymap)))
  (define-key keymap (kbd "C-x C-f") #'find-file)
  (define-key keymap (kbd "C-x C-s") #'save-buffer)
  (define-key keymap (kbd "C-h i") #'info)
  (prin1-to-string `',keymap))

'(keymap
  (?\C-x . (keymap
            (?\C-f . find-file)
            (?\C-s . save-buffer)))
  (?\C-h . (keymap
            (?i    . info))))

Upon pressing an <up> arrow key, a terminal typically sends ESC O A to Emacs.
Emacs consumes one character at a time, so it first tries to find bindings for the ESC character:
1. Emacs consults a dedicated input-decode-map keymap for interpreting an ANSI sequence:
```
(lookup-key input-decode-map (kbd "ESC"))
```
```
(keymap ...)
```
  The result is a nested keymap, meaning the ANSI sequence is incomplete.
2. Emacs then looks up the sequence in the current key bindings (global-map and others):
```
(lookup-key global-map (kbd "ESC"))
```
```
ESC-prefix
```
  And ESC-prefix is also a keymap, meaning ESC is not directly bound. So Emacs continues its search.

Emacs tries to consume one more character and looks up ESC O instead:

(lookup-key input-decode-map (kbd "ESC O"))

(keymap ...) ; A nested keymap - incomplete ANSI sequence

(lookup-key global-map (kbd "ESC O"))

nil ; Not bound at all

Still, no key binding is found and the search continues.

Finally, Emacs decides to try ESC O A, which is transformed by input-decode-map into [up]:
```
(lookup-key input-decode-map (kbd "ESC O A"))
```
```
[up]
```
The transformed sequence is then looked up in key binding maps for its corresponding command:
```
(lookup-key global-map [up])
```
```
previous-line
```
Then, Emacs executes previous-line command that moves up our cursor by one line.

2.2. The Evil Approach¶

Binding to the ESC character in Emacs is problematic because it interferes with input-decode-map. As is shown above, the ANSI sequence ESC O A gets converted into [up] only if both ESC and ESC O are unbound. Otherwise:

If ESC were bound, the lookup would stop prematurely:
1. (lookup-key input-decode-map (kbd "ESC")) yields a nested keymap.
2. (lookup-key global-map (kbd "ESC")) yields a command and Emacs won’t look further for ANSI sequences.

Instead of binding to ESC directly, Evil takes an evil approach by binding a weird menu-item to ESC in input-decode-map:

(menu-item "" (keymap ...) :filter evil-esc)

2.2.1. `menu-item`: Menus Implemented As Keymaps¶

Emacs uses keymaps for many things. Or, you can say it is “key events” that are abused by Emacs to represent all kinds of UI events. But anyway, other than nested keymaps and command functions, one can also put a (menu-item ...) in keymaps, meaning … a menu item.

This is fine, until you realize that one can also use a menu-item for key events along with its :filter option:

:filter function

function must be a function which, if called with one argument—the list of the other menu items—returns the actual items to be displayed in the menu.

And this is exactly what Evil mode uses to make ESC available for binding without interfering with ANSI events:

They bind (menu-item "" (keymap ...) :filter evil-esc) to the ESC key (in input-decode-map). Note that it has the evil-esc function as its :filter.
On ESC key presses, Emacs calls the evil-esc filter, expecting it to return an actual keymap item. However, evil-esc does not return immediate: it waits a few milliseconds (evil-esc-delay) to see if more key events are sent.
- If so, it treats these bursts of key presses as ANSI sequences (similar to ESCDELAY), and returns a keymap to decode the incoming ANSI sequences.
- Otherwise, this ESC event should probably be a real <escape> key press, and evil-esc maps it accordingly. ³

Well, actually things are a bit more complex: Emacs has two separate events for escape keys. ESC is usually sent by terminals, while 'escape is from GUI Emacs. evil-esc maps real ESC presses into 'escape events, so that one can bind to escape keys (with 'escape) without interfering in TUI ESC processing. (No, binding to ESC will still cause chaos.)

2.2.2. Evil Fixes To `M-[` and `M-O` ¶

So the above is how Evil mode handles ESC key presses. And for M-O and M-[, we can follow suit and add menu-item bindings for ESC O and ESC [.

I won’t go into the implementation details here. But if you are interested, here (ansi-workaround.el) is the code snippet I currently use to make M-O and M-[ available under terminal Emacs. ⁴

⁴

Similar to how Evil mode requires ESC and 'escape events to make the magic happen ³, ansi-workaround.el also uses, for example, M-O and 'symbol-M-O to make things work. It’s very hacky.

2.3. The Workaround Approach¶

Of course, if you are an Emacs package author that wishes to bind to M-O or M-[, you cannot rely on the user to apply the ~~hacky~~ fixes above themselves (or use GUI Emacs). Instead, you should ensure these bindings never bind under TUI to avoid messing up ANSI sequences.

The best approach (that I’m aware of) to disabling certain bindings under TUI also uses menu-item: make your binding a menu-item that returns nil under terminal sessions (see evil-cleverparens/pull/119 for an example).

3. Afterwords¶

I actually first spent some time reading through the C code processing key events in Emacs to understand how Evil/viper mode get to bind to the escape key, because, I mean, ANSI sounds quite low-level and very C-ish to me. But then I dug through the mailing list and found that they have the ESC logic mostly implemented in ELisp, and suddenly it all clicked: it’s Emacs after all.