Panic on textDocument/didChange containing a multi-byte unicode character

[teor2345]

Description

rust-analyzer panics when a unicode character is in a comment.

Steps to Reproduce

1. Set up rust-analyzer with emacs using the LSP protocol
2. Open a project that uses rust-analyzer
3. Paste a unicode character into a comment (I used a zebra 🦓)
4. Move the insertion point over the unicode character, or make an edit near it

Sometimes the panic occurs on the paste, sometimes it needs an insertion point change or another edit.

Backtrace

version: f438398 2021-06-02 dev
notification: textDocument/didChange

thread 'main' panicked at 'assertion failed: self.is_char_boundary(n)', /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/alloc/src/string.rs:1698:29

Panic context:
>
version: f438398 2021-06-02 dev
notification: textDocument/didChange

thread 'main' panicked at 'assertion failed: self.is_char_boundary(n)', /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/alloc/src/string.rs:1698:29
stack backtrace:
   0: rust_begin_unwind
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/core/src/panicking.rs:92:14
   2: core::panicking::panic
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/core/src/panicking.rs:50:5
   3: rust_analyzer::lsp_utils::apply_document_changes
   4: core::ops::function::FnOnce::call_once
   5: rust_analyzer::dispatch::NotificationDispatcher::on
   6: rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::handle_event
   7: rust_analyzer::main_loop::<impl rust_analyzer::global_state::GlobalState>::run
   8: rust_analyzer::main_loop::main_loop
   9: rust_analyzer::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: "SendError(..)"', /cargo/registry/src/github.com-1ecc6299db9ec823/lsp-server-0.5.1/src/stdio.rs:29:37
stack backtrace:
   0: rust_begin_unwind
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/std/src/panicking.rs:515:5
   1: core::panicking::panic_fmt
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/core/src/panicking.rs:92:14
   2: core::result::unwrap_failed
             at /rustc/dbe459dedd33470f2cb28101157de316caaffa66/library/core/src/result.rs:1355:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I tried running export RUST_BACKTRACE=full; emacs ... but the env var didn't seem to get to rust-analyzer.

[lnicola]

Can you try in VS Code? CC emacs-lsp/lsp-mode#2080

[teor2345]

Can you try in VS Code? CC emacs-lsp/lsp-mode#2080

I'm sorry, but I don't have VS Code installed, and I don't have a lot of time to help diagnose this issue.

As far as I can tell from the conversation starting at emacs-lsp/lsp-mode#2080 (comment) , it's VS Code that's non-standard here. And there's an open issue for the fix.

There might also be a missing UTF-8 extension in either my emacs-lsp or rust-analyzer. But I don't know enough about LSP internals to know if I need a code fix, version upgrade, or config change. (And on which side that's needed.)

[lnicola]

VS Code isn't non-standard. LSP was defined by the VS Code implementers, and uses UTF-16 because that's what JavaScript likes. rust-analyzer and clangd implement a LSP extension to support UTF-8 offsets, but that's not standard by any means.

I guess lsp-mode doesn't have that yet.

In the meanwhile, the UTF-16 implementation in lsp-mode or emacs is (was?) buggy, which is why you get that crash on emoji.

[teor2345]

I guess I was confused because the missing rust-analyzer feature was implemented in #7657 , but it still panics.

I'll wait for emacs-lsp/lsp-mode#2080 , feel free to close this ticket if there's nothing that can be done here.

[lnicola]

Let's close this since we can't do anything about it.