Why building a Rust LSP is hard

Why building a Rust LSP is hard(rust-glancer.github.io)

104 points by agluszak 2 days ago | 42 comments

weinzierl 1 hour ago |

Reading this made me realize that I want two different things from an LSP that are sometimes at odds with each other:

1. Editing help

2. Reliable and comprehensive analysis

The editing help needs to deal with incomplete and inconsistent state and answers on a best effort basis. This is good when I'm writing code.

When I'm trying to understand code it usually is in a complete and compiling state but best effort is not enough. I expect complete and exhaustive answers.

Independently of that I'd love to read a similar analysis that compares the approaches of rust-anslyzer, rust-glancer and the JetBrains analysis engine in Rust Rover.

ashkankiani 47 minutes ago | |

This is a good observation, with the observation that there are two different levels of latency requirements. Most of the time, I would be pretty happy with an untyped, unsemantic, mildly smart heuristic based ident completion while the asynchronous semantic one finishes.

The nice thing about this dual setup is that I tend to only want the semantic one if I'm thinking more, so there's naturally a larger time budget for it.

One must always think about the experience they want in UX first, rather than the tools they want to build.

Panzerschrek 8 hours ago |

A lot of things described in this article applicable not only for Rust, but for almost any language. Like it's obvious that requests should be handled asynchronously and that conversions from/to UTF-16 are needed. But it's actually not so hard.

I have written a language server for my language too. The hardest thing was to find a way allowing providing useful autocompletion for a document in edited state, when it's not syntactically-correct. This is the trickiest part how to deal with such incorrectness without missing all the context necessary.

rhdunn 7 hours ago | |

I've not written a language server but have written a language plugin for IntelliJ.

I started with writing a correct recursive descent parser. I then extended it to detect, report, and recover from common syntax errors as I encountered them so that the parser is robust. And adding a parser test case for each of these (e.g. one test for each branch through an EBNF construction).

Some examples are:

1. missing keywords when the keyword can be detected from the current context (e.g. missing semicolon at the end of a statement);

2. using the wrong token (e.g. `:` instead of `::` in a C++ namespace qualified name);

3. detecting and ignoring whitespace in a whitespace-sensitive qualification (e.g. in XML QNames);

4. keeping in the prolog state (where functions are defined) when there are errors so that functions after the error don't get lost;

5. lexing incomplete literals like `10e` so they can be handled as integers in the parser and emitting an error for them.

Panzerschrek 7 hours ago | | |

> recover from common syntax errors

It's a dead-end. Sure, it can work in simple cases, but there will be always a case where such syntax recovery isn't possible. That's why relying only on syntax recovery isn't an option.

Because of that I use a different approach. I do parse on each document editing, but such parsing is guaranteed to produce valid results only up to the point with broken syntax, where editing usually takes place. Such parsing is enough to reconstruct location of the point where editing takes place (namespace/class/function) and to reconstruct local context (local variables declared prior to editing place). This allows to perform almost perfect autocompletion by suggesting global and local names available at the editing point. In order to provide proper suggestion of non-local names declared after the editing point, I do keep a structure for the most recent document state with valid syntax.

With features like "go to definition" I do the same. I store a hash-table with location to definition point mapping, but it's updated only from time to time and only if document syntax is valid. In order to be usable for cases with edits made after building such hash-table I just perform text-based position mapping using accumulated edit events.

mitxela 11 hours ago |

Using a JSON TCP connection for what on Windows would be direct function calls (COM) or in Eclipse would be direct function calls between Java modules always felt a bit gross.

klodolph 8 hours ago |

The particulars of Rust make this a little more difficult, I think. There’s a certain tension between making your language more concise and adding useful redundancies, and Rust has generally gone to the “concise” side, with some redundancies that can make the tooling a little more painful. Like with imports.

  impl std::fmt::Display for Blah {
  }

If your language makes you qualify your imports (like above) then your LSP can, delightfully, still reliably do certain ops like renaming, even when chunks of your project aren’t parsing. But if you glob import std::fmt, and glob import something else, you are fucked. Display could come from anywhere (maybe from a module that has a parse error at the moment). I really appreciate languages where glob imports (or their equivalent) are either disallowed entirely or where typical code doesn’t use it.

Meanwhile, if you add a new file, there’s this little dance where you say:

  mod mycoolmod;

And then you create mycoolmod.rs. Or you do it the other way around. A little redundancy (the file exists and it is declared), that seems to just create a little friction in the LSP because mycoolmod doesn’t get a working LSP until it’s declared in the parent (you have to create both, and then you get a transient diagnostic that your module is unused for a while yet). A small issue, just another little bit of friction in the tooling of Rust that has nothing to do with the type system.

verandaguy 1 hour ago | |

Having worked with rust for nearly two years now (granted, on one team with agreed-upon standards):

- glob imports are rare in my experience, less for the LSP’s sake and more for code self-documentation

- the `mod foo` line exists because omitting it cannot fall back to a reasonable default visibility level (`pub`/`pub(crate)`/<none> (private))

dominotw 32 minutes ago |

one thing i like about llm hype is that killed rust hype on hn

octoberfranklin 11 hours ago |

Enter hell: LSP assumes that it's the source of truth, but you still need to access the filesystem yourself, and do it in a synchronized way

LSP is an example of utterly horrid technical design.

Stop letting Microsoft design protocols and APIs. They are so. bad. at. it.

chrisjj 5 hours ago |

> Why building a Rust LSP is hard

Not as hard as understanding what LSP means, apparently.

I was interested until I saw this project is simply an LSP server.