tokens in SemanticHighlightingInformation should be optional

[jackguo380]

Hi,

I am currently working on a PR (autozimu/LanguageClient-neovim#954) to support the proposed semantic highlighting protocol for LanguageClient-neovim. I noticed that the SemanticHighlightingInformation type's tokens field should be optional but is currently not:

#[cfg(feature = "proposed")]
pub struct SemanticHighlightingInformation {
    /**
     * The zero-based line position in the text document.
     */
    pub line: i32,

    /**
     * A base64 encoded string representing every single highlighted characters with its start position, length and the "lookup table" index of
     * of the semantic highlighting [TextMate scopes](https://manual.macromates.com/en/language_grammars).
     * If the `tokens` is empty or not defined, then no highlighted positions are available for the line.
     */
    #[serde(deserialize_with = "SemanticHighlightingToken::deserialize_tokens")]
    #[serde(serialize_with = "SemanticHighlightingToken::serialize_tokens")]
    pub tokens: Vec<SemanticHighlightingToken>,
}

Theres a document describing the protocol here: https://github.com/microsoft/vscode-languageserver-node/blob/5a9b33c23de84c3341e011e79221795a8059375b/protocol/src/protocol.semanticHighlighting.proposed.md

It shows that the tokens field can be omitted for a line if there are no tokens to highlight there. So the tokens field should be a Option<Vec<SemanticHighlightingToken>> instead.

Currently this is a problem since I can't deserialize the following message from eclipse/eclipse.jdt.ls because it omits tokens:

Message: {"jsonrpc":"2.0","method":"textDocument/semanticHighlighting","params":{"lines":[{"line":29},{"line":31},{"line":32},{"line":36},{"line":39},{"line":41},{"line":42},{"line":46},{"line":47},{"line":49},{"line":50},{"line":54},{"line":57},{"line":58},{"line":59},{"line":60},{"line":61},{"line":63},{"line":64},{"line":65},{"line":68},{"line":69},{"line":71},{"line":72},{"line":73},{"line":74},{"line":75},{"line":76},{"line":79},{"line":80},{"line":81},{"line":83},{"line":84},{"line":85},{"line":86},{"line":87},{"line":88},{"line":89},{"line":93},{"line":95},{"line":96},{"line":97},{"line":101},{"line":102},{"line":103},{"line":107},{"line":109},{"line":112},{"line":114},{"line":115},{"line":116},{"line":118},{"line":121},{"line":123},{"line":125},{"line":127},{"line":128},{"line":129},{"line":130},{"line":134},{"line":135},{"line":136},{"line":138},{"line":140},{"line":141},{"line":142},{"line":146},{"line":147},{"line":148},{"line":149},{"line":150},{"line":154},{"line":156},{"line":158},{"line":161},{"line":163}],"textDocument":{"uri":"file:///home/guoj/Documents/Year-4/CPEN-431/assignment2/cpen431-assignment2/src/main/java/com/cpen431/Main.java","version":2}}}

Error("missing field `tokens`", line: 0, column: 0)

Just posting as a issue for now, I can open up a PR if that's preferred.

[Marwes]

Nice catch! A PR would definitely be accepted!

[jackguo380]

Thanks @Marwes,

Could a new version be released so these changes can be used?

[kjeremy]

Is this an earlier version of the semantic tokens proposal located: https://github.com/microsoft/vscode-languageserver-node/blob/dbaeumer/semanticTokens/protocol/src/protocol.sematicTokens.proposed.ts ?

[kjeremy]

I think it is: https://github.com/microsoft/vscode/wiki/Semantic-Highlighting-Overview

[jackguo380]

I'm not entirely sure but but I think that first link might be a vscode internal protocol, not a proposal for LSP. The original PR for the semantic highlighting at microsoft/vscode-languageserver-node#367 still remains the same.

[kjeremy]

The latest proposed for 3.16 is https://github.com/microsoft/vscode-languageserver-node/blob/master/protocol/src/protocol.sematicTokens.proposed.ts just updated a few days ago