Manual impl of Debug on Token

[dhruvmanila]

Summary

I look at the token stream a lot, not specifically in the playground but in the terminal output and it's annoying to scroll a lot to find specific location. Most of the information is also redundant.

The final format we end up with is: <kind> <range> (flags = ...) e.g., String 0..4 (flags = BYTE_STRING) where the flags part is only populated if there are any flags set.

[MichaReiser]

Hmm, I think I preferred the old one. The new representation isn't very self explanatory without e.g. knowing what 0..5 means.

• I agree that kind and Token could be collapsed. We could take inspiration from Rust where the Debug implementation of Token::String gets outputted exactly as such. So we could change it to Token::Lpar
• I agree that we should omit the flags if they're empty.
• We could show the range as Token::Lpar@0..5 but it's not entirely clear to me how we would want to show the flags if they're non empty

How about: Token::Lpar@0..5(flags = ...) which would even be more dense?

[dhruvmanila]

How about: Token::Lpar@0..5(flags = ...) which would even be more dense?

Hmm, I'm not sure about not having any space in between so here's without space:

Name@0..1,
Equal@2..3,
String@4..12(flags = BYTE_STRING),
Newline@11..12,
FStringStart@12..15(flags = F_STRING | RAW_STRING_LOWERCASE),
FStringMiddle@15..19(flags = F_STRING | RAW_STRING_LOWERCASE),
Lbrace@19..20,
Name@20..21,
Plus@22..23,
Name@24..25,
Rbrace@25..26,
FStringMiddle@26..30(flags = F_STRING | RAW_STRING_LOWERCASE),
FStringEnd@30..31(flags = F_STRING | RAW_STRING_LOWERCASE),
Newline@31..32

And, here's with space (I like this):

Name @ 0..1,
Equal @ 2..3,
String @ 4..12 (flags = BYTE_STRING),
Newline @ 12..13,
FStringStart @ 13..16 (flags = F_STRING | RAW_STRING_LOWERCASE),
FStringMiddle @ 16..20 (flags = F_STRING | RAW_STRING_LOWERCASE),
Lbrace @ 20..21,
Name @ 21..22,
Plus @ 23..24,
Name @ 25..26,
Rbrace @ 26..27,
FStringMiddle @ 27..31 (flags = F_STRING | RAW_STRING_LOWERCASE),
FStringEnd @ 31..32 (flags = F_STRING | RAW_STRING_LOWERCASE),
Newline @ 32..33

[MichaReiser]

To me, this has a bit too much spacing. It's unclear where the @ belongs

Name @ 0..1,
Equal @ 2..3,
String @ 4..12 (flags = BYTE_STRING),
Newline @ 12..13,
FStringStart @ 13..16 (flags = F_STRING | RAW_STRING_LOWERCASE),
FStringMiddle @ 16..20 (flags = F_STRING | RAW_STRING_LOWERCASE),
Lbrace @ 20..21,
Name @ 21..22,
Plus @ 23..24,
Name @ 25..26,
Rbrace @ 26..27,
FStringMiddle @ 27..31 (flags = F_STRING | RAW_STRING_LOWERCASE),
FStringEnd @ 31..32 (flags = F_STRING | RAW_STRING_LOWERCASE),
Newline @ 32..33

I think I would prefer a combination of the two

Name@0..1,
Equal@2..3,
String@4..12 (flags = BYTE_STRING),
Newline@12..13,
FStringStart@13..16 (flags = F_STRING | RAW_STRING_LOWERCASE),
FStringMiddle@16..20 (flags = F_STRING | RAW_STRING_LOWERCASE),
Lbrace@20..21,
Name@21..22,
Plus@23..24,
Name@25..26,
Rbrace@26..27,
...

[dhruvmanila]

To me, this has a bit too much spacing. It's unclear where the @ belongs

I guess I'm fine with not having a space around @ but out of curiosity what are the options where @ could belong to in your view? I just find it comparatively easier to read the entire list quickly if there's a space, which indicates that the word ends right there.

[MichaReiser]

With the space, the @ is mainly noise to me. Maybe we just remove it and go with something closer you had in your first version? String 4..12 (flags = ByteString)?

[dhruvmanila]

With the space, the @ is mainly noise to me. Maybe we just remove it and go with something closer you had in your first version? String 4..12 (flags = ByteString)?

Yeah, I think that's good enough.

[github-actions]

ruff-ecosystem results

Linter (stable)

✅ ecosystem check detected no linter changes.

Linter (preview)

✅ ecosystem check detected no linter changes.

Formatter (stable)

✅ ecosystem check detected no format changes.

Formatter (preview)

✅ ecosystem check detected no format changes.