Always preserve None-delimited groups in a captured TokenStream
#83548
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, we would silently remove any `None`-delimiters when capturing a `TokenStream`, 'flattenting' them to their inner tokens. This was not normally visible, since we usually have `TokenKind::Interpolated` (which gets converted to a `None`-delimited group during macro invocation) instead of an actual `None`-delimited group. However, there are a couple of cases where this becomes visible to proc-macros: 1. A cross-crate `macro_rules!` macro has a `None`-delimited group stored in its body (as a result of being produced by another `macro_rules!` macro). The cross-crate `macro_rules!` invocation can then expand to an attribute macro invocation, which needs to be able to see the `None`-delimited group. 2. A proc-macro can invoke an attribute proc-macro with its re-collected input. If there are any nonterminals present in the input, they will get re-collected to `None`-delimited groups, which will then get captured as part of the attribute macro invocation. Both of these cases are incredibly obscure, so there hopefully won't be any breakage. This change will allow more agressive 'flattenting' of nonterminals in rust-lang#82608 without losing `None`-delimited groups.
|
r? @estebank (rust-highfive has picked a reviewer for you, use r? to override) |
rust-highfive
added
the
S-waiting-on-review
| @@ -172,6 +172,13 @@ struct TokenCursor { | |||
| // appended to the captured stream when | |||
| // we evaluate a `LazyTokenStream` | |||
| append_unglued_token: Option<TreeAndSpacing>, | |||
| // If `true`, skip the delimiters for `None`-delimited groups, | |||
| // and just yield the inner tokens. This is `true` during | |||
| // normal parsing, since the parser code is not currently prepared | |||
There was a problem hiding this comment.
To work correctly it should be prepared though, cc #67062.
(In other words, skip_none_delims should eventually become always false.)
|
@bors r+ |
|
📌 Commit f94360f has been approved by |
bors
added
S-waiting-on-bors
Dylan-DPC-zz
pushed a commit
to Dylan-DPC-zz/rust
that referenced
this pull request
Mar 27, 2021
…trochenkov Always preserve `None`-delimited groups in a captured `TokenStream` Previously, we would silently remove any `None`-delimiters when capturing a `TokenStream`, 'flattenting' them to their inner tokens. This was not normally visible, since we usually have `TokenKind::Interpolated` (which gets converted to a `None`-delimited group during macro invocation) instead of an actual `None`-delimited group. However, there are a couple of cases where this becomes visible to proc-macros: 1. A cross-crate `macro_rules!` macro has a `None`-delimited group stored in its body (as a result of being produced by another `macro_rules!` macro). The cross-crate `macro_rules!` invocation can then expand to an attribute macro invocation, which needs to be able to see the `None`-delimited group. 2. A proc-macro can invoke an attribute proc-macro with its re-collected input. If there are any nonterminals present in the input, they will get re-collected to `None`-delimited groups, which will then get captured as part of the attribute macro invocation. Both of these cases are incredibly obscure, so there hopefully won't be any breakage. This change will allow more agressive 'flattenting' of nonterminals in rust-lang#82608 without losing `None`-delimited groups.
bors
added a commit
to rust-lang-ci/rust
that referenced
this pull request
Mar 27, 2021
Rollup of 8 pull requests Successful merges: - rust-lang#81351 (combine: stop eagerly evaluating consts) - rust-lang#82525 (make unaligned_references future-incompat lint warn-by-default) - rust-lang#82626 (update array missing `IntoIterator` msg) - rust-lang#82917 (Add function core::iter::zip) - rust-lang#82993 (rustdoc: Use diagnostics for error when including sources) - rust-lang#83522 (Improve fs error open_from unix) - rust-lang#83548 (Always preserve `None`-delimited groups in a captured `TokenStream`) - rust-lang#83555 (Add #[inline] to io::Error methods) Failed merges: - rust-lang#83130 (escape_ascii take 2) r? `@ghost` `@rustbot` modify labels: rollup
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, we would silently remove any
None-delimiters whencapturing a
TokenStream, 'flattenting' them to their inner tokens.This was not normally visible, since we usually have
TokenKind::Interpolated(which gets converted to aNone-delimitedgroup during macro invocation) instead of an actual
None-delimitedgroup.
However, there are a couple of cases where this becomes visible to
proc-macros:
macro_rules!macro has aNone-delimited groupstored in its body (as a result of being produced by another
macro_rules!macro). The cross-cratemacro_rules!invocationcan then expand to an attribute macro invocation, which needs
to be able to see the
None-delimited group.input. If there are any nonterminals present in the input, they will
get re-collected to
None-delimited groups, which will then getcaptured as part of the attribute macro invocation.
Both of these cases are incredibly obscure, so there hopefully won't be
any breakage. This change will allow more agressive 'flattenting' of
nonterminals in #82608 without losing
None-delimited groups.