Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove Nonterminal and TokenKind::Interpolated #763

Closed
1 of 3 tasks
nnethercote opened this issue Jun 23, 2024 · 3 comments
Closed
1 of 3 tasks

Remove Nonterminal and TokenKind::Interpolated #763

nnethercote opened this issue Jun 23, 2024 · 3 comments
Labels
major-change A proposal to make a major change to rustc major-change-accepted A major change proposal that was accepted T-compiler Add this label so rfcbot knows to poll the compiler team

Comments

@nnethercote
Copy link

nnethercote commented Jun 23, 2024

Proposal

Declarative macro expansion involves the parsing of token sequences into AST nodes, which are then pasted back into the token stream as TokenKind::Interpolated tokens. Each such token contains a Nonterminal, an enum that can contain an AST expr, or stmt, or item, or block, etc.

This MCP proposes to instead convert the AST node back into tokens and insert those tokens into the token stream, with invisible (a.k.a. "none") delimiters around the token sequence to protect precedence.

One reason for the change is that it's really weird to have AST pieces interpolated through a token stream, because tokens and AST nodes are two different levels. A bit like having a sequence of words and punctuation in natural language, but some of the "words" are themselves phrases or sentences or paragraphs. When I first encountered Interpolated tokens it took me some time to understand them.

Another reason is that it makes the implementation of declarative macros and proc macros more similar. Currently if you pass an Interpolated token to a proc macro, the proc macro bridge converts it into a Group that is delimited by invisible delimiters. Then if that gets returned from the proc macro the invisible delimiters remain. (Proc macros can also create invisible-delimited sequences from scratch.) In other words, proc macros work entirely with token streams, so it will be nice for declarative macros (and the parser) to do the same.

Also, currently the parser just ignores all invisible delimiters! This leads to occasional precedence issues like rust-lang/rust#67062. Fixing this bug was one of my original motivations with this work. It turns out more complicated than I originally expected, and this MCP won't be enough to fix the bug, though it's definitely a step in the right direction because it will result in us having a single mechanism for grouping tokens instead of two, and the parser will no longer eliminate all invisible delimiters.

This change will also completely eliminate the "forwarding a match fragment" limitation of declarative macros. (Update: this is closer to being eliminated, but is still necessary.)

There will be some minor perf effects. Having to tokenize and reparse AST fragments has non-zero cost. Most rustc-perf benchmarks aren't affected. deep-vector is the big exception. It's an artificial stress test containing a single vec! call with 100,000+ zeroes in it, which is a pathological case for this change. Currently the biggest regression is 90% for an incr-unchanged check build, but there's an easy change that will reduce that to 25-30%. hyper and libc also see some moderate regressions, up to 7% in the worst case. I think these regressions can be reduced some more, but probably not fully eliminated. Some other benchmarks see slight improvements of up to 1.5%, probably because TokenKind can now be made Copy, and tokens get copied around a lot.

Mentors or Reviewers

@petrochenkov will review, and has helped a lot along the way.

rust-lang/rust#124141 has a draft implementation, which is very closely to completely working. This is my third attempt at this change in three years (rust-lang/rust#96724 and rust-lang/rust#114647 were my previous attempts) and I'm confident it will succeed this time. Other than the "forwarding a match fragment" limitation being lifted, there shouldn't be any user-visible changes.

Process

The main points of the Major Change Process are as follows:

  • File an issue describing the proposal.
  • A compiler team member or contributor who is knowledgeable in the area can second by writing @rustbot second.
    • Finding a "second" suffices for internal changes. If however, you are proposing a new public-facing feature, such as a -C flag, then full team check-off is required.
    • Compiler team members can initiate a check-off via @rfcbot fcp merge on either the MCP or the PR.
  • Once an MCP is seconded, the Final Comment Period begins. If no objections are raised after 10 days, the MCP is considered approved.

You can read more about Major Change Proposals on forge.

Comments

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

@nnethercote nnethercote added T-compiler Add this label so rfcbot knows to poll the compiler team major-change A proposal to make a major change to rustc labels Jun 23, 2024
@rustbot
Copy link
Collaborator

rustbot commented Jun 23, 2024

This issue is not meant to be used for technical discussion. There is a Zulip stream for that. Use this issue to leave procedural comments, such as volunteering to review, indicating that you second the proposal (or third, etc), or raising a concern that you would like to be addressed.

Concerns or objections to the proposal should be discussed on Zulip and formally registered here by adding a comment with the following syntax:

@rustbot concern reason-for-concern 
<description of the concern> 

Concerns can be lifted with:

@rustbot resolve reason-for-concern 

See documentation at https://forge.rust-lang.org

cc @rust-lang/compiler @rust-lang/compiler-contributors

@rustbot rustbot added the to-announce Announce this issue on triage meeting label Jun 23, 2024
@petrochenkov
Copy link

@rustbot second

@rustbot rustbot added the final-comment-period The FCP has started, most (if not all) team members are in agreement label Jun 24, 2024
@apiraino apiraino removed the to-announce Announce this issue on triage meeting label Jul 4, 2024
@apiraino
Copy link
Contributor

apiraino commented Jul 8, 2024

@rustbot label -final-comment-period +major-change-accepted

@apiraino apiraino closed this as completed Jul 8, 2024
@rustbot rustbot added major-change-accepted A major change proposal that was accepted to-announce Announce this issue on triage meeting and removed final-comment-period The FCP has started, most (if not all) team members are in agreement labels Jul 8, 2024
@apiraino apiraino removed the to-announce Announce this issue on triage meeting label Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
major-change A proposal to make a major change to rustc major-change-accepted A major change proposal that was accepted T-compiler Add this label so rfcbot knows to poll the compiler team
Projects
None yet
Development

No branches or pull requests

4 participants