Generate metadata by iterating on DefId instead of traversing the HIR tree

[cjgillot]

Most of the work in metadata encoding consists in walking the HIR, and calling the different queries on each node.
This can be done by simply looping on all the DefIds, and invoking each query directly.

This requires to create non-panicking versions of some queries.

Papercuts:

• fix polymorphize tests;
• find a better way to iterate on DefIds;
• handle def_kind, entry_kind and children tables.

[rust-highfive]

r? @oli-obk

(rust-highfive has picked a reviewer for you, use r? to override)

[oli-obk]

I like the idea, we definitely need to check perf and see how it fares, but I don't see actual problems with it.

I still think it needs an MCP, can you open one so the compiler team knows about it? We have a meeting later today, so it should go fast.

[petrochenkov]

This is an interesting idea, and there are multiple factors in play here

• the code certainly looks cleaner
• it's harder to forget to encode something like it happened in rustc_metadata: Do not forget to encode inherent impls for foreign types #77375
• on the other hand it's easier to encode something dummy that otherwise wouldn't be encoded at all (this is a performance concern)
• also HIR visiting should be very cheap compared to actual encoding so we may not win much here
• if property X only exists for a small portion of DefIds then we may spend a lot of time querying it for all DefIds unnecessarily (also a performance concern)
• on the other hand the query system can potentially support bulk queries ("give me Xs for all DefIds") instead of "for def_id in all_def_ids { give me X for def_id }" if the former has performance benefits

[cjgillot]

* on the other hand it's easier to encode something dummy that otherwise wouldn't be encoded at all (this is a performance concern)
* if property X only exists for a small portion of `DefId`s then we may spend a lot of time querying it for all `DefId`s unnecessarily (also a performance concern)

I am also worried about this. This will definitely need a perf run.

* also HIR visiting should be very cheap compared to actual encoding so we may not win much here

I would like to drop the HIR earlier in the future. If this ever happens, HIR visiting will be impossible at such a late stage.

* on the other hand the query system can potentially support bulk queries ("give me Xs for all `DefId`s") instead of "for def_id in all_def_ids { give me X for def_id }" if the former has performance benefits

In the initial implementation, I wanted to use the query caches directly, like this:

tcx.queries.def_span.iter_results(|iter| {
  for (k, _, v) in iter {
    if !k.is_local() { continue }
    record!(self.tables.def_span[k] <- v);
  }
}

However, this did not force the queries if the def_id had not been forced earlier.
Since the forcing has to be done at some point, the proposed solution seemed to me the most robust.

[cjgillot]

I filed an MCP. The Zulip discussion can be found here:
https://rust-lang.zulipchat.com/#narrow/stream/233931-t-compiler.2Fmajor-changes/topic/Do.20not.20traverse.20HIR.20to.20generate.20metadata.20compiler-team.23392

[oli-obk]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit 5cdfe091c3363720dcd399d458ff4b317bf6f066 with merge 2e325a37b8ff6a4cce0abcacff6dc1b64a3cf624...

[bors]

☀️ Try build successful - checks-actions
Build commit: 2e325a37b8ff6a4cce0abcacff6dc1b64a3cf624 (2e325a37b8ff6a4cce0abcacff6dc1b64a3cf624)

[rust-timer]

Queued 2e325a37b8ff6a4cce0abcacff6dc1b64a3cf624 with parent 30a4273, future comparison URL.

@rustbot label: +S-waiting-on-perf

[rust-timer]

Finished benchmarking try commit (2e325a37b8ff6a4cce0abcacff6dc1b64a3cf624): comparison url.

Benchmarking this pull request likely means that it is perf-sensitive, so we're automatically marking it as not fit for rolling up. Please note that if the perf results are neutral, you should likely undo the rollup=never given below by specifying rollup- to bors.

Importantly, though, if the results of this run are non-neutral do not roll this PR up -- it will mask other regressions or improvements in the roll up.

@bors rollup=never
@rustbot label: +S-waiting-on-review -S-waiting-on-perf

[bjorn3]

Significant regressions of up to 50%. This is due to optimized_mir being invoked several times as much.

[oli-obk]

@bors try @rust-timer queue

[rust-timer]

Awaiting bors try build completion.

[bors]

⌛ Trying commit 7ad4bee45bbafeae6ea1350d427bd22610062ae2 with merge 7ea7454bf3e5d01fd6f85df9620da4191ef48eaa...

[bors]

☔ The latest upstream changes (presumably #75384) made this pull request unmergeable. Please resolve the merge conflicts.

[crlf0710]

@cjgillot Ping from triage! There's merge conflicts now.

[jackh726]

@cjgillot What are the next steps here? Should I review this PR as-is? Or do you want to continue splitting this into separate PRs?

[crlf0710]

@cjgillot Ping from triage, any updates on this?

[camelid]

triage: @cjgillot What's the status of this? Does this PR need to be split up or do the merge conflicts just need to be fixed?