[self-profiler] Make the profiler faster/more efficient

[wesleywiser]

Related to #58372

r? @michaelwoerister

[michaelwoerister]

Once #58309 lands, we should do a perf run to measure the overhead. I suspect that emitting JSON instead of binary data is still more expensive than we'd like it to be.

[wesleywiser]

Probably, I haven't looked into doing binary serialization yet. Is there any existing infrastructure in the compiler for that or should I roll my own?

[wesleywiser]

(FYI, in case you were remembering the performance of serialization as it was at All-Hands, that was without buffering the writes. I added a BufWriter around the file and it significantly improved the performance.)

[michaelwoerister]

There is rustc's libserialize which works OK for metadata and incremental caches.

[michaelwoerister]

I don't know how easy or hard it is to use libserialize outside of the compiler though. Let's check how the JSON version does first.

[wesleywiser]

I think it's just a matter of #[derive(RustcDecodable)] but I'm not sure.

[bors]

☔ The latest upstream changes (presumably #58455) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

I think it's just a matter of #[derive(RustcDecodable)] but I'm not sure.

Yes, I think that's true as long as both the compiler and importing tool are compiled with the same compiler version.

Given that there's just a couple of data structures that need to be serialized, doing something handwritten might be a viable option though. libserialize has some overhead because it uses leb128 for encoding integers. Since we care more about speed than about file size libserialize might not be a good trade off.

[michaelwoerister]

Can you switch this to use a parking_lot::Mutex? Then we can do a first perf run to see how be the overhead is now.

[wesleywiser]

Sure, I can do that tonight. Are you trying to measure the record overhead or the save results overhead?

I'll need to push a commit to enable whichever option you want to record by default since the entire feature is disabled by default.

[michaelwoerister]

I think we should do both.

[wesleywiser]

Ok. Do you want to do seperate perf runs or just one with the option turned on by default?

[wesleywiser]

Rebased

@bors r=michaelwoerister

[bors]

📌 Commit c0c7ac62b8c570ba581795d0182b720c48092b74 has been approved by michaelwoerister

[bors]

🔒 Merge conflict

This pull request and the master branch diverged in a way that cannot be automatically merged. Please rebase on top of the latest master branch, and let the reviewer approve again.

How do I rebase?

Assuming self is your fork and upstream is this repository, you can resolve the conflict following these steps:

1. git checkout more_profiler_changes (switch to your branch)
2. git fetch upstream master (retrieve the latest master)
3. git rebase upstream/master -p (rebase on top of it)
4. Follow the on-screen instruction to resolve conflicts (check git status if you got lost).
5. git push self more_profiler_changes --force-with-lease (update this PR)

You may also read Git Rebasing to Resolve Conflicts by Drew Blessing for a short tutorial.

Please avoid the "Resolve conflicts" button on GitHub. It uses git merge instead of git rebase which makes the PR commit history more difficult to read.

Sometimes step 4 will complete without asking for resolution. This is usually due to difference between how Cargo.lock conflict is handled during merge and rebase. This is normal, and you should still perform step 5 to update this PR.

Error message

warning: Cannot merge binary files: Cargo.lock (HEAD vs. heads/homu-tmp)
Auto-merging Cargo.lock
CONFLICT (content): Merge conflict in Cargo.lock
Automatic merge failed; fix conflicts and then commit the result.

[wesleywiser]

Rebased

@bors r=michaelwoerister

[bors]

📌 Commit f20ad70 has been approved by michaelwoerister

[bors]

⌛ Testing commit f20ad70 with merge 87a4363...

[bors]

☀️ Test successful - checks-travis, status-appveyor
Approved by: michaelwoerister
Pushing 87a4363 to master...

[jens1o]

What is the perf-graph looking like today?

[Mark-Simulacrum]

This won't affect perf.r-l.o since it isn't run with self-profile enabled.

[michaelwoerister]

I still expect it to be rather slow. Here are some thoughts on how to make it faster: #58372