Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

builds: Enable lld, optimize linking more, faster cargo-test build #25116

Merged
merged 1 commit into from
Feb 14, 2024

Conversation

def-
Copy link
Contributor

@def- def- commented Feb 9, 2024

Seems to be ~10% faster for cargo-test.

Checklist

  • This PR has adequate test coverage / QA involvement has been duly considered.
  • This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
  • If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
  • If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).
  • This PR includes the following user-facing behavior changes:

@def- def- force-pushed the pr-ci-cargo-test branch 15 times, most recently from 82134dd to e3556e5 Compare February 10, 2024 12:20
@def- def- changed the title cargo-test: Try running without debug symbols in CI builds: Enable lld, optimize linking more, faster cargo-test build Feb 10, 2024
@def- def- marked this pull request as ready for review February 10, 2024 14:34
@def- def- requested review from a team and benesch as code owners February 10, 2024 14:34
"-C",
"link-arg=-Wl,--compress-debug-sections=zlib-gabi",
"-Clink-arg=-Wl,--compress-debug-sections=zlib",
"-Clink-arg=-Wl,-O2",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is going to apply to local developer builds too, no? I don’t think everyone has easy access to lld.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but on Linux only. I think we should set lld everywhere on Linux actually, and devs who are on Linux should be able to easily install it or override RUSTFLAGS. lld is definitely faster than ld, so I don't see a downside.

Copy link
Member

@benesch benesch Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm somewhat reticent to set any compile flags that will make it so that cargo build doesn't work with a standard Rust installation. Historically the main motivation behind this philosophy has been to encourage external contributions from folks who might not want to customize their toolchains to match Materialize's specifications. That's less of an issue now that we've gone cloud native. But recently @ParkMyCar ran into trouble while trying to get the Fivetran destination deployed because Fivetran's hardware is too old for the target CPU/features we specify in .cargo/config. So I'm increasingly pondering whether we should declare that .cargo/config should be the lowest common denominator, and we'd provide a custom profile like dev-fast for folks who want to opt into faster compiles at the cost of installing extra tools. But that's not viable until custom profiles learn to support configuring rustflags.

Anyway, no qualms with trying this out. We can always find a different course if it proves to be a big problem for external contributors or partners who need to build the repository. Just one ask: can you check with folks in #epd-announce and make sure everyone's up for installing lld? We should also update the developer guide with a note that you need to install lld on Linux (or override rustflags).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The experience with getting the Fivetran Destination built on their hardware definitely surprised me. Also not opposed to adding these for now, but I agree the best approach would be having separate profiles like dev-fast and prod-like, with the default being the lowest common denominator, possibly even no special flags.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I asked in #epd-announce but didn't get much feedback either way: https://materializeinc.slack.com/archives/CUXUBJH7B/p1707756913865829

@nrainer-materialize
Copy link
Contributor

Deferring to @benesch here.

@nrainer-materialize nrainer-materialize removed their request for review February 12, 2024 08:16
@def- def- modified the milestone: t Feb 12, 2024
Copy link
Member

@benesch benesch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, assuming no blocking concerns from the folks I just pinged in #epd-announce.

@def- def- merged commit e1691f1 into MaterializeInc:main Feb 14, 2024
72 checks passed
@def- def- deleted the pr-ci-cargo-test branch February 14, 2024 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants