Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include Git branch in CircleCI cache keys and bump key version #476

Merged
merged 2 commits into from
May 10, 2024

Conversation

benjamn
Copy link
Member

@benjamn benjamn commented May 8, 2024

This should resolve an issue affecting the 2.7.5 release, where an old federation-2/target/ cache was restored from an unrelated PR branch build (because Cargo.lock was the same), causing the compiled supergraph binary to call op_composition_result with a string rather than an object in some cases, triggering a panic on the Rust side.

I would have liked to disable this caching completely when publishing an official release, to prioritize correctness over build speed, but it doesn't seem like conditional caching is supported by CircleCI: https://circleci.com/docs/caching/#clearing-cache

We could potentially run cargo clean conditionally for release builds, after restore_cache (thereby throwing away the cache), but that doesn't seem ideal. Open to ideas!

@benjamn benjamn requested a review from dariuszkuc May 8, 2024 18:43
@benjamn benjamn self-assigned this May 8, 2024
@benjamn benjamn requested a review from a team as a code owner May 8, 2024 18:43
@benjamn
Copy link
Member Author

benjamn commented May 8, 2024

On @dariuszkuc's suggestion, I was able to SSH into a CircleCI runner and reproduce the error with slightly more information:

$ RUST_BACKTRACE=1 ../target/aarch64-apple-darwin/release/supergraph compose repro.yaml 
thread 'main' panicked at harmonizer/src/lib.rs:117:10:
unable to invoke composition in JavaScript runtime: TypeError: Error parsing args at position 0: serde_v8 error: invalid type; expected: object, got: string
    at done (<init>:12:43)
    at do_compose:8:5

This is why I think done (which calls Deno.core.ops["op_composition_result"]) must have been called with a string, though I still don't have a complete theory of how that could happen.

@@ -418,14 +418,14 @@ commands:
steps:
- restore_cache:
keys:
- rust-target-v2-<< parameters.platform >>-{{ checksum "Cargo.lock" }}
- &target-cache-key rust-target-v3-<< parameters.platform >>-{{ .Branch }}-{{ checksum "Cargo.lock" }}

- run:
command: cargo xtask << parameters.command >> << parameters.options >>
working_directory: << parameters.working_directory >>

- save_cache:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we just wrap this in condition to save cache from main branch?

     - when:
          condition:
            matches: // or equal
              pattern: "/main/"
              value: << get target branch >> 
          steps:
            - save_cache:
                key: *target-cache-key
                paths:
                  - target/

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

*not a circle ci expert so unsure if it is even possible but I think we might be able to get some condition to match the main branch OR something like tag value

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I'm no expert either, so I decided to keep it simple and disable caching instead of having to wait for the next release to find out if we did it correctly.

Copy link
Member

@abernix abernix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I fully support the change, though I feel like caching is such an under-needed ... need ... on this repository that we could and should just remove it completely. This repository's velocity is only meant to trend down, and is mostly just a fresh branch + release workflow anyhow. The taxes of compiling v8 twice vs once are high, but the rewards of the safest releases are higher.

@@ -418,14 +418,14 @@ commands:
steps:
- restore_cache:
keys:
- rust-target-v2-<< parameters.platform >>-{{ checksum "Cargo.lock" }}
- &target-cache-key rust-target-v3-<< parameters.platform >>-{{ .Branch }}-{{ checksum "Cargo.lock" }}
Copy link
Member

@abernix abernix May 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Btw, it's relatively important that you DO bump to v3 here (as you did) because without it, CircleCI does this "tiered disposal" of arguments on the cache key until it gets a hit, which means without the bump, you might still get a previous cache segment based on just platform.

This should resolve an issue affecting the 2.7.5 release, where an old
federation-2/target/ cache was restored from an unrelated PR branch
build (because Cargo.lock was the same), causing the compiled supergraph
binary to call op_composition_result with a string rather than an object
in some cases, triggering a panic on the Rust side.

I would have liked to disable this caching completely when publishing an
official release, to prioritize correctness over build speed, but it
doesn't seem like conditional caching is supported by CircleCI:
https://circleci.com/docs/caching/#clearing-cache

We could potentially run `cargo clean` conditionally for release builds,
after restore_cache (thereby throwing away the cache), but that doesn't
seem ideal.
@benjamn benjamn force-pushed the benjamn/include-branch-in-circleci-cache-keys branch from f32a30a to eb4feff Compare May 10, 2024 14:24
@benjamn
Copy link
Member Author

benjamn commented May 10, 2024

@abernix @dariuszkuc After sleeping on it, I've come around to disabling the caching completely, to prioritize correctness.

@benjamn benjamn merged commit 0119566 into main May 10, 2024
9 checks passed
@benjamn benjamn deleted the benjamn/include-branch-in-circleci-cache-keys branch May 10, 2024 14:35
dariuszkuc pushed a commit to apollographql/federation that referenced this pull request May 10, 2024
#2999)

The
[harmonizer](https://github.com/apollographql/federation-rs/tree/main/federation-2/harmonizer)
package depends on this repository, and its release was recently broken
by a CircleCI caching bug that was fixed in
apollographql/federation-rs#476. In order to cut
a new release over there, we need to re-release the npm packages in this
repository with a new patch version (2.7.7).

While I was at it, I ran `npm audit fix` to resolve 4 moderate
vulnerability reports in various transitive dependencies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants