Skip to content

Commit

Permalink
Auto merge of #10079 - dtolnay-contrib:fetch, r=ehuss
Browse files Browse the repository at this point in the history
Fetch GitHub commits by long hash more efficiently

Closes #10078.

**Tested with the following Cargo.toml:**

```toml
[package]
name = "repro"
version = "0.0.0"
edition = "2021"
publish = false

[dependencies]
cargo = { git = "https://github.com/rust-lang/cargo", rev = "b30694b4d9b29141298870b7993e9aee10940524" }
```

```console
$ rm -rf ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-*
$ time $CARGO generate-lockfile
$ du -shc ~/.cargo/git/db/cargo-* ~/.cargo/git/checkouts/cargo-*
```

Using current cargo from the most recent nightly, the `generate-lockfile` command downloads 69704 git objects in 7.0 seconds, consuming 41 MB on disk.

Using cargo built from this PR by `cargo build --release`, the same command downloads 21481 objects in 2.2 seconds, consuming 17 MB on disk.

Once libgit2 is able to do shallow clones (libgit2/libgit2#3058) this can be even more of a speedup. Using command-line git (which does not use libgit2) and `time git fetch --depth=1 https://github.com/rust-lang/cargo b30694b` indicates that it downloads just 262 objects in 1.1 seconds.
  • Loading branch information
bors committed Jun 24, 2022
2 parents 47620e2 + 7da3c36 commit dbff32b
Showing 1 changed file with 14 additions and 1 deletion.
15 changes: 14 additions & 1 deletion src/cargo/sources/git/utils.rs
Original file line number Diff line number Diff line change
Expand Up @@ -815,8 +815,11 @@ pub fn fetch(
}

GitReference::Rev(rev) => {
let is_github = || Url::parse(url).map_or(false, |url| is_github(&url));
if rev.starts_with("refs/") {
refspecs.push(format!("+{0}:{0}", rev));
} else if is_github() && is_long_hash(rev) {
refspecs.push(format!("+{0}:refs/commit/{0}", rev));
} else {
// We don't know what the rev will point to. To handle this
// situation we fetch all branches and tags, and then we pray
Expand Down Expand Up @@ -1036,7 +1039,7 @@ fn github_up_to_date(
config: &Config,
) -> CargoResult<bool> {
let url = Url::parse(url)?;
if url.host_str() != Some("github.com") {
if !is_github(&url) {
return Ok(false);
}

Expand All @@ -1047,6 +1050,8 @@ fn github_up_to_date(
GitReference::Rev(rev) => {
if rev.starts_with("refs/") {
rev
} else if is_long_hash(rev) {
return Ok(reference.resolve(repo).is_ok());
} else {
debug!("can't use github fast path with `rev = \"{}\"`", rev);
return Ok(false);
Expand Down Expand Up @@ -1089,3 +1094,11 @@ fn github_up_to_date(
handle.perform()?;
Ok(handle.response_code()? == 304)
}

fn is_github(url: &Url) -> bool {
url.host_str() == Some("github.com")
}

fn is_long_hash(rev: &str) -> bool {
rev.len() == 40 && rev.chars().all(|ch| ch.is_ascii_hexdigit())
}

0 comments on commit dbff32b

Please sign in to comment.