Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linker copies files as a fallback when ref-linking fails #1773

Merged
merged 5 commits into from
Feb 22, 2024

Conversation

snowsignal
Copy link
Contributor

Summary

Fixes #1444.

In situations where the installer fails to perform a reflink, a regular file copy is also attempted, as a fallback. This circumvents issues with linking files across filesystems or volumes.

Test Plan

N/A

@snowsignal snowsignal added enhancement New feature or request installer Related to package installation labels Feb 20, 2024
@konstin konstin added bug Something isn't working and removed enhancement New feature or request labels Feb 20, 2024
@charliermarsh
Copy link
Member

Oh nice, I didn't know this existed.

Copy link
Member

@konstin konstin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what the right strategy to test this is, can we create a ram disk or a virtual filesystem as non-root?

crates/install-wheel-rs/src/linker.rs Outdated Show resolved Hide resolved
crates/install-wheel-rs/src/linker.rs Outdated Show resolved Hide resolved
@charliermarsh
Copy link
Member

I would suggest just stubbing out reflink with a method that always errors.

@konstin
Copy link
Member

konstin commented Feb 21, 2024

How you implement this rust-wise, without unittest.mock?

@charliermarsh
Copy link
Member

@konstin - Oh sorry, I meant for manual testing (while implementing).

…`, properly recurse in copy fallback attempts / create directories as needed, implement fallbacks when overwriting via reflinks fails.
Copy link
Member

@charliermarsh charliermarsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, thanks!

@charliermarsh charliermarsh merged commit da3a7ec into main Feb 22, 2024
7 checks passed
@charliermarsh charliermarsh deleted the jane/reflink/fallback-with-copy branch February 22, 2024 02:57
}

if *attempt == Attempt::Initial {
*attempt = Attempt::Subsequent;
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this different than including it as an else in the Attempt::Initial arm?

E.g.:

diff --git a/crates/install-wheel-rs/src/linker.rs b/crates/install-wheel-rs/src/linker.rs
index 6275994f..69eff653 100644
--- a/crates/install-wheel-rs/src/linker.rs
+++ b/crates/install-wheel-rs/src/linker.rs
@@ -356,6 +356,9 @@ fn clone_recursive(
                     *attempt = Attempt::UseCopyFallback;
                     clone_recursive(site_packages, wheel, entry, attempt)?;
                 }
+            } else {
+                // If the first reflink succeeded, we'll use reflink for the rest of the install.
+                *attempt = Attempt::Subsequent;
             }
         }
         Attempt::Subsequent => {
@@ -390,9 +393,6 @@ fn clone_recursive(
         }
     }

-    if *attempt == Attempt::Initial {
-        *attempt = Attempt::Subsequent;
-    }
     Ok(())
 }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be, because we also want to update the attempt value in the case of an AlreadyExists error that we handle. For example, if we have to overwrite a file, but don't need to fall-back to copying, this else branch would not be hit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working installer Related to package installation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add copy fallback for reflink-based linker
3 participants