-
-
Notifications
You must be signed in to change notification settings - Fork 157
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[RFC 0131] Enforce the usage of SRI hashes in Nixpkgs #131
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,50 @@ | ||
--- | ||
feature: sri-hashes | ||
start-date: 2022-07-25 | ||
author: Winter | ||
co-authors: (find a buddy later to help out with the RFC) | ||
shepherd-team: (names, to be nominated and accepted by RFC steering committee) | ||
shepherd-leader: (name to be appointed by RFC steering committee) | ||
related-issues: (will contain links to implementation PRs) | ||
--- | ||
|
||
# Summary | ||
[summary]: #summary | ||
|
||
Use SRI hashes and the `hash` parameter for fetchers. | ||
|
||
# Motivation | ||
[motivation]: #motivation | ||
|
||
Currently in Nixpkgs, the hash formats that are used in parameters to fetchers range from SRI and base-32 (the most common ones) to hex. In addition to this, packages use the algorithm-specific arguments (`sha256`, `sha512`, etc.) or the generic `hash` argument at random. This creates inconsistencies that we'd rather not have. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. What is SRI in this context? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Subresource Integrity, whose hashes look like There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Why? This is not a motivation, just a claim. Using the different hash encodings genuinely has value, for example I frequently just copy over the base16 hash from a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think a link to what SRI is in the RFC would be good to have. I also didn't know what SRI was. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I agree with this point, but at the same time most people use either the SRI hashes given by Nix's hash mismatch errors (which I now realize is only in 2.4), or convert those to base-32 (the default in 2.3 and below). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @vojta001 That's why these interfaces tend to have an
Well that's actually a very good point: There is no stable interface for converting hashes into the SRI format, because There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
no need to convert the hash to base64 SRI hashes only require a prefix: {
sha256 = "1cc9110705165dc09dd09974dd7c0b6709c9351d6b6b1cef5a711055f891dd0f";
sha256 = "sha256-1cc9110705165dc09dd09974dd7c0b6709c9351d6b6b1cef5a711055f891dd0f";
hash = "sha256-1cc9110705165dc09dd09974dd7c0b6709c9351d6b6b1cef5a711055f891dd0f";
} base16 hashes are found with
sha256 is de-facto standard in nixpkgs, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
MD5 was de facto standard no more than a decade ago. Also, this was not about "good enough algorithms", but about a better format for the expression - namely, There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
@winterqt could you add something more specific here, considering this is the whole motivation for the RFC? The discussion above has brought up some more points:
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @milahu: conversion would be needed because SRI is currently only accepting base64. However, it would be relatively easy to change: NixOS/nixpkgs#199660 |
||
|
||
# Detailed design | ||
[design]: #detailed-design | ||
|
||
1. Require all new packages to use SRI hashes + `hash` when possible | ||
2. Do a treewide migration for the majority of cases (single fetcher in `src`, using a fetcher that supports `hash` and SRI hashes (`fetchFromGitHub`, `fetchurl`, etc.)) to `hash` + SRI hashes | ||
|
||
# Drawbacks | ||
[drawbacks]: #drawbacks | ||
|
||
None at this time. | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There's (anecdotically) still a significant number of Nix 2.3 users who contributing to nixpkgs would be made unnecessarily harder for. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh no, another ten years of deprecation window... There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You joke, but I seriously think this would be a sensible thing to do. (Also, things can be deprecated without the intent to eventually removing them, see Java which did not remove deprecated APIs until Java 9?) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Overall we have developed a strange culture of removing stuff unnecessarily in nixpkgs, I feel like. In this case, we shouldn't remove support for these formats from Nix anyways (to continue to be able to evaluate older nixpkgs revisions (the ability to reproduce ancient stuff truely is a strength of Nix)) and for existing APIs in nixpkgs it's laughably simple to keep supporting it (you don't need to change anything, in fact!). For all I care, stop supporting the There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The text does not talk about removal, I suppose, but about enforcement. Older expressions can be rendered by newer versions, whereas new expressions should be written in newer versions of Nix. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So how does something like this sound: we strongly encourage using SRI hashes and I'm not sure about that last reason, but I also want to provide more than one example of a reason. I'm also not sure about how to handle converting old packages (with the same criteria of having no justifiable reasoning), if at all. Thoughts welcome. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Old packages can be converted by anyone interested ion converting them. I have started a preliminar work a long time ago :P Further I believe this should be decided on a case by case basis. Maybe allowing old Nix tools is acceptable, whereas upstream hashes is not. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Not sure how apt this comparison is, as there's nothing fundamentally different about a sha256 hash in SRI format or one in base16 format.
Stale hashes is not something caused by a particular hash format. It happens because expressions bitrot and we'll have stale hashes in SRI format before long. Doing a mass conversion of hashes may turn up stale hashes, but doesn't have to since you would likely just convert them without refetching, since that is much quicker. If you are concerned about stale hashes, I'd suggest trying to refetch fixed output derivations in nixpkgs with cache disabled. I believe this has been done quite a few times over the years and it should be easy to turn up useful tools for this.
Newer Nix versions already default to this, so there is already a lot of encouragement… There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "A lot of encouragement" that ends with "my version is too old" or "this is mere personal preferences, don't force them on me"? Wow cool. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The version is probably too old for the format to be convenient to produce but can evaluate the expressions with either hash format. For «mere personal preferences» argument new Nix making use of SRI format easier than non-SRI is an encouragement, as submitters will typically use whatever is the easiest and changing the format is pointless churn so the initial hash format will stay. |
||
|
||
# Alternatives | ||
[alternatives]: #alternatives | ||
|
||
- Do nothing: further inconsistencies will continue to be introduced | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think there are a number of alternatives. Different rules can be considered. For each of the following formats it could be decided to allow/disallow/preferred:
I've gathered there are a number of ways these rules can be 'enforced':
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
I'm mentioning this option, as at the moment PRs are being reviewed where the reviewers suggest their own 'opinion' on the matter; even though there is no real rule. One outcome of this RFC could be to just let it be. |
||
|
||
# Unresolved questions | ||
[unresolved]: #unresolved-questions | ||
|
||
- How can we do the treewide migration efficently? | ||
- Text replacement for the majority of cases, with an AST-based solution for others? | ||
- Do we want to eventually deprecate the use of `sha256`, `sha512`, etc. in fetchers that support SRI hashes, leading to their eventual removal? | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We can deprecate them without removing them. |
||
- This would require updating every script that uses these arguments, and would cause breakage for any usage of them out-of-tree; we'd need to weigh the benefits of this | ||
- Is `hash` the correct name to use for the argument? | ||
- Nix builtins use `narHash` | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't care about the name, but built-in fetchers and nixpkgs fetchers should be consistent.
Comment on lines
+44
to
+45
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't see much value in renaming hash. A lot of work which we can safe ourselves. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See NixOS/nixpkgs#79987 (comment) for context, I added it here just to include it since the question was raised (should probably cite that discussion, though). There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think we are not talking about narHashes here, so this should be removed. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. NAR hashes are necessary for any cases where the result is more than a single file, e.g. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Actually, as we have flat hashes for normal files and NAR hashes over arbitrary trees (or normal files), I'd say it makes sense to call them differently. At least if I don't consider any backwards compatibility questions for now. For example, keep There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this something that could be done under the hood in the fetchers? It seems confusing for new contributors to have to think about whether to use a flat hash or a NAR hash. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes. We already do this with |
||
|
||
# Future work | ||
[future]: #future-work | ||
|
||
Currently none. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"SRI hash" needs a reference link with a definition and globally this RFC needs some more context to be explained.
FWIW, since I became aware that this kind of hash was preferred, I have tried to use them more systematically, but I'm annoyed by
nix-prefetch-url
not generating this type of hash (at least all the way to Nix 2.12) and not knowing how to convert a hash generated bynix-prefetch-url
into an SRI hash.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
FYI, for conversion I'm using
EDIT: or you could use
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! But another thing that is becoming annoying to me lately is to be forced to activate the
nix-command
experimental feature for more and more basic things that I cannot avoid.Another solution that I've found meanwhile was
openssl dgst -sha256 -binary /path/to/file | openssl base64 -A
, but it's not as convenient.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would rather view it as a missing feature of
nix-hash
that needs to be implemented to get this RFC merged. In other words, we should haveand
Update: I just created NixOS/nix#7690. Please take a look.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! And thanks to you, I also learned that, in the meantime, I can use
nix --extra-experimental-features nix-command file to-sri --type sha256 ...
. I didn't realize that I don't need to activate the experimental feature globally, and this is already better than nothing.