-
-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ValidPathInfo: make ca field a proper datatype #3649
Merged
edolstra
merged 34 commits into
NixOS:master
from
obsidiansystems:validPathInfo-ca-proper-datatype
Jun 22, 2020
Merged
Changes from 44 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
bcde545
Flip dependency so store-api.hh includes derivations.hh
Ericson2314 f4b89e1
Merge branch 'no-stringly-typed-derivation-output' of github.com:Eric…
meditans 0cb67ec
Merge branch 'derivation-header-include-order' of github.com:Ericson2…
meditans 0e9438b
Create new file-hash files
meditans da39092
WIP
meditans 754c910
WIP more progress
Ericson2314 efcd30d
WIP
Ericson2314 1b6461f
Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-…
Ericson2314 a5d820a
Change parseCa(Opt) to parseContentAddress(Opt)
meditans 25e6181
Apply suggestions from code review
Ericson2314 a33270c
Clean up `ValidPathInfo::isContentAddressed` with `std::visit`
Ericson2314 390bf64
WIP
meditans 343c20a
WIP Completed implementation
meditans fd2eb41
Move file-hash to content-address
meditans a5cdf18
Add assertions for SHA256 in fixed case
meditans 78f137e
Validate text version instead, throw Errors
meditans 75d2581
Typo
meditans 39ba87b
Missing `#include <cassert>` in `lru-cache.hh`
Ericson2314 fecff16
Merge remote-tracking branch 'obsidian/missing-include-0' into validP…
Ericson2314 3c78ac3
Merge remote-tracking branch 'obsidian/no-hash-type-unknown' into val…
Ericson2314 53bc8ff
No C++ designated initializers yet with Clang 7
Ericson2314 574d546
Make sure info.ca tag bit is set in `nix add-to-store`
Ericson2314 ed86acf
Use some `std::optional::has_value` for clarity
Ericson2314 2041499
Flip boolean
Ericson2314 744ce9c
Merge branch 'master' of github.com:NixOS/nix into validPathInfo-ca-p…
Ericson2314 94ddea9
Use `readString` rather than `>> temporary`
Ericson2314 d614166
Fix condition error and make test suite pass
meditans 669c399
Merge branch 'no-hash-type-unknown' into validPathInfo-temp
Ericson2314 3f8dcfe
Merge branch 'validPathInfo-temp' into validPathInfo-ca-proper-datatype
Ericson2314 3fc58a9
Remove some `Base::` that crept in
Ericson2314 2f0e395
Merge remote-tracking branch 'me/no-stringly-typed-derivation-output'…
Ericson2314 911fc88
More designated initializers
Ericson2314 29691ed
Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-…
Ericson2314 e288c09
Merge remote-tracking branch 'upstream/master' into validPathInfo-ca-…
Ericson2314 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,79 @@ | ||
#include "content-address.hh" | ||
|
||
namespace nix { | ||
|
||
std::string FixedOutputHash::printMethodAlgo() const { | ||
return makeFileIngestionPrefix(method) + printHashType(*hash.type); | ||
} | ||
|
||
std::string makeFileIngestionPrefix(const FileIngestionMethod m) { | ||
switch (m) { | ||
case FileIngestionMethod::Flat: | ||
return ""; | ||
case FileIngestionMethod::Recursive: | ||
return "r:"; | ||
default: | ||
throw Error("impossible, caught both cases"); | ||
} | ||
} | ||
|
||
std::string makeFixedOutputCA(FileIngestionMethod method, const Hash & hash) | ||
{ | ||
return "fixed:" | ||
+ makeFileIngestionPrefix(method) | ||
+ hash.to_string(Base32, true); | ||
} | ||
|
||
// FIXME Put this somewhere? | ||
template<class... Ts> struct overloaded : Ts... { using Ts::operator()...; }; | ||
template<class... Ts> overloaded(Ts...) -> overloaded<Ts...>; | ||
|
||
std::string renderContentAddress(ContentAddress ca) { | ||
return std::visit(overloaded { | ||
[](TextHash th) { | ||
return "text:" + th.hash.to_string(Base32, true); | ||
}, | ||
[](FixedOutputHash fsh) { | ||
return makeFixedOutputCA(fsh.method, fsh.hash); | ||
} | ||
}, ca); | ||
} | ||
|
||
ContentAddress parseContentAddress(std::string_view rawCa) { | ||
auto prefixSeparator = rawCa.find(':'); | ||
if (prefixSeparator != string::npos) { | ||
auto prefix = string(rawCa, 0, prefixSeparator); | ||
if (prefix == "text") { | ||
auto hashTypeAndHash = rawCa.substr(prefixSeparator+1, string::npos); | ||
Hash hash = Hash(string(hashTypeAndHash)); | ||
if (*hash.type != htSHA256) { | ||
throw Error("parseContentAddress: the text hash should have type SHA256"); | ||
} | ||
return TextHash { hash }; | ||
} else if (prefix == "fixed") { | ||
// This has to be an inverse of makeFixedOutputCA | ||
auto methodAndHash = rawCa.substr(prefixSeparator+1, string::npos); | ||
if (methodAndHash.substr(0,2) == "r:") { | ||
std::string_view hashRaw = methodAndHash.substr(2,string::npos); | ||
return FixedOutputHash { FileIngestionMethod::Recursive, Hash(string(hashRaw)) }; | ||
} else { | ||
std::string_view hashRaw = methodAndHash; | ||
return FixedOutputHash { FileIngestionMethod::Flat, Hash(string(hashRaw)) }; | ||
} | ||
} else { | ||
throw Error("parseContentAddress: format not recognized; has to be text or fixed"); | ||
} | ||
} else { | ||
throw Error("Not a content address because it lacks an appropriate prefix"); | ||
} | ||
}; | ||
|
||
std::optional<ContentAddress> parseContentAddressOpt(std::string_view rawCaOpt) { | ||
return rawCaOpt == "" ? std::optional<ContentAddress> {} : parseContentAddress(rawCaOpt); | ||
}; | ||
|
||
std::string renderContentAddress(std::optional<ContentAddress> ca) { | ||
return ca ? renderContentAddress(*ca) : ""; | ||
} | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
#pragma once | ||
|
||
#include <variant> | ||
#include "hash.hh" | ||
|
||
namespace nix { | ||
|
||
enum struct FileIngestionMethod : uint8_t { | ||
Flat = false, | ||
Recursive = true | ||
}; | ||
|
||
struct TextHash { | ||
Hash hash; | ||
}; | ||
|
||
/// Pair of a hash, and how the file system was ingested | ||
struct FixedOutputHash { | ||
FileIngestionMethod method; | ||
Hash hash; | ||
std::string printMethodAlgo() const; | ||
}; | ||
|
||
/* | ||
We've accumulated several types of content-addressed paths over the years; | ||
fixed-output derivations support multiple hash algorithms and serialisation | ||
methods (flat file vs NAR). Thus, ‘ca’ has one of the following forms: | ||
|
||
* ‘text:sha256:<sha256 hash of file contents>’: For paths | ||
computed by makeTextPath() / addTextToStore(). | ||
|
||
* ‘fixed:<r?>:<ht>:<h>’: For paths computed by | ||
makeFixedOutputPath() / addToStore(). | ||
*/ | ||
typedef std::variant< | ||
TextHash, // for paths computed by makeTextPath() / addTextToStore | ||
FixedOutputHash // for path computed by makeFixedOutputPath | ||
> ContentAddress; | ||
|
||
/* Compute the prefix to the hash algorithm which indicates how the files were | ||
ingested. */ | ||
std::string makeFileIngestionPrefix(const FileIngestionMethod m); | ||
|
||
/* Compute the content-addressability assertion (ValidPathInfo::ca) | ||
for paths created by makeFixedOutputPath() / addToStore(). */ | ||
std::string makeFixedOutputCA(FileIngestionMethod method, const Hash & hash); | ||
|
||
std::string renderContentAddress(ContentAddress ca); | ||
|
||
std::string renderContentAddress(std::optional<ContentAddress> ca); | ||
|
||
ContentAddress parseContentAddress(std::string_view rawCa); | ||
|
||
std::optional<ContentAddress> parseContentAddressOpt(std::string_view rawCaOpt); | ||
|
||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I went with
FixedOutputHash
for this PR, now that it is not just used in derivations.