Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Automatic UID allocation #3600

Merged
merged 49 commits into from
Nov 29, 2022
Merged
Show file tree
Hide file tree
Changes from 36 commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
836573a
Dynamically allocate UIDs
edolstra Oct 31, 2017
c3e0a68
canonicalisePathMetaData(): Support a UID range
edolstra May 14, 2020
f5fa3de
Run builds in their own cgroup
edolstra May 14, 2020
ca2f64b
Reduce # of UIDs per build to 65536
edolstra May 14, 2020
7bdcf43
Destroy the cgroup prior to building
edolstra May 16, 2020
570c443
Simplify cgroup creation
edolstra May 16, 2020
ba50c3e
Add "uid-range" and "systemd-cgroup" system features
edolstra May 19, 2020
8c4cce5
Fix macOS build
edolstra May 20, 2020
7349f25
Only mount /sys in uid-range builds
edolstra May 20, 2020
2546c63
Merge commit 'f66bbd8c7bb1472facf8917e58e3cd4f6ddfa1b5' into auto-uid…
Ericson2314 Oct 17, 2020
81c5f75
Merge commit '6aa64627c8e431c3b187f7bb44c943d06e39b929' into auto-uid…
Ericson2314 Oct 17, 2020
2c9a8e7
Merge commit '97ffc1e0139e124b7e36b5d1a62b90300f231118' into auto-uid…
Ericson2314 Oct 17, 2020
801e6d9
Merge commit 'd334fd48824b41b57e267cd2926fa9619b7718e3' into auto-uid…
Ericson2314 Oct 17, 2020
d3cfc14
Merge remote-tracking branch 'upstream/master' into auto-uid-allocation
Ericson2314 Oct 17, 2020
e023c98
Merge remote-tracking branch 'upstream/master' into auto-uid-allocation
Ericson2314 Apr 27, 2021
3b82c1a
Merge remote-tracking branch 'upstream/master' into auto-uid-allocation
mkenigs Sep 15, 2021
79152e3
Merge pull request #5212 from mkenigs/auto-uid-allocation
edolstra Sep 15, 2021
dc92b01
Merge remote-tracking branch 'upstream/master' into auto-uid-allocation
Ericson2314 Feb 28, 2022
47dec82
Merge pull request #6181 from obsidiansystems/auto-uid-allocation
thufschmitt Mar 1, 2022
b95facc
Merge remote-tracking branch 'origin/master' into auto-uid-allocation
edolstra Nov 3, 2022
6259fd7
Fix indentation
edolstra Nov 4, 2022
40911d7
Remove stray tab
edolstra Nov 4, 2022
2fde7e0
Split auto UID allocation from cgroups
edolstra Nov 8, 2022
05d2586
Fix build on macOS
edolstra Nov 8, 2022
6c6eff8
Remove the SystemdCgroup feature
edolstra Nov 10, 2022
f423d44
Fix segfault in unprivileged mode
edolstra Nov 17, 2022
f1ab082
createTempDir(): Use std::atomic
edolstra Nov 18, 2022
128910b
Separate cgroup support from auto-uid-allocation
edolstra Nov 18, 2022
20f66c6
Indentation
edolstra Nov 18, 2022
fa68eb3
Get CPU stats from the cgroup
edolstra Nov 18, 2022
e6b71f8
Use cgroup.kill to quickly kill cgroups
edolstra Nov 18, 2022
f538ee4
Rename derivedPathsWithHintsToJSON -> builtPathsToJSON
edolstra Nov 21, 2022
300753d
nix build --json: Include build statistics
edolstra Nov 21, 2022
ec45f4b
Fix indentation
edolstra Nov 21, 2022
653b32a
Merge remote-tracking branch 'origin/master' into auto-uid-allocation
edolstra Nov 21, 2022
82d5cf2
Fix macOS build
edolstra Nov 21, 2022
9d17ce0
AutoUserLock: If sandboxing is disabled, use the build users group
edolstra Nov 21, 2022
c776dfb
Use hex for startId
edolstra Nov 21, 2022
b37c2d8
Always call setgroups()
edolstra Nov 22, 2022
3d23b9d
SimpleUserLock::getSupplementaryGIDs(): Filter out main gid
edolstra Nov 22, 2022
6292d56
Merge remote-tracking branch 'origin/master' into auto-uid-allocation
edolstra Nov 23, 2022
989fc8a
Add release notes
edolstra Nov 23, 2022
2aa3f2e
Include UID in hex
edolstra Nov 23, 2022
f1b5c68
Add tests for auto-uid-allocation, uid-range and cgroups
edolstra Nov 27, 2022
fc14585
Fix evaluation
edolstra Nov 27, 2022
ff12d1c
Check that auto-allocated UIDs don't clash with existing accounts
edolstra Nov 28, 2022
67bcb99
Add a setting for enabling cgroups
edolstra Nov 28, 2022
7dd3e1f
Add example
edolstra Nov 28, 2022
4f762e2
Restore ownership of / for non-uid-range builds
edolstra Nov 29, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 17 additions & 14 deletions src/libcmd/installables.cc
Original file line number Diff line number Diff line change
Expand Up @@ -844,20 +844,20 @@ std::shared_ptr<Installable> SourceExprCommand::parseInstallable(
return installables.front();
}

BuiltPaths Installable::build(
std::vector<BuiltPathWithResult> Installable::build(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
const std::vector<std::shared_ptr<Installable>> & installables,
BuildMode bMode)
{
BuiltPaths res;
for (auto & [_, builtPath] : build2(evalStore, store, mode, installables, bMode))
res.push_back(builtPath);
std::vector<BuiltPathWithResult> res;
for (auto & [_, builtPathWithResult] : build2(evalStore, store, mode, installables, bMode))
res.push_back(builtPathWithResult);
return res;
}

std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> Installable::build2(
std::vector<std::pair<std::shared_ptr<Installable>, BuiltPathWithResult>> Installable::build2(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
Expand All @@ -877,7 +877,7 @@ std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> Installable::bui
}
}

std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> res;
std::vector<std::pair<std::shared_ptr<Installable>, BuiltPathWithResult>> res;

switch (mode) {

Expand Down Expand Up @@ -918,10 +918,10 @@ std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> Installable::bui
output, *drvOutput->second);
}
}
res.push_back({installable, BuiltPath::Built { bfd.drvPath, outputs }});
res.push_back({installable, {.path = BuiltPath::Built { bfd.drvPath, outputs }}});
},
[&](const DerivedPath::Opaque & bo) {
res.push_back({installable, BuiltPath::Opaque { bo.path }});
res.push_back({installable, {.path = BuiltPath::Opaque { bo.path }}});
},
}, path.raw());
}
Expand All @@ -931,7 +931,7 @@ std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> Installable::bui

case Realise::Outputs: {
if (settings.printMissing)
printMissing(store, pathsToBuild, lvlInfo);
printMissing(store, pathsToBuild, lvlInfo);

for (auto & buildResult : store->buildPathsWithResults(pathsToBuild, bMode, evalStore)) {
if (!buildResult.success())
Expand All @@ -943,10 +943,10 @@ std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> Installable::bui
std::map<std::string, StorePath> outputs;
for (auto & path : buildResult.builtOutputs)
outputs.emplace(path.first.outputName, path.second.outPath);
res.push_back({installable, BuiltPath::Built { bfd.drvPath, outputs }});
res.push_back({installable, {.path = BuiltPath::Built { bfd.drvPath, outputs }, .result = buildResult}});
},
[&](const DerivedPath::Opaque & bo) {
res.push_back({installable, BuiltPath::Opaque { bo.path }});
res.push_back({installable, {.path = BuiltPath::Opaque { bo.path }, .result = buildResult}});
},
}, buildResult.path.raw());
}
Expand All @@ -969,9 +969,12 @@ BuiltPaths Installable::toBuiltPaths(
OperateOn operateOn,
const std::vector<std::shared_ptr<Installable>> & installables)
{
if (operateOn == OperateOn::Output)
return Installable::build(evalStore, store, mode, installables);
else {
if (operateOn == OperateOn::Output) {
BuiltPaths res;
for (auto & p : Installable::build(evalStore, store, mode, installables))
res.push_back(p.path);
return res;
} else {
if (mode == Realise::Nothing)
settings.readOnlyMode = true;

Expand Down
11 changes: 9 additions & 2 deletions src/libcmd/installables.hh
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#include "eval.hh"
#include "store-api.hh"
#include "flake/flake.hh"
#include "build-result.hh"

#include <optional>

Expand Down Expand Up @@ -51,6 +52,12 @@ enum class OperateOn {
Derivation
};

struct BuiltPathWithResult
{
BuiltPath path;
std::optional<BuildResult> result;
};

struct Installable
{
virtual ~Installable() { }
Expand Down Expand Up @@ -91,14 +98,14 @@ struct Installable
return FlakeRef::fromAttrs({{"type","indirect"}, {"id", "nixpkgs"}});
}

static BuiltPaths build(
static std::vector<BuiltPathWithResult> build(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
const std::vector<std::shared_ptr<Installable>> & installables,
BuildMode bMode = bmNormal);

static std::vector<std::pair<std::shared_ptr<Installable>, BuiltPath>> build2(
static std::vector<std::pair<std::shared_ptr<Installable>, BuiltPathWithResult>> build2(
ref<Store> evalStore,
ref<Store> store,
Realise mode,
Expand Down
5 changes: 4 additions & 1 deletion src/libstore/build-result.hh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@

#include <string>
#include <chrono>

#include <optional>

namespace nix {

Expand Down Expand Up @@ -78,6 +78,9 @@ struct BuildResult
was repeated). */
time_t startTime = 0, stopTime = 0;

/* User and system CPU time the build took. */
std::optional<std::chrono::microseconds> cpuUser, cpuSystem;

bool success()
{
return status == Built || status == Substituted || status == AlreadyValid || status == ResolvesToAlreadyValid;
Expand Down
8 changes: 8 additions & 0 deletions src/libstore/build/derivation-goal.cc
Original file line number Diff line number Diff line change
Expand Up @@ -887,6 +887,14 @@ void DerivationGoal::buildDone()

cleanupPostChildKill();

if (buildResult.cpuUser && buildResult.cpuSystem) {
debug("builder for '%s' terminated with status %d, user CPU %.3fs, system CPU %.3fs",
worker.store.printStorePath(drvPath),
status,
((double) buildResult.cpuUser->count()) / 1000000,
((double) buildResult.cpuSystem->count()) / 1000000);
}

bool diskFull = false;

try {
Expand Down
Loading