No way to run action with a modified version of default_shell_env

[Reflexe]

First of all, I started writing this issue as a bug since a recent commit broke my code but then I thought it could be also a feature request. If you think it is more of a FR than a bug, please tell me and I will edit it to match the correct format.

Description of the problem / feature request:

Since 0f5679e, ctx.configuration.default_shell_env would return an empty dictionary on Linux; Bazel no longer inherits environment variables from the compiling host in the analysis phase but only in execution time (according to @ulfjack 's comments on BazelConfiguration.java#135). However, that make it impossible to append environment variable to the default shell environment.

I can't think of a good way to solve this problem without a magic value that get changed on execution time (similar to cmake's generator expressions). Another way is to maybe transparently return a unique UncookedDefaultEnv from ctx.configuration.default_shell_env that will behave like a regular dictionary and would transparently forward null keys to the action, to be replaced in execution time.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

example.bzl:

def _impl(ctx):
    # Add 'TheAnswer' to the default shell environment.
    environment_vars = ctx.configuration.default_shell_env
    environment_vars['TheAnswer'] = '42'

    ctx.actions.run_shell(
        outputs = [ctx.outputs.dummy],
        command='env && touch dummy',
        env = environment_vars,
    )

example = rule(
    implementation=_impl, 
    outputs = {
        'dummy':'dummy'
    }
)

BUILD:

load(':example.bzl', 'example')
example(name='example')

Example output with bazel 0.11.1

...
PATH=/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/bin/site_perl:/usr/bin/vendor_perl
...

Example output for the current master:

PWD=/home/reflexe/.cache/bazel/_bazel_reflexe/95fc1ed1afdc0650f9a0db1c7a4b7c34/sandbox/9014715435859486979/execroot/__main__
TMPDIR=/tmp
TheAnswer=42
SHLVL=1
_=/usr/bin/env

Result: you should see that the output of env does not contain the default PATH.

What operating system are you running Bazel on?

Linux - ArchLinux

What's the output of bazel info release?

development version

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

$ git clone github://bazelbuild/bazel
$ bazel build //src:bazel

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/bazelbuild/bazel
bffa2db380cb3ca2fd9262ac5a45d02518376e03
bffa2db380cb3ca2fd9262ac5a45d02518376e03

And I need to say that every time, thank you very much for this amazing thing called Bazel. It feels so right to use it; and the best and somehow the simplest build system I ever had to use. Just thanks :)

[jmmv]

@ulfjack I see you did the original change referenced in this bug and this is assigned to you. Any thoughts on what the right behavior should be here?

[ulfjack]

The problem is that we need to merge the environment set on the command-line with --action_env and --test_env with the environment specified by the Skylark rule in some way. Which of these sources should have precedence over the other is a critical question here. If we allow whatever, then rules will almost certainly end up inconsistent wrt. environment handling, which makes it a pain for users.

My personal preference is that the --action_env flag overrides any per-rule defaults, possibly with a more fine-grained action_env spec, e.g., based on mnemonics like --strategy. Maybe also allow rules to error out if the env is conflicting with rule-specific requirements. The built-in rules currently do their own thing, although most of them don't set any additional env variables, I think.

[ulfjack]

Note also that default_shell_env is a misnomer. We probably need to enforce that --action_env is taken into account during Skylark action creation, and not make it optional.

[Globegitter]

We have been facing similar issues and it would be great if it was possible to somehow merge env variables defined by the rule, as well as variables passed in via --action_env and also allow a more fine-grained --action_env spec.

Somewhat related: #7364 & #6011

[flaub]

This is the exact problem I'm running into as well. The ctx.configuration.default_shell_env is empty, but I need the host's default shell environment vars so that I can append my own for a particular rule.

[tillahoffmann]

This workaround doesn't properly inherit values, but it allows for setting defaults that can be copied from the host environment.

# .env.bazelrc: this file is like a .env file but in bazel CLI style.
build --action_env FOO=bar --action_env BAR=foo

# .bazelrc: the configuration simply imports the environment if available.
try-import %workspace%/.env.bazelrc

# example.bzl: example implementation showing that environment variables are available.
def _impl(ctx):
    print(ctx.configuration.default_shell_env)

example = rule(
    implementation=_impl,
)

# BUILD: example invocation.
load("//:example.bzl", "example")
example(name="example")

bazel build //:example
# DEBUG: /path/to/BUILD: {"FOO": "bar", "BAR": "foo"}

This is of course not as nice as being able to use the host environment directly, but copying the relevant environment variables to .env.bazelrc isn't too cumbersome.

[burdiyan]

Got hit by this too.

I'm using --incompatible_strict_action_env because normally I don't want any host env vars to be leaked into the build actions. But for one specific rule I want to inherit a variable from host. So I configure it as --action_env=MYVAR in the .bazelrc file. I was expecting to get it inside ctx.configuration.default_shell_env and then pass it manually to my custom build action, but it is empty. Although if I invoke Bazel and manually pass the actual value like --actions_env=MYVAR=$MYVAR" then it works. But it can't be configured within .bazelrc this way, so I have to use a wrapper script, which is annoying.

Can someone confirm if this is actually a bug or an expected behavior?

[jesses-canva]

Is this fixed by #19317?

[fmeum]

Is this fixed by #19317?

Only partially. You can now choose to inherit the --action_env via use_default_shell_env and still set environment variables via env, but it's not possible to modify the inherited values short of fully replacing them.