Implement CARGO_CONFIG_PATH config file search override

[jsgf]

This implements the CARGO_CONFIG_PATH environment variable, which defines
in which order cargo searches for config files. This overrides the
default "search parent directories" order.

Fixes #7887

[rust-highfive]

r? @Eh2406

(rust_highfive has picked a reviewer for you, use r? to override)

[Eh2406]

This is a significant change and will require the team to talk it over, if not an RFC. I will add it to our agenda.

[joshtriplett]

We discussed this in the Cargo meeting, and came to the following conclusions:

• This generally looked reasonable to us.
• We would like some clear documentation on this.
• This interacts with other proposals to change config-file searching, such as "don't search upwards outside the workspace". We'll need to discuss the interaction of such changes.

[joshtriplett]

Personal opinions:

• I'd support merging this, as soon as we have documentation for it.
• I'd also support changing cargo to stop searching parent directories above the workspace for config files. (For instance, a build in /tmp/foo should never look at /tmp/.cargo (which anyone could create).

[jsgf]

@joshtriplett Ping. I've updated the documentation.

[bors]

☔ The latest upstream changes (presumably #8124) made this pull request unmergeable. Please resolve the merge conflicts.

[jsgf]

@joshtriplett Ping?

[ehuss]

@jsgf can you provide more details on your specific use case and motivations? I see that the issue mentions excluding /tmp/.cargo/config, is that the only concern? Is that something that has happened in practice? When would someone use this? Are there other scenarios besides the "don't look above the current workspace" that this would be used to address?

I'm not opposed to this change, but I'm concerned that it is not very ergonomic. It seems like it would only be useful in the context of another build system (presumably you want to use this with Buck?). There are a number of other issues around Cargo's config discovery (#2930, #6728, #7621, #7723, #8148), and I think it would be good to avoid making the search rules too complicated.

I think it would be good to explore what the alternatives are. If the main motivation is to restrict configs to the current workspace, maybe that would be a specific option? How would this interact with a possible enhancement where Cargo honors per-package config files? Maybe more options should be in Cargo.toml to avoid needing configs in the first place? Should Cargo check the file ownership of the config file, and avoid loading it if it is from a different user?

Presumably, also, on Windows the path would be separated by ;, so perhaps it should use std::env::split_paths.

I also wonder if it makes sense to have explicit filenames in the search list instead of always inferring .cargo/config. Like CARGO_CONFIG_PATH=/path/to/my_config.toml.

[jsgf]

@ehuss

Yes, I want this in a somewhat Buck-related context. I have a tool which can take a Cargo.toml specifying a set of third-party dependencies we want to use, and it vendors them and auto-generates Buck build rules for them.

This tool uses cargo vendor and cargo metadata, and wants to run them with extremely controlled environments. Since any Rust developer can use this tool to add new third-party crates to import, I need to make sure they'll always generate identical outputs regardless of any environmental factors, like what the current working directory is, what other Cargo config they have scattered around, etc.

While Cargo's current behaviour of "search up through .. and then $CARGO_HOME/config` is easy to describe and generally useful for individual developers, it's extremely non-hermetic. It means that it's possible to accidentally override Cargo's precise behaviour through both local config and latent config from current directory. (True story: I spent a couple of hours trying to debug a problem where I was using a temp directory in /tmp/random-generated-name as a working dir for cargo to try and isolate it from the environment, only to find that I had a left over /tmp/.cargo/config from some experiment weeks before which was interfering.)

This PR is an attempt to give scripting or programmatic users of Cargo absolute control over where it finds its config to allow complete reproducability. I'm not precious about this - I'd be fine with any other proposal which gives the same level of control.

That said, it looks to me like this mechanism is sufficiently powerful and general to address the needs of #2930, #6728, #7621, and maybe #7723 to an extent.

In #2930's case, I think there's a lot to be said for the proposal that Cargo should have searched relative to the package dir rather than its own current directory, but it's too late to change now and having two separate "search up .. mechanisms" is too confusing to be useful. This PR at least allows that to be manually enforced if needed.

#7723 proposes an include mechanism, which needs its own definition of where to search for includes. I don't think that's strictly related to this PR, other than whatever that search mechanism is, it should be subject to the same control constraints that this PR aims for. Or depending on what the goal of having an include mechanism is, maybe this PR's mechanism subsumes them? (The primary functional difference is that this allows common definitions and overrides to be specified and controlled by the environment, whereas an include mechanism in Cargo.toml means that it's specified by the Cargo.toml author.)

#8148 seems to be more related to the current directory in which build scripts are run, so is orthogonal to this.

[bors]

☔ The latest upstream changes (presumably #8454) made this pull request unmergeable. Please resolve the merge conflicts.

[ehuss]

Sorry for the long radio silence. We discussed this (several times) in the Cargo team meetings, and I think it would be best to have a change of this nature go through the RFC process. There are a lot of facets to consider, and I think it would be best to ensure that the community at least has some input to it, to help ensure that we reach a solution that will address most concerns. Another problem is that nobody on the team has the time to really dig deep into this and help work out the design. I hope this doesn't sound like we are brushing you off, because we do think this is an important thing to improve in Cargo, but team-wise we are very limited.

In regards to this specific design, one concern I would have is that it requires setting an environment variable to change the behavior. This would require setting the environment variable very specifically, and may need to be changed based on the project, which could be awkward. Cargo's current search behavior is not ideal, and it is certainly on the table to change it (perhaps with some manifest key?), though how it should work isn't clear. There could be more than one mechanism to change its behavior as well (though making it too complex isn't ideal).

Would you be willing to try to solicit greater community feedback and work towards a design that could potentially address most of those issues? It might also be helpful if there are other tools of a similar nature that do config-file merging, and see what they do. And maybe collect a few alternative approaches. Also note PR #8652 also wanting to change the search behavior.

[ehuss]

Closing as this PR is quite old and per the comment above, I think we would prefer to make sure changes to config are considered very carefully (and unfortunately none of us have much time to invest in this). I have opened #9769 as a meta issue to cover this.