Manage adding/updating git repos through chezmoi

[johnw188]

Is your feature request related to a problem? Please describe.

I've been setting up my dotfiles and I have found a super common pattern is writing a script called updateX.sh, which basically looks like this:

#!/usr/bin/env zsh

echo Installing ZSH syntax highlighting
echo Install dir: $ZSH_CUSTOM/plugins/zsh-syntax-highlighting
mkdir -p $ZSH_CUSTOM/plugins/zsh-syntax-highlighting
chezmoi add $ZSH_CUSTOM/plugins/zsh-syntax-highlighting
curl -s -L -o zshsyntax.tar.gz https://github.com/zsh-users/zsh-syntax-highlighting/archive/master.tar.gz
chezmoi import --strip-components 1 --destination $ZSH_CUSTOM/plugins/zsh-syntax-highlighting zshsyntax.tar.gz
rm zshsyntax.tar.gz

I would like to deal with this common situation inside the tool that I am using to manage the dotfiles

Describe the solution you'd like

I like having all of my files in a single repo and hate dealing with git submodules. I would like to be able to do a chezmoi update-packaged-files or something similar to download from external sources, unpack into the right place, then clean up the archives and add to chezmoi.

One solution I could see would be a new special filetype inside the chezmoi filesystem called, say, chezmoi_data_source, containing a URL to an archive. This way the chezmoi add-remote-archive [target dir] would just interact with creating this file and another command could do the work of deleting everything apart from chezmoi_data_source, unpacking into the target dir, and doing an add of the content. This way the link to the remote source is recorded.

There would be no nesting support - for an update all command it would stop digging down the directory structure when it hit a chezmoi_data_source file. Data sources would only be allowed and processed in subdirectories.

For github specifically there could be a special case of github repo url -> tarball from master.

Describe alternatives you've considered

The scripting solution I am using needs to be made more generic, and I figured if I was going to make it more generic it would be nice to have it be a core part of the tooling. This would also help with the docs for things like oh-my-zsh

Additional context

I am proficient in golang and could probably implement this feature. I would like to know if it's something you'd be interested in collaborating on, if not I will just go build my generalized script and be done with it.

[twpayne]

Thank you @johnw188 for this really clear and well thought through feature request :)

This is a really good idea. I'm in a similar position, and have similar (but not identical) scripts to handle populating directories from tarballs downloaded from a URL and unpacked.

I think chezmoi should have a feature like you suggest.

A few thoughts off the top of my head:

• I have a bit of recursion in my setup. Specifically, I use both oh-my-zsh and zsh-syntax-highlighting. ~/.oh-my-zsh needs to be populated from one tarball, and ~/.oh-my-zsh/custom/plugins/zsh-syntax-highlighting needs to be populated from another. The eventual feature should support this.

• Nit picking: chezmoi's control files in the source directory start with .chezmoi so I'd change chezmoi_data_source to be .chezmoisource or something like that.

This would also help with the docs for things like oh-my-zsh

Indeed! I think that your feature request would simplify common workflows for many chezmoi users.

Thank you very much for your offer of implementing this feature. I'm very happy to receive contributions :)

Right now I'm in the process of re-working the core of chezmoi in #654 for a version two release that will have quite an impact on the code, but please don't hold back in submitting a PR.

[johnw188]

Do you think that recursion is necessary? I know for oh-my-zsh I have export ZSH_CUSTOM=$HOME/.oh-my-zsh-custom so that my plugins/themes can be maintained outside of the tarball contents, and I'm wondering if there's a case involving standard tools that doesn't have a built in workaround. Ordering/recursing is definitely a solvable problem, but I think that having a .chezmoisource file that is committed to the correct directory is the cleanest way to avoid having to maintain consistency between some root file and the current directory situation.

For example, if I had a tarball extracted to ~/.oh-my-zsh-custom/themes/mytheme and later did a chezmoi remove ~/.oh-my-zsh-custom/themes/mytheme, if there was a .chezmoisource file at dot_oh-my-zsh-custom/themes/mytheme/.chezmoisource the removal would work exactly as expected (as all the files plus the source file would be deleted).

If on the other hand there needed to be a file in the root of the repo called .chezmoisource that had an entry of [github url] -> dot_oh-my-zsh-custom/themes/mytheme, then chezmoi remove would have to update the state of that file. Furthermore, if the user changed their mind and wanted to undo in the first case they could just git checkout dot_oh-my-zsh-custom/themes/mytheme and all the state would be returned with the recreation of the source file, whereas maintaining a mapping would end up being out of date if you did not also git checkout the mapping.

My initial thoughts on command layout would be something like:

• chezmoi package - root command
• chezmoi package add --source-url="" .path/to/target/dir - ensures that dot_path/to/target/dir exists, warns the user that existing content in that dir will be deleted if there is data there already, then creates .chezmoisource and places it in that directory. It then calls chezmoi package update ./path/to/target/dir to do the initial population of the directory.
• chezmoi package update ./path/to/target/dir - does a curl [url in .chezmoisource] to some temporary location, deletes everything under dir apart from .chezmoisource, , then does a chezmoi import to import the archive to the chosen directory. We can just do a reset from source control if any of these steps fail.
• chezmoi package list - searches the chezmoi source directory tree for .chezmoisource files, then outputs a list of descriptions like .path/to/target/dir -> https://github.com/etc/etc
• chezmoi package update --all - searches for all .chezmoisource files and updates them.

[twpayne]

Do you think that recursion is necessary? I know for oh-my-zsh I have export ZSH_CUSTOM=$HOME/.oh-my-zsh-custom so that my plugins/themes can be maintained outside of the tarball contents, and I'm wondering if there's a case involving standard tools that doesn't have a built in workaround.

Thanks, I did not know that we could avoid recursion here, and doing so would definitely makes things simpler. I would also add that there is (probably) a clear and understandable algorithm here: clean out everything from the root .chezmoisource upwards (except for .chezmoi* files) and then re-populate everything from the root upwards.

My inclination towards a .chezmoisource file containing a single URL only rather than a more generic configuration file with entries like [github url] -> dot_oh-my-zsh-custom/themes/mytheme is to do with ambiguity and preventing the user from being able to create inconsistent configurations. What happens if there are multiple URLs in the same file? If we only allow one URL per file then we avoid that.

My initial thoughts on command layout would be something like:

Your proposal for a chezmoi package command is excellent :) This would be awesome.

I realize that you're using curl commands to communicate intent. Please use Go's net/http package for URL downloads.

Thanks again - you're leading something good here :)

[twpayne]

Another thought:

The command and the configuration file should have consistent names. If the command is chezmoi package the the configuration file should be .chezmoipackage or similar.

[johnw188]

My inclination towards a .chezmoisource file containing a single URL only rather than a more generic configuration file with entries like [github url] -> dot_oh-my-zsh-custom/themes/mytheme is to do with ambiguity and preventing the user from being able to create inconsistent configurations. What happens if there are multiple URLs in the same file? If we only allow one URL per file then we avoid that.

Strong agree here, I figure .chezmoipackage would just be one line. I was just saying an info command would have an output like:

> chezmoi package list
[Location]                                           [Source URL]
.oh-my-zsh-custom/plugins/fzf-tab                    https://github.com/Aloxaf/fzf-tab
.oh-my-zsh-custom/plugins/zsh-syntax-highlighting    https://github.com/zsh-users/zsh-syntax-highlighting

I'll probably have some time this next week to poke at this. Is your work in #654 going to affect how commands are added and source is laid out? This feature seems pretty minor as all the heavy lifting is sitting on top of chezmoi import, if you think it isn't going to be a merge nightmare I could try now or else I could wait for you to finish that work. I'm in no rush!

[twpayne]

That all sounds good. Bikeshedding a little, I'd drop the column headers from the chezmoi package list output as they are kinda evident.

#654 will affect the internals, but not anything externally visible to users (apart from fixing a couple of subtle bugs and enabling some new functionality). I think it's worth you starting on chezmoi packages without waiting for it.