Implement blackbox in Golang

[tlimoncelli]

Here's a draft of how we could begin to migrate blackbox to be written in Go. I've created a command called "blackbox" that takes various subcommands. Right now, they all call the bash shell script. In the future we can port the shell code to Go. I've written "status" and "nlist" in native Go.

[jbuchbinder]

I've done some similar armoring/dearmoring work with Golang, if you need an extra set of hands with the implementation. Will there be a branch set up that I could work off of?

[TomOnTime]

Sure! Free free to hack on a branch called "golang". I put some very basic scaffolding in place.

Right now each command just calls the appropriate bash script. My goal was to replace each bash script one at a time:

• Phase 1: Call the bash script.
• Phase 2: Replace bash scripts with golang code that shells out to run gpg2
• Phase 3: Replace calls to gpg2 with calls to golang.org/x/crypto/openpgp
• Phase 4: Add new functionality like converting blackbox-files.txt to a JSON format.

There's no unit or integration testing yet. I have no idea what to do there. Help wanted!

The golang branch only adds files to the master branch. Eventually we should be able to simply merge the two.

[jbuchbinder]

The one major issue I see is that we would have to find a way to convert the .kbx GPG 2.1 "keybox" file format back into something that golang's GPG support works with. (There's no 2.1 compatibility support at the moment.)

I can hack on getting decrypt/encrypt support working with the older keybox format, but I'm not going to be able to test locally, since all of my repos had moved to the newer format a while back. (I'd also like to get Subversion support worked in, since I usually use git-crypt for my git repos and blackbox for my Subversion ones.)

[mhenderson-so]

I actually have a fork (that is not compatible with Tom's work unfortunately) of a 75% complete GoLang implementation of Blackbox. It doesn't work with the new GPG 2.1 format, but it does support Linux/Windows and gpgagent. Tom and I have discussed it and there's good stuff in both our versions that we should put together.

If you want to reduce duplication of work, feel free to use parts of mine:

https://github.com/mhenderson-so/go-blackbox

Mine can do cat and addadmin and edit start fine, but it has issues re-encrypting with anything but vanilla keychains. It has bugs when it sees expired certs in keychains or any kind of mismatch.

fwiw I've been using my port internally for a few months just for cating end editing files in Windows and it works great (I've just had to switch to WSL to re-encrypt). My idea was to keep the command structure the same as the original Blackbox, just with spaces instead of underscores. Tom's layout prefers a new command structure, which is probably beneficial in the long run.

[tlimoncelli]

I'm not sure what to do about backwards compatibility.

We could fall back to calling /usr/bin/gpg2 for operations for file formats we can't grok. (just thinking out loud)

[captncraig]

What if we said to that: "Forget backwards compatibility"? The cli is already incompatible. So you have to do work to upgrade.

My vote would be, document the various key formats we support, give examples of how to use different scenarios, and don't worry about supporting older setups we don't want to deal with.

[jcrben]

fyi on backwards compatibility (meaning backwards compatibility with GPG if I understand correctly?), as I mentioned over at #184 (comment):

over at git-secret, a RFC on "A stable and forwards compatible public key storage format" was merged recently https://github.com/sobolevn/git-secret/pull/207/files

might be an approach worth investigating?

[jbuchbinder]

The other possibility (in regard to supporting a blackbox-compatible keystore) is to port the kbx code. It's relatively straighforward C code, so we could theoretically create a simple native Go port.

https://github.com/gpg/gnupg/tree/master/kbx

[tlimoncelli]

Why not drop backwards compatibility? Stack Overflow has eliminated use of GPG1. Is it really in heavy use by other blackbox users?

[jcrben]

@tlimoncelli as far as backwards compatibility, thoughts on this comment from the git-secret RFC linked above:

GPG maintains backwards compatibility but not forwards compatibility. Running a new GPG version can and will upgrade the keyring storage files in a way that is not recognised by older versions of GPG. This is not normally a problem for typical GPG usage. Users will upgrade and rarely downgrade. It is a problem for git-secret as the keyring storage is committed to git and shared between users. Someone using an older version of GPG can no longer open the upgraded keyring file.

GPG1 versus GPG2 isn't the only issue - it seems like GPG will likely continue to break compatibility. They also don't have CI running last I checked either.

[TomOnTime]

@jcrben Thank you for that quote. I learned a lot!

I'm going to add a warning to the README that mixing GPG versions is not supported and could lead to a corrupted key ring.

In the future...

I think the best way to deal with that is to store the public keys in a format that is intended to be portable: gpg --export --armor

Anyway... if we follow the Unix philosophy of "if you change the file format, change the filename" it should be easy to support pubring.gpg/trustdb.gpg for old-style (not very compatible) and a different filename for gpg --export -armor for newer repos; and offer a conversion command.

I'd rather do this in Go than bash. Not that it can't be done in Bash, I just feel like I'd write better error checking in Go.

[mnewswanger]

Adding a comment to hide this from my active feed

[hpoul]

@tlimoncelli is this still something which is in the works, or has this been abandoned in favor of sticking with bash? I would be interested in a clean go implementation, what is the status - anything which would make sense contributing to?

[tlimoncelli]

@hpoul It hasn't been abandoned, I just haven't had time to work on it lately.

I made a branch called "golang" which has the start of a go-lang version. For anything I haven't implemented in golang, it just runs the Bash script. So, in theory you can just start using it now. As we rewrite more commands in bash it just gets cleaner and cleaner.

I'd love to have people rewrite individual commands. I think the basic infrastructure is strong enough that others can hack on it.

Once it has more momentum I'll merge it into master. It has no conflicts with the master branch.

[tlimoncelli]

I am merging this. It isn't complete, but it is good enough to show others. None of the filenames conflict with v1, so it should be possible to maintain both at least for a while.