blackbox_update_all_files fails with unknown reason

[Constantin07]

When I run blackbox_update_all_files I get this meaningless error with doesn't provide a clue on what the problem could be. It's on latest Mac OS.

$ blackbox_update_all_files
========== blackbox administrators are:
...
========== Importing keychain: START
...
gpg: Total number processed: 106
gpg:              unchanged: 106
========== Importing keychain: DONE
========== ENCRYPTED FILES TO BE RE-ENCRYPTED:
    service/creds.json.gpg
...
========== FILES IN THE WAY:
All OK.
========== RE-ENCRYPTING FILES:
========== PROCESSING "service/creds.json"
ERROR: service/creds.json not found in /Users/user/Git/secrets/keyrings/live/blackbox-files.txt
PWD=/Users/user/git/infra-jobs-secrets
Exiting...

But when I cat blackbox-files.txt I can see there service/creds.json listed.

Any idea what could be wrong ?

[TomOnTime]

Is there a chance that blackbox-files.txt was rewritten using DOS-style line endings? https://github.com/StackExchange/blackbox#protect-the-line-endings

[Constantin07]

I have checked both files the blackbox-admins.txt and the blackbox-files.txt and they do not have any Windows like CR characters inside.

The version of gnupg installed on MacOS:

$ brew info gnupg
gnupg: stable 2.2.6 (bottled)
GNU Pretty Good Privacy (PGP) package
https://gnupg.org/
/usr/local/Cellar/gnupg/2.2.6 (133 files, 10.4MB) *
  Poured from bottle on 2018-04-18 at 16:42:42
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/gnupg.rb
==> Dependencies
Build: pkg-config ✔
Required: npth ✔, gnutls ✔, libgpg-error ✔, libgcrypt ✔, libksba ✔, libassuan ✔, pinentry ✔, gettext ✔, adns ✔
Recommended: libusb ✔
Optional: readline ✔, encfs ✘l

$ echo $BASH_VERSION
3.2.57(1)-release

[tlimoncelli]

That's interesting. Do other blackbox_* commands work?

Does "make test" run to completion? (you'll need to move your ~/.gnupg directory out of the way temporarily)

Could you add this debugging line and show me the output?

diff --git a/bin/_blackbox_common.sh b/bin/_blackbox_common.sh
index 6d98786..965e392 100755
--- a/bin/_blackbox_common.sh
+++ b/bin/_blackbox_common.sh
@@ -391,6 +391,7 @@ function remove_line() {
 function file_contains_line() {
   # $1: the file
   # $2: the line
+  echo grep -xsqF X"$2"X X"$1"X
   grep -xsqF "$2" "$1"
 }

[Constantin07]

Thanks @TomOnTime @tlimoncelli for helping out.

It looks like the root cause was the MacOS shell behaviour (tested in iTerm at least) which converts directories starting with capital letter to lower case. See below difference in the output the directory name Git in ERROR vs git in PWD:

ERROR: service/creds.json not found in /Users/user/Git/secrets/keyrings/live/blackbox-files.txt
PWD=/Users/user/git/infra-jobs-secrets

It used to work before but after the latest upgrade to Mac OS Sierra ver. 10.13.4 it stopped working.
I don't know what went wrong but renaming directory from Git to git sorted the issue.

[selfcommit]

Previous osx file systems were case preserving, but insensitive. Is it possible your recent upgrade also changed your filesystem? https://www.quora.com/Why-does-OS-X-choose-to-have-a-case-insensitive-file-system-instead-of-a-case-sensitive-one

…

On Thu, Apr 19, 2018, 7:50 AM Constantin Bugneac ***@***.***> wrote: Thanks @TomOnTime <https://github.com/TomOnTime> @tlimoncelli <https://github.com/tlimoncelli> for helping out. It looks like the root cause was the MacOS shell behaviour (tested in iTerm at least) which converts directories starting with capital letter to lower case. See below difference in the output the directory name Git in ERROR vs git in PWD: ERROR: service/creds.json not found in /Users/user/Git/secrets/keyrings/live/blackbox-files.txt PWD=/Users/user/git/infra-jobs-secrets It used to work before but after the latest upgrade to Mac OS Sierra ver. 10.13.4 it stopped working. I don't know what went wrong but renaming directory from Git to git sorted the issue. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#249 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-zpsGPE2nAbD1p3gRxjz5RQZ_Iut-7ks5tqHoDgaJpZM4TYpdw> .

[Constantin07]

I don't think the filesystem has changed during upgrade. Here is what I've got now:

$diskutil list
/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.3 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                 Apple_APFS Container disk1         500.1 GB   disk0s2

/dev/disk1 (synthesized):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      APFS Container Scheme -                      +500.1 GB   disk1
                                 Physical Store disk0s2
   1:                APFS Volume Macintosh HD            56.4 GB    disk1s1
   2:                APFS Volume Preboot                 21.0 MB    disk1s2
   3:                APFS Volume Recovery                517.8 MB   disk1s3
   4:                APFS Volume VM                      2.1 GB     disk1s4

Looks like blackbox bash script(s) have some portions which differently treat directories with capital letters thus the issue.

[tlimoncelli]

I'd like to fix this... or at least have the error message give a hint about what to do. Sadly I haven't been able to reproduce the problem. I've tried a number of things but no luck.

If I had to guess, the problem is in vcs_relative_path()

[charlesverdad]

Having the same error message in Ubuntu 16.04 running in a VM. Managed to solve the problem when I realized the project folder was sym-linked elsewhere. I just had to run it in the original path.

[VZhuck]

The same issue on Windows, tried to add file from windows. I tried to decrypt file, but got the same issue
========== PLAINFILE "env-secrets.EPAM.json" ERROR: env-secrets.json not found in /d/secrets/.blackbox/blackbox-files.txt PWD=/d/secrets/ah-secrets Exiting...
Troubleshooting

1. Tried to add file from windows $ blackbox_register_new_file test.json
2. Actual Result::
   then checked blackbox-files.txt and find following record
   d/repo_ah/ah-secrets/test.json
3. Expected result
   test.json

[tlimoncelli]

@VZhuck Are you using Cygwin or MinGW?

[VZhuck]

Hi @tlimoncelli,
I am using "MinGW", which come by default with git.

Looks like root cause:

• when I navigate to blackbox git repo folder I typed **cd c:/bb_test/ **, however folder name is Bb_Test (camel case)

However when I run the same commands under Linux Subsystem on Windows (Ubuntu), it works as expected

[TomOnTime]

That makes sense. The system was designed for a filesystem that is case sensitive. If a directory is named "foo" and you get there by "cd FOO", all bets are off.