Simple CLI to transform a gcs bucket into your filesystem cache during your builds
gcs-cache
is a CLI utility on top of gsutil
in order to facilitate storing & loading filesystem
content to/from a google-cloud-storage bucket.
-
Authenticate against a gcloud service account to gain access to your gcs bucket
-
Provide caching mechanism : for example, you can decide to cache your
node_modules
folder and rely on its cached content based onpackage.json
checksum :- if
package.json
checksum in the cache is the same than your localpackage.json
checksum, then the CLI will download your cachednode_modules
folder - otherwise, a new
npm install
will be performed locally, and the resultingnode_modules
content will be cached for further use
- if
-
Store local filesystem to your gcs bucket
-
Load remote gcs bucket into your local filesystem
-
Decide whether you want to compress (or not) your files prior to storing it into gcs
npm install -g gcs-cache
- in order to make it work, you will need
gcloud
andgsutil
commands available into yourPATH
. You can find autilities/install-gcloud.sh
installation script utility to facilitate this in the repository :
curl "https://raw.githubusercontent.com/fcamblor/gcs-cache/1.2.4/utilities/install-gcloud.sh" | bash -s [<path/to/bin/folder>] [<path/to/gcloud/installation/folder>]
- you will need
node@14
to run it properly
gcs-cache auth "--key-config-url=https://url/targetting/google-service-account.json"
-or-
gcs-cache auth "--key-config-file=/local/path/to/google-service-account.json"
This is a required step to authenticate against your google cloud service account prior to making any manipulation through the CLI.
gcs-cache cached-fs --bucket-url=gs://my-bucket --app=my-app --branch=my-branch --cache-name=npm-packages --checksum-file=package.json "--cacheable-command=npm install" node_modules:node_modules
This will :
- Retrieve checksum from
checksum-file
(package.json
) - Compare it with gcs
npm-packages
cache metadata for branchmy-branch
in appmy-app
:- If checksums differ, then
cacheable-command
(npm install
) will be executed, andnode_modules
directory will then be put intonpm-packages
cache for branchmy-branch
- If checksums are the same, then
npm-packages
cache will be downloaded and applied onnode_modules
directory
- If checksums differ, then
gcs-cache store-fs --bucket-url=gs://my-bucket --app=my-app --branch=my-branch --cache-name=my-cache directory1:directory1 directory2:./path/to/directory2
This will store directory1
and directory2
contents into cache my-cache
for branch my-branch
in app my-app
You can avoid passing any directory: in that case, the whole current directory content will be put into the cache.
gcs-cache load-fs --bucket-url=gs://my-bucket --app=my-app --branch=my-branch --cache-name=my-cache directory1:directory1 directory2:./path/to/directory2
This will load directory1
and directory2
content from cache my-cache
for branch my-branch
in app my-appp
into current directory's directory1
and directory2
You can avoid passing any directory: in that case, the special __all__
cache content will be
put into current directory.
Whenever you provide directories to cached-fs
/ store-fs
/ load-fs
commands, those directories are
"named" directories, following format content-name:directory-path
When content-name
and directory-path
are the same, you can shortcut it to simply directory-path
, meaning that
those 2 different commands are the same :
gcs-cache store-fs --bucket-url=gs://my-bucket --app=my-app --branch=my-branch --content-name=my-cache directory1:directory1 directory2:./path/to/directory2
gcs-cache store-fs --bucket-url=gs://my-bucket --app=my-app --branch=my-branch --content-name=my-cache directory1 directory2:./path/to/directory2
Cache contents will differ depending on whether content has been created using cached-fs
or store-fs
/load-fs
commands.
Every caches can be resolved through 4 coordinates :
- bucket url :
gs://my-bucket
- app : a unique app id identifier used to isolate contents from 1 app to another inside the same bucket Important note: note that by giving access to a bucket, you give access to every app content stored on this bucket (no ACL is setup to limit access to apps content from a single bucket)
- branch name : when omitted, we fallback on the
unknown-branch
name - cache name
Bucket gs://my-bucket
cached content will differ depending on whether content is compressed
or not.
For compressed content :
<app>/metadata/<branch-name>/<cache-name>.json
with following content :
{
"checksum": "bc694811390c7b4da545a364a3993d7d",
"compressed": true,
"all": false
}
<app>/content/<branch-name>/<cache-name>/<directory1>.tar.gz
with compresseddirectory1
content<app>/content/<branch-name>/<cache-name>/<directory2>.tar.gz
with compresseddirectory2
content
etc.
For uncompressed content :
<app>/metadata/<branch-name>/<cache-name>.json
with following content :
{
"checksum": "bc694811390c7b4da545a364a3993d7d",
"compressed": false,
"all": false
}
<app>/content/<branch-name>/<cache-name>/directory1/*
withdirectory1
content<app>/content/<branch-name>/<cache-name>/directory2/*
withdirectory2
content
etc.
Bucket gs://my-bucket
cached content will differ depending on whether content is compressed
or not.
For compressed content :
<app>/metadata/<branch-name>/<cache-name>.json
with following content :
{
"compressed": true,
"all": false
}
<app>/content/<branch-name>/<cache-name>/<directory1>.tar.gz
with compresseddirectory1
content<app>/content/<branch-name>/<cache-name>/<directory2>.tar.gz
with compresseddirectory2
content
etc.
For uncompressed content :
<app>/metadata/<branch-name>/<cache-name>.json
with following content :
{
"compressed": false,
"all": false
}
<app>/content/<branch-name>/<cache-name>/directory1/*
withdirectory1
content<app>/content/<branch-name>/<cache-name>/directory2/*
withdirectory2
content
etc.
When no directory is specificed to store-fs
/ load-fs
commands, then have following content :
<app>/metadata/<branch-name>/<cache-name>.json
with following content :
{
"compressed": true,
"all": true
}
<app>/content/<branch-name>/<cache-name>/__all__.tar.gz
with compressed current directory content
For uncompressed no directory, this is the same idea, except that __all__.tar.gz
archive is replaced
by a __all__
directory