ecr-assets: incomplete .dockerignore support

[kolomied]

Introduced by #4104.

CDK does not handle the common "exclude everything except" pattern for .dockerignore files:

# Ignore everything
*
# Allow files and directories
!/src/**

Quote from Docker docs:

... you may want to specify which files to include in the context, rather than which to exclude. To achieve this, specify * as the first pattern, followed by one or more ! exception patterns.

With the current implementation the first 'star' glob effectively excludes everything from the asset folder dyring synth stage. Consequently, deploy stage fails to build the docker image.

Reproduction Steps

1. Create a .dockerignore file next to you Dockerfile with * pattern, like this:

   # Ignore everything
   *
   # Allow files and directories
   !/src/**

2. Run cdk deploy

Error Log

unable to prepare context: unable to evaluate symlinks in Dockerfile path: CreateFile C:\Dev\GitHub\playground\EcsWindowsCluster\src\Deployment\cdk.out\asset.1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0\Dockerfile: The system cannot find the file specified.

 ❌  win failed: Error: docker build --tag 305168429857.dkr.ecr.eu-west-1.amazonaws.com/cdk/win:latest cdk.out\asset.1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0 exited with error code 1
docker build --tag 305168429857.dkr.ecr.eu-west-1.amazonaws.com/cdk/win:latest cdk.out\asset.1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0 exited with error code 1

Environment

• CLI Version : 1.12.0 (build 923055e)
• OS : Windows

---

This is 🐛 Bug Report

[nmussy]

Sorry about my previous comment, from what I can see it's working completely fine, assets/fs is doing its magic. I've tested the following successfully:

> tree test/dockerignore-image-advanced/

dockerignore-image-advanced/
├── deep
│   ├── dir
│   │   └── struct
│   │       └── qux.txt
│   └── include_me
│       └── sub
│           └── dir
│               └── quuz.txt
├── Dockerfile
├── foobar.txt
├── foo.txt
├── index.py
└── subdirectory
    ├── baz.txt
    └── quux.txt

# The following line should be ignored
#index.py

# This shouldn't ignore foo.txt
foo.?
# This shoul ignore foobar.txt
foobar.???
# This should catch qux.txt
deep/**/*.txt
# but quuz should be added back
!deep/include_me/**

# baz and quux should be ignored
subdirectory/**
# but baz should be added back
!subdirectory/baz*

const expectedFiles = [
  '.dockerignore',
  'Dockerfile',
  'index.py',
  'foo.txt',
  path.join('subdirectory', 'baz.txt'),
  path.join('deep', 'include_me', 'sub', 'dir', 'quuz.txt'),
];
const unexpectedFiles = [
  'foobar.txt',
  path.join('deep', 'dir', 'struct', 'qux.txt'),
  path.join('subdirectory', 'quux.txt'),
];

for (const expectedFile of expectedFiles) {
  test.ok(fs.existsSync(path.join(session.directory, `asset.${image.sourceHash}`, expectedFile)), expectedFile);
}
for (const unexpectedFile of unexpectedFiles) {
  test.ok(!fs.existsSync(path.join(session.directory, `asset.${image.sourceHash}`, unexpectedFile)), unexpectedFile);
}

I can add this test case to test.image-asset.ts to make it a more comprehensive, but I can't reproduce @kolomied's issue.

[kolomied]

@nmussy Please try to add a '*' as your first pattern in .dockerignore. Here is a small example that I use to reproduce the issue:

Test          
├── .dockerignore
├── Dockerfile   
├── bad.txt      
└── publish      
    └── good.txt 

.dockerignore is as follows:

*
!publish\*

After the synth phase I get the following asset:

asset.1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0          
└── .dockerignore

Howerver, I really expect to see this:

asset.1ebc9d3ac2033816c4abb63e4afd69d350b4aba8704cc9236b82ea520b74f4b0   
├── .dockerignore
├── Dockerfile      
└── publish      
    └── good.txt 

[nmussy]

I see, thanks for clearing it up for me.

But I don't understand why you would expect anything but publish/good.txt to be present with your .dockerignore:

You can even use the .dockerignore file to exclude the Dockerfile and .dockerignore files. These files are still sent to the daemon because it needs them to do its job. But the ADD and COPY instructions do not copy them to the image.

[kolomied]

Hm... My assumption is that synth stage has to prepare a self-contained folder that I can archive and use later to actually deploy the application (CI/CD scenario, for example).

Without Dockerfile there is not way to build the container during deploy stage, so it has to be in the asset folder.

Am I missing something here?

[nmussy]

You're completely right, the Dockerfile would be required to build the image, we should probably always add it even if the user accidentally ignores it.

[nmussy]

Here's what I've found:

copyDirectory walks through each directory recursively, making sure they're not excluded. If the directory is excluded, its files and sub-directories won't be explored:

aws-cdk/packages/@aws-cdk/assets/lib/fs/copy.ts

Lines 21 to 23 in dcd8d1e

	if (shouldExclude(exclude, path.relative(rootDir, sourceFilePath))) {
	continue;
	}

shouldExclude on the other hand matches .dockerignore lines against the path it's being provided:

aws-cdk/packages/@aws-cdk/assets/lib/fs/utils.ts

Line 20 in dcd8d1e

const match = minimatch(filePath, pattern, { matchBase: true, flipNegate: true });

So in the case of publish/good.txt, publish is evaluated against each .dockerignore line.

• * matches publish, and excludes the directory
• !publish\* does not match publish
• copyDirectory leaves the directory early, without having checked its files

We need to figure out how to affect either shouldExclude or copyDirectory to prevent the recursion from stopping early

[enricopesce]

I have today the same problem with:

cdk 1.103.0 (build bc13a66)

@aws-cdk/assets 1.103.0

OS: osx 11.3.1

typescript language