Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve filepaths before scanning for changes #1069

Conversation

cvgw
Copy link
Contributor

@cvgw cvgw commented Feb 20, 2020

Description

Implement the new filepath resolver described in https://github.com/GoogleContainerTools/kaniko/blob/d31d2bc74c884ff5d7932b1f6e7209c6fc63cc5e/docs/design_proposals/filesystem-resolution-proposal-01.md

One change to note is that ancestor directories will no longer be added regardless of whether they are changed. The ancestor directory logic now occurs before the change checking, so ancestor directories will only be added if they have changed.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

  • Includes unit tests
  • Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

  • The code flow looks good.
  • Unit tests and or integration tests added.

@googlebot googlebot added the cla: yes CLA signed by all commit authors label Feb 20, 2020
@cvgw cvgw force-pushed the u/cgwippern/implement-filepath-resolver branch 5 times, most recently from ade927b to 92cef5c Compare February 20, 2020 17:21
@cvgw cvgw force-pushed the u/cgwippern/implement-filepath-resolver branch from 92cef5c to a675ad9 Compare February 20, 2020 17:45
Copy link
Contributor

@samos123 samos123 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor nits regarding commenting out code instead of removing


for _, file := range files {
file = filepath.Clean(file)
filesSet[file] = true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can skip traversing if fileSet[file] = true

Suggested change
filesSet[file] = true
if _, ok := fileSet[file] ; !ok {
filesSet[file] = true
for _, dir := range util.ParentDirectories(file) {
dir = filepath.Clean(dir)
filesSet[dir] = true
}
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function filesWithParentDirs was copy/pasted from the util pkg. If you think it won't clutter the change I'm happy to take this opportunity to make these updates.

Copy link
Member

@tejal29 tejal29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some more tests for resolveSymlinkAncestor

}
}

newFiles := []string{}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can optimize this by allocation memory of expected size

newFiles := make([]string, 0, len(filesSet))

pkg/snapshot/snapshot.go Outdated Show resolved Hide resolved
pkg/filesystem/resolve.go Show resolved Hide resolved
pkg/filesystem/resolve.go Show resolved Hide resolved
}

if target != newPath {
last = filepath.Base(newPath)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shd this be target?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't believe so. We want to return the link, not the target.

pkg/filesystem/resolve.go Outdated Show resolved Hide resolved
@cvgw cvgw force-pushed the u/cgwippern/implement-filepath-resolver branch 3 times, most recently from 012d8cd to ce8eebb Compare February 22, 2020 19:10
@cvgw cvgw force-pushed the u/cgwippern/implement-filepath-resolver branch 2 times, most recently from b380112 to 2f2914d Compare February 22, 2020 19:29
@cvgw cvgw force-pushed the u/cgwippern/implement-filepath-resolver branch from 2f2914d to 965b606 Compare February 23, 2020 21:38
@cvgw cvgw requested a review from tejal29 February 23, 2020 21:56
@tejal29 tejal29 merged commit a1af057 into GoogleContainerTools:master Feb 25, 2020
@cvgw cvgw deleted the u/cgwippern/implement-filepath-resolver branch February 25, 2020 16:43
@cvgw
Copy link
Contributor Author

cvgw commented Feb 25, 2020

related issues #1039 #1038 #1024

dinvlad added a commit to broadinstitute/dsp-appsec-infrastructure-apps that referenced this pull request Mar 6, 2020
This is needed until 'latest'/'debug' tag is applied to
GoogleContainerTools/kaniko#1069
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cla: yes CLA signed by all commit authors
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants