Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

izogfif · 2024-03-02T14:59:50Z

Fixes #4141 🛠️

izogfif · 2024-03-02T15:01:25Z

This is a copy of this pull request with an extra commit that fixes the formatting, test, and modification time set by JIB issues.

chanseokoh · 2024-03-02T20:20:08Z

jib-core/src/main/java/com/google/cloud/tools/jib/image/ReproducibleLayerBuilder.java

+        // In this particular place we have to use default modification time (1 second past epoch)
+        // instead the modification time user might've set in JIB settings. That's because we have
+        // no way to find out the modification time user might've set in JIB settings here.


This is not true. Jib intentionally doesn't allow setting the mod time of directories. Let's just remove the comment.

chanseokoh · 2024-03-02T20:25:47Z

jib-core/src/test/java/com/google/cloud/tools/jib/image/ReproducibleLayerBuilderTest.java

+  /**
+   * Verifies that the modification time has been reset in the PAX headers.
+   *
+   * @param entry The archive entry in the layer.
+   */
+  private static void verifyThatModificationTimeIsReset(ArchiveEntry entry) {
+    assertThat(entry.getLastModifiedDate().toInstant())
+        .isEqualTo(FileEntriesLayer.DEFAULT_MODIFICATION_TIME);
+  }
+
+  private static FileEntry layerEntry(
+      Path source, AbsoluteUnixPath destination, FilePermissions permissions, String ownership)
+      throws IOException {
    return new FileEntry(
        source,
        destination,
-        FileEntriesLayer.DEFAULT_FILE_PERMISSIONS_PROVIDER.get(source, destination),
-        FileEntriesLayer.DEFAULT_MODIFICATION_TIME);
+        permissions,
+        // Here we make sure to use a default modification time
+        // since JIB will set modification time for all files to
+        // the one set in JIB settings by a user. In case user
+        // hasn't set it, JIB will use default modification time,
+        // i.e. 1 second past January 1st, 1970 (1 second past epoch).
+        FileEntriesLayer.DEFAULT_MODIFICATION_TIME,
+        ownership);
+  }
+
+  private static FileEntry layerEntry(
+      Path source, AbsoluteUnixPath destination, FilePermissions permissions) throws IOException {
+    return layerEntry(source, destination, permissions, "");
+  }
+
+  private static FileEntry layerEntry(Path source, AbsoluteUnixPath destination)
+      throws IOException {
+    return layerEntry(
+        source,
+        destination,
+        FileEntriesLayer.DEFAULT_FILE_PERMISSIONS_PROVIDER.get(source, destination));


Just revert this file back to its original state. The original author misunderstood this part. The PR shouldn't touch this file, and the existing tests will verify the correctness of your changes.

chanseokoh

Also, upgrade the apache-commons-compress (#3964) to verify if it's working correctly.

izogfif · 2024-03-03T03:07:41Z

@chanseokoh done. I had to update TarExtractorTest since it was failing now. Probably, newer version of apache-commons-compress (I used 1.26.0 instead of 1.23.0 from #3964) extracts modification timetamp with millisecond precision.

chanseokoh

Normally you should update jib-core/CHANGELOG, jib-maven-plugin/CHANGELOG and jib-gradle-plugin/CHANGELOG about what is fixed.

BTW, I'm not the maintainer of this repo.

chanseokoh · 2024-03-04T16:47:05Z

jib-core/src/main/java/com/google/cloud/tools/jib/image/ReproducibleLayerBuilder.java

+    entry.setModTime(modTime.toEpochMilli());
+
+    String headerTime = Long.toString(modTime.getEpochSecond());
+    final long nanos = modTime.getNano();


I'm not familiar with the PAX headers, so just asking: does it have to be the nanosecond precision? Like, what if someone decides to go with milli or microseconds? Is it possible? Not that I care as long as nano works.

Here is a post I was able to find:

This is important for example with the tarfile module with the pax tar format. The POSIX tar standard mandates storing the mtime in the extended header (if it is not an integer) with as much precision as is available in the underlying file system, and likewise to restore this time properly upon extraction.

So I'd say that it's better to store time with nanosecond precision. File systems that don't support this precision won't fail upon extraction: they'll simply truncate values.

izogfif · 2024-03-04T18:08:17Z

CHANGELOG files were updated.

izogfif · 2024-03-05T07:52:53Z

Rebased on master.

Co-authored-by: Mridula <66699525+mpeddada1@users.noreply.github.com>

…s now used for PAX headers. Some parts of test were reverted. Code reformat.

…omment was removed. ReproducibleLayerBuilderTest.java was reverted to its original state.

…rieved with higher precision.

izogfif · 2024-03-06T12:07:03Z

Rebased on master (2).

mpeddada1 · 2024-03-06T15:13:13Z

jib-core/src/test/java/com/google/cloud/tools/jib/tar/TarExtractorTest.java

+    // We have to use millisecond precision here. Taken from Instant.parse.
+    final DateTimeFormatter parser =
+        new DateTimeFormatterBuilder().parseCaseInsensitive().appendInstant(3).toFormatter();
+


@izogfif Is there is a reason we're testing with millisecond precision here? Curious since we opted for nanosecond precision above (ReproducibleLayerBuilder)

@mpeddada1 For some reason, the code below can't extract micro- and nanoseconds, it's limited to milliseconds (at least on Windows machine with NTFS).
In one of the methods below I had to add centiseconds and avoided having to use the same DateTimeFormatter since, conveniently, truncation to milliseconds yielded "0" as the third digit, and Instant.parse can parse centiseconds.

mpeddada1 · 2024-03-06T15:30:15Z

Pasting stacktrace of failing checks below:

kokoro-macos:

Successfully started process 'Gradle Test Executor 4'
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv6Addresses=true

com.google.cloud.tools.jib.registry.DockerRegistryBearerTokenTest > testDecode_invalidToken_actionsArray SKIPPED

com.google.cloud.tools.jib.tar.TarExtractorTest > testExtract_modificationTimePreserved FAILED
    value of: getLastModifiedTime(...)
    expected: 2019-08-01T16:13:09.72Z
    but was : 2019-08-01T16:13:09Z
        at com.google.cloud.tools.jib.tar.TarExtractorTest.testExtract_modificationTimePreserved(TarExtractorTest.java:102)

com.google.cloud.tools.jib.tar.TarExtractorTest > testExtract_reproducibleTimestampsEnabled FAILED
    value of: getLastModifiedTime(...)
    expected: 2021-01-29T21:10:02.78Z
    but was : 2021-01-29T21:10:02Z
        at com.google.cloud.tools.jib.tar.TarExtractorTest.testExtract_reproducibleTimestampsEnabled(TarExtractorTest.java:129)

Gradle Test Executor 4 finished executing tests.

> Task :jib-core:test FAILED

kokoro-ubuntu

com.google.cloud.tools.jib.api.ContainerizerIntegrationTest STANDARD_ERROR
    SLF4J: No SLF4J providers were found.
    SLF4J: Defaulting to no-operation (NOP) logger implementation
    SLF4J: See [https://www.slf4j.org/codes.html#noProviders](https://www.google.com/url?q=https://www.slf4j.org/codes.html%23noProviders&sa=D) for further details.

com.google.cloud.tools.jib.api.ReproducibleImageTest > testConfiguration FAILED
    value of: extractFromTarFileAsString(...)
    expected: ?"diff_ids":["sha256:18e4f44e6d1835bd968339b166057bd17ab7d4cbb56dc7262a5cafea7cf8d405","sha256:13369c34f073f2b9c1fa6431e23d925f1a8eac65b1726c8cc8fcc2596c69b414","sha256:4f92c507112d7880ca0f504ef8272b7fdee107263270125036a260a741565923"]}}
    but was : ?"diff_ids":["sha256:2fcc2157bf42c89195676ef6e973a96d7b018c9d30ba89db95e9e0722e1c8ef3","sha256:21f521f3217067d277af37512a08c72281d90fdd02d7174db632c8c3a34403bd","sha256:6beba018395265af5061864b7f4678e831eb2daebb1045487c641fc8b142e319"]}}
        at com.google.cloud.tools.jib.api.ReproducibleImageTest.testConfiguration(ReproducibleImageTest.java:130)

com.google.cloud.tools.jib.api.ReproducibleImageTest > testTarballStructure FAILED
    missing (3)
    #1      : c46572ef74f58d95e44dd36c1fbdfebd3752e8b56a794a13c11cfed35a1a6e1c.tar.gz
    #2      : 6d2763b0f3940d324ea6b55386429e5b173899608abf7d1bff62e25dd2e4dcea.tar.gz
    #3      : 530c1954a2b087d0b989895ea56435c9dc739a973f2d2b6cb9bb98e55bbea7ac.tar.gz

    unexpected (3)
    #1      : 98682a867906d9d07cf3c51a4fb9e08e9d5baddd1ca5dc7834f58f434c9cb15c.tar.gz
    #2      : 527db49d4e0c4159346119b4971d59016bfedceed874abab2b510ce433f6b15c.tar.gz
    #3      : 16d03883198935b4119896dcea0ea14e1bf105b6ac0a35a88820d08bc0263306.tar.gz
    ---
    expected: [c46572ef74f58d95e44dd36c1fbdfebd3752e8b56a794a13c11cfed35a1a6e1c.tar.gz, 6d2763b0f3940d324ea6b55386429e5b173899608abf7d1bff62e25dd2e4dcea.tar.gz, 530c1954a2b087d0b989895ea56435c9dc739a973f2d2b6cb9bb98e55bbea7ac.tar.gz, config.json, manifest.json]
    but was : [98682a867906d9d07cf3c51a4fb9e08e9d5baddd1ca5dc7834f58f434c9cb15c.tar.gz, 527db49d4e0c4159346119b4971d59016bfedceed874abab2b510ce433f6b15c.tar.gz, 16d03883198935b4119896dcea0ea14e1bf105b6ac0a35a88820d08bc0263306.tar.gz, config.json, manifest.json]
        at com.google.cloud.tools.jib.api.ReproducibleImageTest.testTarballStructure(ReproducibleImageTest.java:104)

com.google.cloud.tools.jib.api.ReproducibleImageTest > testManifest FAILED
    value of: extractFromTarFileAsString(...)
    expected: [{"Config":"config.json","RepoTags":["jib-core/reproducible:latest"],"Layers":["c46572ef74f58d95e44dd36c1fbdfebd3752e8b56a794a13c11cfed35a1a6e1c.tar.gz","6d2763b0f3940d324ea6b55386429e5b173899608abf7d1bff62e25dd2e4dcea.tar.gz","530c1954a2b087d0b989895ea56435c9dc739a973f2d2b6cb9bb98e55bbea7ac.tar.gz"]}]
    but was : [{"Config":"config.json","RepoTags":["jib-core/reproducible:latest"],"Layers":["98682a867906d9d07cf3c51a4fb9e08e9d5baddd1ca5dc7834f58f434c9cb15c.tar.gz","527db49d4e0c4159346119b4971d59016bfedceed874abab2b510ce433f6b15c.tar.gz","16d03883198935b4119896dcea0ea14e1bf105b6ac0a35a88820d08bc0263306.tar.gz"]}]
        at com.google.cloud.tools.jib.api.ReproducibleImageTest.testManifest(ReproducibleImageTest.java:119)

Gradle Test Executor 7 finished executing tests.

> Task :jib-core:integrationTest FAILED

izogfif · 2024-03-06T17:00:52Z

@mpeddada1

Pasting stacktrace of failing checks below:

kokoro-macos:

com.google.cloud.tools.jib.tar.TarExtractorTest > testExtract_modificationTimePreserved FAILED
    value of: getLastModifiedTime(...)
    expected: 2019-08-01T16:13:09.72Z
    but was : 2019-08-01T16:13:09Z
        at com.google.cloud.tools.jib.tar.TarExtractorTest.testExtract_modificationTimePreserved(TarExtractorTest.java:102)

How can I check that current OS is macOS? It seems that storing files with millisecond precision on macOS is not supported. It seems that this is what's happening:

TarExtractor.extract extracts a .tar file from resources into a temporary folder on test machine.
Underlying file system truncates file modifications timestamps (they're stored with nanosecond precision inside tar file). On Windows and Ubuntu, truncation is done to millisecond precision. On macOS, apparently, to seconds.
Tests pass on Ubuntu and Windows. They fail on macOS.

Possible solutions:

Make macOS provide file modification timestamps with millisecond precision (like Windows and Ubuntu do).
Read file modification timestamps with any precision, then truncate them to seconds, and then check these truncated values in tests.
Third way.

kokoro-ubuntu

About these failures: if I update values ("expected") with new ones ("but was"), it should fix Ubuntu build. But won't this fail Windows build?

…odification time with millisecond precision.

izogfif · 2024-03-07T11:07:38Z

@mpeddada1 I finally found the contributor guide and was able to run tests on Ubuntu via:

export JIB_INTEGRATION_TESTING_LOCATION=localhost:9990
./gradlew clean goJF build integrationTest

but then testWar_jetty test fails with:

java.lang.AssertionError: expected:<Hello world> but was:<null>

Apparently, I'm unable to make Jetty container start in Docker. Please re-run those workflows / kokoro-xxx tests on your end.

chanseokoh · 2024-03-07T13:26:00Z

I also noticed the Jetty failure locally. Don't know why. It is the test in jib-cli. Since you are not changing the jib-cli code, for now, you can do the following which will cover your changes:

./gradlew jib-core:integrationTest jib-maven-plugin:integrationTest jib-gradle-plugin:integrationTest

mpeddada1 · 2024-03-07T21:31:22Z

TarExtractor.extract extracts a .tar file from resources into a temporary folder on test machine.

Underlying file system truncates file modifications timestamps (they're stored with nanosecond precision inside tar file). On Windows and Ubuntu, truncation is done to millisecond precision. On macOS, apparently, to seconds.

Tests pass on Ubuntu and Windows. They fail on macOS.

Thanks for looking into this! It is interesting that macos only goes to seconds precision. It is possible that the job still uses the old HFS+ format drive. That being said, I think verifying seconds precision should be sufficient.

Re kokoro-ubuntu: I would assume that the digest value change is due to the new PAX header logic we added so i would expect it to match in both Ubuntu and windows. But I could also be missing something here, why do we think the windows tests would fail?

izogfif · 2024-03-07T21:59:40Z

Re kokoro-ubuntu: I would assume that the digest value change is due to the new PAX header logic we added so i would expect it to match in both Ubuntu and windows. But I could also be missing something here, why do we think the windows tests would fail?

Yes, I think that PAX header changed checksums. I was expecting tests to fail on all OS (Windows, macOS, Ubuntu), but, apparently, before I fixed checksum checks, kokoro-windows tests were successful. No idea why. The tests have passed this time, too.

mpeddada1 · 2024-03-08T21:39:46Z

Yes, I think that PAX header changed checksums. I was expecting tests to fail on all OS (Windows, macOS, Ubuntu), but, apparently, before I fixed checksum checks, kokoro-windows tests were successful. No idea why. The tests have passed this time, too.

Ah our coverage on Windows is fairly limited at the moment as we run integration tests only in ubuntu and MacOS. That's probably why we weren't able to see the failure in windows.

izogfif · 2024-03-09T15:04:56Z

So, about this pull request, what are the next steps? Is there something I need to do in order for it to be accepted (and a new build of JIB to be released)?

mpeddada1 · 2024-03-11T17:18:11Z

No further action need, I'll go ahead and merge the PR. Thanks for your contribution @izogfif!

glasser · 2024-03-18T16:23:48Z

Thanks for everyone who worked on this! Looking forward to seeing this get into a release!

izogfif mentioned this pull request Mar 2, 2024

Make image builds reproducible #4142

Closed

chanseokoh reviewed Mar 2, 2024

View reviewed changes

chanseokoh approved these changes Mar 4, 2024

View reviewed changes

bjornbugge and others added 12 commits March 6, 2024 12:06

Set access, creation, and last-modified time to zero

2f19840

Add small test that ensures that modtime doesn't change layer

8d5bedd

Also clear time-related PAX headers on parent-dir entries

3180eda

Add blank line for checkstyle

db224ae

Co-authored-by: Mridula <66699525+mpeddada1@users.noreply.github.com>

Fix the modification-time tests

a352e30

Rename clearTimeHeaders to clearPaxTimeHeaders

0894421

Set PAX headers to EPOCH plus 1

85c9dd1

Rename toclearTimeHeaders and call setModTime in helper

028db50

File modification time set by JIB (and possibly overridden by user) i…

909bfcd

…s now used for PAX headers. Some parts of test were reverted. Code reformat.

Bump org.apache.commons:commons-compress from 1.21 to 1.26.0. Wrong c…

b885461

…omment was removed. ReproducibleLayerBuilderTest.java was reverted to its original state.

TarExtractorTest fixes: modification time from tar archive is now ret…

6b22b68

…rieved with higher precision.

CHANGELOG files were updated.

2663c80

izogfif mentioned this pull request Mar 6, 2024

Builds not reproducible #4141

Closed

mpeddada1 added the kokoro:force-run label Mar 6, 2024

kokoro-team removed the kokoro:force-run label Mar 6, 2024

mpeddada1 reviewed Mar 6, 2024

View reviewed changes

mpeddada1 added the kokoro:force-run label Mar 6, 2024

kokoro-team removed the kokoro:force-run label Mar 6, 2024

izogfif added 3 commits March 7, 2024 10:08

Tests should now accommodate file systems incapable of storing file m…

b979655

…odification time with millisecond precision.

Tar hashes and file names were updated.

8256b8a

Typo fix.

5af8470

ReproducibleImageTest error fix.

042bcb1

mpeddada1 added the kokoro:force-run label Mar 7, 2024

kokoro-team removed the kokoro:force-run label Mar 7, 2024

mpeddada1 approved these changes Mar 8, 2024

View reviewed changes

mpeddada1 changed the title ~~Make image builds reproducible (2)~~ Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue Mar 11, 2024

mpeddada1 merged commit 05e93d0 into GoogleContainerTools:master Mar 11, 2024
8 checks passed

chanseokoh mentioned this pull request Mar 20, 2024

Reproducibility and test suite issues from upgrading to commons-compress 1.22+ #3987

Closed

glasser mentioned this pull request Mar 26, 2024

Publish a release with #4204 #4219

Closed

This was referenced Apr 8, 2024

spring boot 3.2 jib build is not reproducible #4163

Closed

problem achieving reproducibility #4131

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

izogfif commented Mar 2, 2024

izogfif commented Mar 2, 2024

chanseokoh Mar 2, 2024

chanseokoh Mar 2, 2024

chanseokoh left a comment

izogfif commented Mar 3, 2024 •

edited

Loading

chanseokoh left a comment •

edited

Loading

chanseokoh Mar 4, 2024

izogfif Mar 4, 2024 •

edited

Loading

izogfif commented Mar 4, 2024

izogfif commented Mar 5, 2024

izogfif commented Mar 6, 2024

mpeddada1 Mar 6, 2024

izogfif Mar 6, 2024 •

edited

Loading

mpeddada1 commented Mar 6, 2024

izogfif commented Mar 6, 2024 •

edited

Loading

izogfif commented Mar 7, 2024

chanseokoh commented Mar 7, 2024

mpeddada1 commented Mar 7, 2024

izogfif commented Mar 7, 2024 •

edited

Loading

mpeddada1 commented Mar 8, 2024

izogfif commented Mar 9, 2024

mpeddada1 commented Mar 11, 2024

glasser commented Mar 18, 2024

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

Update apache-commons to 1.26.0 and set PAX headers to address build reproducibility issue #4204

Conversation

izogfif commented Mar 2, 2024

izogfif commented Mar 2, 2024

chanseokoh Mar 2, 2024

Choose a reason for hiding this comment

chanseokoh Mar 2, 2024

Choose a reason for hiding this comment

chanseokoh left a comment

Choose a reason for hiding this comment

izogfif commented Mar 3, 2024 • edited Loading

chanseokoh left a comment • edited Loading

Choose a reason for hiding this comment

chanseokoh Mar 4, 2024

Choose a reason for hiding this comment

izogfif Mar 4, 2024 • edited Loading

Choose a reason for hiding this comment

izogfif commented Mar 4, 2024

izogfif commented Mar 5, 2024

izogfif commented Mar 6, 2024

mpeddada1 Mar 6, 2024

Choose a reason for hiding this comment

izogfif Mar 6, 2024 • edited Loading

Choose a reason for hiding this comment

mpeddada1 commented Mar 6, 2024

izogfif commented Mar 6, 2024 • edited Loading

izogfif commented Mar 7, 2024

chanseokoh commented Mar 7, 2024

mpeddada1 commented Mar 7, 2024

izogfif commented Mar 7, 2024 • edited Loading

mpeddada1 commented Mar 8, 2024

izogfif commented Mar 9, 2024

mpeddada1 commented Mar 11, 2024

glasser commented Mar 18, 2024

izogfif commented Mar 3, 2024 •

edited

Loading

chanseokoh left a comment •

edited

Loading

izogfif Mar 4, 2024 •

edited

Loading

izogfif Mar 6, 2024 •

edited

Loading

izogfif commented Mar 6, 2024 •

edited

Loading

izogfif commented Mar 7, 2024 •

edited

Loading