Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Latest Docker Image Corrupted? #7

Closed
dmlb2000 opened this issue May 7, 2020 · 27 comments
Closed

Latest Docker Image Corrupted? #7

dmlb2000 opened this issue May 7, 2020 · 27 comments

Comments

@dmlb2000
Copy link

dmlb2000 commented May 7, 2020

Jeff,

I think the latest image that got pushed to hub.docker.com has a corrupted RPM database in it for some reason.

$ docker run -it --rm geerlingguy/docker-centos8-ansible:latest dnf -y install openssh-cl
ients
error: rpmdb: damaged header #173 retrieved -- skipping.
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
CentOS-8 - AppStream                                                                    2.0 MB/s | 7.0 MB     00:03
CentOS-8 - Base                                                                         986 kB/s | 2.2 MB     00:02
CentOS-8 - Extras                                                                       6.6 kB/s | 5.9 kB     00:00
Extra Packages for Enterprise Linux Modular 8 - x86_64                                   70 kB/s | 116 kB     00:01
Extra Packages for Enterprise Linux 8 - x86_64                                          774 kB/s | 6.5 MB     00:08
Dependencies resolved.
========================================================================================================================
 Package                        Architecture          Version                               Repository             Size
========================================================================================================================
Installing:
 openssh-clients                x86_64                8.0p1-4.el8_1                         BaseOS                704 k
Installing dependencies:
 fipscheck                      x86_64                1.5.0-4.el8                           BaseOS                 28 k
 fipscheck-lib                  x86_64                1.5.0-4.el8                           BaseOS                 16 k
 libedit                        x86_64                3.1-23.20170329cvs.el8                BaseOS                102 k
 openssh                        x86_64                8.0p1-4.el8_1                         BaseOS                496 k

Transaction Summary
========================================================================================================================
Install  5 Packages

Total download size: 1.3 M
Installed size: 5.8 M
Downloading Packages:
(1/5): fipscheck-lib-1.5.0-4.el8.x86_64.rpm                                              86 kB/s |  16 kB     00:00
(2/5): fipscheck-1.5.0-4.el8.x86_64.rpm                                                 117 kB/s |  28 kB     00:00
(3/5): libedit-3.1-23.20170329cvs.el8.x86_64.rpm                                        300 kB/s | 102 kB     00:00
(4/5): openssh-8.0p1-4.el8_1.x86_64.rpm                                                 1.2 MB/s | 496 kB     00:00
(5/5): openssh-clients-8.0p1-4.el8_1.x86_64.rpm                                         1.7 MB/s | 704 kB     00:00
------------------------------------------------------------------------------------------------------------------------
Total                                                                                   990 kB/s | 1.3 MB     00:01
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
warning: /var/cache/dnf/BaseOS-f6a80ba95cf937f2/packages/fipscheck-1.5.0-4.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS-8 - Base                                                                         1.6 MB/s | 1.6 kB     00:00
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security@centos.org>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
Key imported successfully
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
Import of key(s) didn't help, wrong key(s)?
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
Public key for fipscheck-1.5.0-4.el8.x86_64.rpm is not installed. Failing package is: fipscheck-1.5.0-4.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Public key for fipscheck-lib-1.5.0-4.el8.x86_64.rpm is not installed. Failing package is: fipscheck-lib-1.5.0-4.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Public key for libedit-3.1-23.20170329cvs.el8.x86_64.rpm is not installed. Failing package is: libedit-3.1-23.20170329cvs.el8.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Public key for openssh-8.0p1-4.el8_1.x86_64.rpm is not installed. Failing package is: openssh-8.0p1-4.el8_1.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Public key for openssh-clients-8.0p1-4.el8_1.x86_64.rpm is not installed. Failing package is: openssh-clients-8.0p1-4.el8_1.x86_64
 GPG Keys are configured as: file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'dnf clean packages'.
Error: GPG check FAILED
@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

This doesn't seem like an issue with the upstream container.

$ docker run -it --rm centos:8 dnf -y install openssh-clients
CentOS-8 - AppStream                                                                    2.6 MB/s | 7.0 MB     00:02
CentOS-8 - Base                                                                         919 kB/s | 2.2 MB     00:02
CentOS-8 - Extras                                                                       5.2 kB/s | 5.9 kB     00:01
Dependencies resolved.
========================================================================================================================
 Package                        Architecture          Version                               Repository             Size
========================================================================================================================
Installing:
 openssh-clients                x86_64                8.0p1-4.el8_1                         BaseOS                704 k
Installing dependencies:
 fipscheck                      x86_64                1.5.0-4.el8                           BaseOS                 28 k
 fipscheck-lib                  x86_64                1.5.0-4.el8                           BaseOS                 16 k
 libedit                        x86_64                3.1-23.20170329cvs.el8                BaseOS                102 k
 openssh                        x86_64                8.0p1-4.el8_1                         BaseOS                496 k

Transaction Summary
========================================================================================================================
Install  5 Packages

Total download size: 1.3 M
Installed size: 5.8 M
Downloading Packages:
(1/5): fipscheck-lib-1.5.0-4.el8.x86_64.rpm                                              47 kB/s |  16 kB     00:00
(2/5): fipscheck-1.5.0-4.el8.x86_64.rpm                                                  79 kB/s |  28 kB     00:00
(3/5): libedit-3.1-23.20170329cvs.el8.x86_64.rpm                                        180 kB/s | 102 kB     00:00
(4/5): openssh-8.0p1-4.el8_1.x86_64.rpm                                                 593 kB/s | 496 kB     00:00
(5/5): openssh-clients-8.0p1-4.el8_1.x86_64.rpm                                         667 kB/s | 704 kB     00:01
------------------------------------------------------------------------------------------------------------------------
Total                                                                                   745 kB/s | 1.3 MB     00:01
warning: /var/cache/dnf/BaseOS-f6a80ba95cf937f2/packages/fipscheck-1.5.0-4.el8.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS-8 - Base                                                                         1.6 MB/s | 1.6 kB     00:00
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security@centos.org>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                1/1
  Installing       : fipscheck-lib-1.5.0-4.el8.x86_64                                                               1/5
  Running scriptlet: fipscheck-lib-1.5.0-4.el8.x86_64                                                               1/5
  Installing       : fipscheck-1.5.0-4.el8.x86_64                                                                   2/5
  Running scriptlet: openssh-8.0p1-4.el8_1.x86_64                                                                   3/5
  Installing       : openssh-8.0p1-4.el8_1.x86_64                                                                   3/5
  Installing       : libedit-3.1-23.20170329cvs.el8.x86_64                                                          4/5
  Installing       : openssh-clients-8.0p1-4.el8_1.x86_64                                                           5/5
  Running scriptlet: openssh-clients-8.0p1-4.el8_1.x86_64                                                           5/5
  Verifying        : fipscheck-1.5.0-4.el8.x86_64                                                                   1/5
  Verifying        : fipscheck-lib-1.5.0-4.el8.x86_64                                                               2/5
  Verifying        : libedit-3.1-23.20170329cvs.el8.x86_64                                                          3/5
  Verifying        : openssh-8.0p1-4.el8_1.x86_64                                                                   4/5
  Verifying        : openssh-clients-8.0p1-4.el8_1.x86_64                                                           5/5

Installed:
  openssh-clients-8.0p1-4.el8_1.x86_64        fipscheck-1.5.0-4.el8.x86_64       fipscheck-lib-1.5.0-4.el8.x86_64
  libedit-3.1-23.20170329cvs.el8.x86_64       openssh-8.0p1-4.el8_1.x86_64

Complete!

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

Here's the image ID for my image anyway.

$ docker images | grep docker-centos
geerlingguy/docker-centos8-ansible                                                    latest                        f4fd87b872f0        4 hours ago         522MB

@dmlb2000 dmlb2000 mentioned this issue May 7, 2020
Closed
@geerlingguy
Copy link
Owner

Gah... this is probably related to #5

And I've been debugging the rpmdb issue quite a while over in geerlingguy/ansible-role-kubernetes#67

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

I'm thinking the update pushing the new version of the rpm package maybe an issue on some docker environments.

@geerlingguy
Copy link
Owner

Ah... and just found this:

Please reconsider this. For CI testing in Nmstate, we use stock base images that setup with yum/dnf to contain the requirements for the test environment. It seems that using yum/dnf in two layers corrupts the RPM db (when building the image on docker hub or quay). This then breaks installing test RPM packages during the CI run. Rebuilding the rpmddb would fix the DB. Ideally, the DB would not break in the first place but I guess this might be related. I re-opened this issue so this comment does not get lost.

From: https://bugzilla.redhat.com/show_bug.cgi?id=1680124

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

My thought is to let CentOS manage the update of their base image. You just keep your image down to installing Python bits you need and ansible. And yes if you have to install packages and updates do it in one command.

RUN dnf -y update && dnf -y install stuff && dnf clean all

thoughts?

geerlingguy added a commit that referenced this issue May 7, 2020
@geerlingguy
Issue #7: Avoid updating everything.
156d5ce
@geerlingguy
Copy link
Owner

Yeah, I think I'll go that route. For CentOS 6 and 7, I always ran a general yum -y update on the image no matter what, but I'm okay just relying on upstream image to be up to date for prepackaged things.

Just pushed up a commit that may fix it, but it will be 20-30 min before it's built on Docker Hub.

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

@geerlingguy Thanks for the help, these images are really useful!

@geerlingguy geerlingguy mentioned this issue May 7, 2020
Open
@geerlingguy
Copy link
Owner

@dmlb2000 - New image is up; can you see if it works for you now?

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

@geerlingguy I'm able to install things, but we are having issues with GPG keys now... so an update doesn't seem to work for example and I refuse to think the GPG key for glibc-common is not installed.

@geerlingguy
Copy link
Owner

Before:

# yum --version
4.2.7
error: rpmdbNextIterator: skipping h#     173 blob size(4836): BAD, 8 + 16 * il(70) + dl(3708)
  Installed: dnf-0:4.2.7-7.el8_1.noarch at Mon 13 Jan 2020 09:49:19 PM GMT
  Built    : CentOS Buildsys <bugs@centos.org> at Thu 19 Dec 2019 03:44:23 PM GMT

  Installed: rpm-0:4.14.2-26.el8_1.x86_64 at Thu 07 May 2020 04:50:58 PM GMT
  Built    : CentOS Buildsys <bugs@centos.org> at Thu 09 Apr 2020 06:59:01 PM GMT

After:

# yum --version
Failed to set locale, defaulting to C.UTF-8
4.2.7
  Installed: dnf-0:4.2.7-7.el8_1.noarch at Mon Jan 13 21:49:19 2020
  Built    : CentOS Buildsys <bugs@centos.org> at Thu Dec 19 15:44:23 2019

  Installed: rpm-0:4.14.2-25.el8.x86_64 at Mon Jan 13 21:49:16 2020
  Built    : CentOS Buildsys <bugs@centos.org> at Fri Nov  8 22:56:14 2019

So it seems it's not corrupt out of the box, at least.

And yeah, I'm seeing GPG key issues too—over in geerlingguy/ansible-role-kubernetes#67 I re-ran the tests but it's still just adding GPG keys over and over and getting some errors there in CI :(

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

I'm guessing there's some sort of package we are missing to install.

@geerlingguy
Copy link
Owner

Just testing a yum install -y wget:

Total                                                                                   1.3 MB/s | 852 kB     00:00     
warning: /var/cache/dnf/AppStream-02e86d1c976ab532/packages/wget-1.19.5-8.el8_1.1.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS-8 - AppStream                                                                    1.6 MB/s | 1.6 kB     00:00    
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security@centos.org>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Import of key(s) didn't help, wrong key(s)?

So I found: https://bugs.centos.org/view.php?id=16655 — which is pretty much the same issue.

I tried on the centos:8 container and it worked fine (docker run -it --rm centos:8 /bin/bash).

geerlingguy added a commit that referenced this issue May 7, 2020
@geerlingguy
Issue #7: Don't mess with yum's cache at all.
7b0f7b8
@geerlingguy
Copy link
Owner

I just pushed another commit that basically strips the cache-related operations, and just does a yum install in the container. Maybe that'll fix things? In any case, it seems like there's a very annoying bug in CentOS 8 in the container, that didn't exist prior to ~April 22, that causes DNF/YUM to explode whenever you do certain things... not sure exactly what, but it seems like every other related bug report I've found has the same condition:

  • Someone running the centos8 docker image
  • Someone doing something with yum / gpg / dnf / rpmdb
  • And it blows up the rpmdb, and only clue is maybe it's related to the overlayfs in Docker...

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020 •
edited
Loading

Yup seems like the base image without the cache manipulations works, you can run for example...

docker run -it --rm centos:8 /bin/bash -c 'dnf -y update && dnf -y install python3-pip'

But it doesn't work on your image, might be the cache issue.

@geerlingguy
Copy link
Owner

Weird. Image is built, I pulled it, and still getting the GPG issues. Now I wonder if one of the packages here is causing the issue:

# Install requirements.
RUN yum -y install \
      epel-release \
      initscripts \
      sudo \
      which \
      hostname \
      python3 \
      python3-pip

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

Try adding the dnf -y update && just before your install, same layer?

@geerlingguy
Copy link
Owner

The plot thickens... If I build the image on my local workstation (exact same Dockerfile), and then run:

docker run --rm centos8-ansible bash -c "yum install -y wget"

It succeeds.

If I build the image on Travis CI or Docker Hub and do the same, I get the GPG key errors.

So something about the Docker installation/configuration on Travis CI and Docker Hub seems to be causing this issue. Grr.

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

It's the Linux hosts they run to do the build? Do you run a Mac?

@geerlingguy
Copy link
Owner

Yeah I'm on macOS.

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

Yup ran it on my CentOS 7 server and reproduced the GPG error locally

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020 

# Install requirements.
RUN dnf -y install rpm centos-release && \
    dnf -y update && \
    dnf -y install \
      epel-release \
      initscripts \
      sudo \
      which \
      hostname \
      python3 \
      python3-pip

Seems to not produce GPG errors on my CentOS 7 server

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

Then I can run a dnf -y install wget on the built image and it works without rpmdb errors.

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020 

RUN dnf -y install rpm centos-release && \
    dnf -y install \
      epel-release \
      initscripts \
      sudo \
      which \
      hostname \
      python3 \
      python3-pip

Seems like removing the update also works.

dmlb2000 added a commit to dmlb2000/docker-centos8-ansible that referenced this issue May 7, 2020
@dmlb2000
Resolve Corrupted RPM Database
c64ed9f 
The CentOS guys need to rebuild their image so that updates don't
apply later and corrupt their base image.

This fix seems to work for me.

Fix geerlingguy#7

Signed-off-by: David Brown <dmlb2000@gmail.com>
@dmlb2000 dmlb2000 mentioned this issue May 7, 2020
Closed
geerlingguy added a commit that referenced this issue May 7, 2020
@geerlingguy
Issue #7: Try fix from dmlb2000 for updates.
e38b7ea
@geerlingguy
Copy link
Owner

@dmlb2000 Testing that on Docker Hub... thanks for helping debug this!

@dmlb2000
Copy link
Author

dmlb2000 commented May 7, 2020

@geerlingguy Just pulled the image seems to work fine now!

@geerlingguy
Copy link
Owner

Yay! Confirmed here too.

@hluaces hluaces mentioned this issue Jun 17, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Resolve Corrupted RPM Database dmlb2000/docker-centos8-ansible
2 participants
@geerlingguy @dmlb2000