Docker image fails during initdb

[alf]

When I try to start the docker image it fails almost immediately:

$ docker run --name timescaledb --rm -it -e POSTGRES_PASSWORD=pass timescale/timescaledb-ha:pg14-latest 
popen failure: Cannot allocate memory
initdb: error: The program "postgres" is needed by initdb but was not found in the
same directory as "/usr/lib/postgresql/14/bin/initdb".
Check your installation.

Since I haven't set any resource limits and I'm running this on my laptop with a lot of available memory, I suspect the error message is misleading. I dig some further digging by starting a container and enter it:

docker run --name timescaledb --rm -it -e POSTGRES_PASSWORD=pass timescale/timescaledb-ha:pg14-latest bash

From this shell we can reproduce the original error:

postgres@03797ae55395:~$ initdb
popen failure: Cannot allocate memory
initdb: error: The program "postgres" is needed by initdb but was not found in the
same directory as "/usr/lib/postgresql/14/bin/initdb".
Check your installation.
postgres@03797ae55395:~$ 

I'm no expert, but as I understand it the ulimit command tells us of any resource constraints:

postgres@03797ae55395:~$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 255666
max locked memory           (kbytes, -l) 64
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1048576
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) 8192
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) unlimited
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

When I run strace initdb I find the following lines quite interesting. As far as I can tell, initdb tries to execute the postgres command in a subprocess but fails:

...
newfstatat(AT_FDCWD, "/usr/lib/postgresql/14/bin/postgres", {st_mode=S_IFREG|0755, st_size=9489216, ...}, 0) = 0
access("/usr/lib/postgresql/14/bin/postgres", R_OK) = 0
access("/usr/lib/postgresql/14/bin/postgres", X_OK) = 0
pipe2([3, 4], O_CLOEXEC)                = 0
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
prlimit64(0, RLIMIT_NOFILE, NULL, {rlim_cur=1024*1024, rlim_max=1024*1024}) = 0
mmap(NULL, 36864, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7fc170d95000
rt_sigprocmask(SIG_BLOCK, ~[], [], 8)   = 0
clone3({flags=CLONE_VM|CLONE_VFORK, exit_signal=SIGCHLD, stack=0x7fc170d95000, stack_size=0x9000}, 88) = -1 EPERM (Operation not permitted)
munmap(0x7fc170d95000, 36864)           = 0
...

Here's the entire output, in case my snippet doesn't include the necessary details: strace_initdb.txt.

This is all on my laptop:

$ uname -a
Linux powerpad 5.10.111 #1-NixOS SMP Wed Apr 13 19:01:11 UTC 2022 x86_64 GNU/Linux

$ docker version
Client:
 Version:           20.10.9
 API version:       1.41
 Go version:        go1.16.13
 Git commit:        v20.10.9
 Built:             Thu Jan  1 00:00:00 1970
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.9
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.13
  Git commit:       v20.10.9
  Built:            Tue Jan  1 00:00:00 1980
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.11
  GitCommit:        v1.4.11
 runc:
  Version:          1.0.2
  GitCommit:        
 docker-init:
  Version:          0.18.0
  GitCommit:        

Any idea how to proceed? The old alpine-images works great for me, but since they're going away I gather it's best if we can get this up and running.

[alf]

I believe my issues are related to this: moby/moby#42680

In other words, it seems that I must update to docker >= 20.10.10 to use the new image.

The alternative workaround is to run the image with the --privileged flag, however this is far from ideal.

[alf]

Can confirm that the image works for me after upgrading docker.

[F3lixP4lm]

Thanks for your tip @alf , haven't found any solution on the internet except yours. Would never have thought to check the Docker version....

[rijnhard]

tried on docker 20.10.17 and docker 20.10.21 and same issue with pg14-latest

and its not kernel limits on the host..

kernel.shmall = 18446744073692774399
kernel.shmmax = 18446744073692774399

and --privileged also doesn't work.

edit:
did an strace, and exactly the same clone3 error...

[rijnhard]

I'm running this in jelastic though, and have a funny feeling it could be outdated versions.

uname -a
Linux node4696-tag-data-swarm.za.cloudlet.cloud 3.10.0 #1 SMP Thu Sep 23 18:26:47 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[bitcoinbrisbane]

Upgrading fixed it for me too.

[rijnhard]

I'm running this in jelastic though, and have a funny feeling it could be outdated versions.

uname -a
Linux node4696-tag-data-swarm.za.cloudlet.cloud 3.10.0 #1 SMP Thu Sep 23 18:26:47 MSK 2021 x86_64 x86_64 x86_64 GNU/Linux

docker version
Client: Docker Engine - Community
 Version:           20.10.17
 API version:       1.41
 Go version:        go1.17.11
 Git commit:        100c701
 Built:             Mon Jun  6 23:05:12 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.17
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.11
  Git commit:       a89b842
  Built:            Mon Jun  6 23:03:33 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.9
  GitCommit:        ea765aba0d05254012b0b9e595e995c09186427f
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Forgot to update.
It was the runc version, it needed to be v1.3+ in my case I had to get the Vendor to update their docker runtimes dependencies and it worked.