Cron uses 100% CPU with 2022.04 on Raspberry

[mhavelant]

This is a: Bug

Details

I updated from 2022.02.1 to 2022.04 just now, and noticed 100% CPU usage for the container.
Reverting to 2022.02.1 fixed it.

Related Issues

• I have searched this repository/Pi-hole forums for existing issues and pull requests that look similar

How to reproduce the issue

1. Environment data

• Operating System: Raspbian (Linux raspberrypi 5.10.103-v7+)
• Hardware: RasPi 3B
• Kernel Architecture: armv7l
• Docker Install Info and version:
  • Software source: Official docker-ce 20.10.14
  • Supplimentary Software: Docker Compose v2.3.3; docker swarm
• Hardware architecture: ARMv7

2. docker-compose.yml contents, docker run shell command, or paste a screenshot of any UI based configuration of containers here
3. any additional info to help reproduce

Test-running it with docker run -d --name tmppihole pihole/pihole:2022.04

Output of docker top:

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                24181               24152               0                   11:45               ?                   00:00:00            s6-svscan -t0 /var/run/s6/services
root                24302               24181               0                   11:45               ?                   00:00:00            s6-supervise s6-fdholderd
root                24834               24181               0                   11:46               ?                   00:00:00            s6-supervise lighttpd
root                24835               24181               0                   11:46               ?                   00:00:00            s6-supervise pihole-FTL
root                24836               24181               0                   11:46               ?                   00:00:00            s6-supervise lighttpd-access-log
root                24837               24181               0                   11:46               ?                   00:00:00            s6-supervise cron
root                24838               24181               0                   11:46               ?                   00:00:00            s6-supervise lighttpd-error-log
root                24842               24834               0                   11:46               ?                   00:00:00            bash ./run
root                24865               24837               0                   11:46               ?                   00:00:00            bash ./run
www-data            24870               24842               0                   11:46               ?                   00:00:00            lighttpd -D -f /etc/lighttpd/lighttpd.conf
root                24875               24865               99                  11:46               ?                   00:00:40            /usr/sbin/cron -f
www-data            24896               24870               0                   11:46               ?                   00:00:00            /usr/bin/php-cgi
www-data            24907               24896               0                   11:46               ?                   00:00:00            /usr/bin/php-cgi
www-data            24908               24896               0                   11:46               ?                   00:00:00            /usr/bin/php-cgi
www-data            24909               24896               0                   11:46               ?                   00:00:00            /usr/bin/php-cgi
www-data            24910               24896               0                   11:46               ?                   00:00:00            /usr/bin/php-cgi

With the most relevant line being

root                24875               24865               99                  11:46               ?                   00:00:40            /usr/sbin/cron -f

Output of docker stats:

29ae0076388a   tmppihole                                     106.11%   10.32MiB / 966.8MiB   1.07%     75.1kB / 8.78kB   0B / 0B     16

I waited for about 10-15 minutes, hoping it's just some long-running startup thing, but no luck, still 100% CPU usage.

Tried it on my main machine, Pop!_OS 21.04, it seems to work there better.

These common fixes didn't work for my issue

• I have tried removing/destroying my container, and re-creating a new container
• I have tried fresh volume data by backing up and moving/removing the old volume data
• I have tried running the stock docker run example(s) in the readme (removing any customizations I added)
• I have tried a newer or older version of Docker Pi-hole (depending what version the issue started in for me)
• I have tried running without my volume data mounts to eliminate volumes as the cause

If the above debugging / fixes revealed any new information note it here.
Add any other debugging steps you've taken or theories on root cause that may help.

[PromoFaux]

Are you able to reproduce this with 2022.04.1?

[mhavelant]

I just tried it, too, sadly it's still 100%-ing the CPU.

[julhei]

I observed the same high CPU usage with 2022.04 on Raspberry

Output of docker top pihole | grep cron

root                19606               19083               0                   15:36               ?                   00:00:00            s6-supervise cron
root                19624               19614               99                  15:36               ?                   00:12:58            /usr/sbin/cron -f

One of the cron processes utilizes 99% CPU.

Output of strace -p 19624

syscall_0x197(0, 0, 0x7e884a50, 0x7e884a60, 0, 0x76f5d000) = -1 EPERM (Operation not permitted)

[PromoFaux]

Unable to repro this on a Pi 4 (32 bit Bullseye):

CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O     PIDS
cfe106de71c1   tmppihole   0.01%     0B / 0B             0.00%     847kB / 7.87kB   8.19kB / 0B   28

Pi 3b (64 bit Bullseye)

CONTAINER ID   NAME        CPU %     MEM USAGE / LIMIT   MEM %     NET I/O          BLOCK I/O    PIDS
3507c3a7d4f8   tmppihole   0.02%     0B / 0B             0.00%     947kB / 31.9kB   209kB / 0B   23

Is there any more detail you can give about your systems? Docker run commands/compose files?

[mhavelant]

Mine is a Pi3 B+, running this:

cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"

uname -a
Linux raspberrypi 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux

(so 32bit)

[hugalafutro]

Run the container as privileged. I have 3 pi 3b+ with identical docker+docker-compose version and docker-compose.yml, one of them does this and spams

pihole  | sudo: error in event loop: Operation not permitted
pihole  | sudo: unexpected child termination condition: 0

while running unprivileged. Other 2 don't do this. No idea why.

[rdwebdesign]

Maybe a comparison between them can bring us more information.

Which OS are you running on each of your 3 raspberry pi 3B+?

[hugalafutro]

pi1 is the one that does what's described in this issue

OS:
pi1: Linux pi-hole 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
pi2: Linux pi-hole2 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
pi3: Linux fr24 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux

all 3 use Docker Compose version v2.4.0
all 3 use this docker version:

Client: Docker Engine - Community
 Version:           20.10.14
 API version:       1.41
 Go version:        go1.16.15
 Git commit:        a224086
 Built:             Thu Mar 24 01:48:21 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.14
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.15
  Git commit:       87a90dc
  Built:            Thu Mar 24 01:46:07 2022
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.5.11
  GitCommit:        3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc:
  Version:          1.0.3
  GitCommit:        v1.0.3-0-gf46b6ba
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker-compose.yml
pi1: https://o.o5.ddns.net/Xsijc
pi2: https://o.o5.ddns.net/pdcRX
pi3: https://o.o5.ddns.net/W0j2l

edit: the env variables are mostly the same like this:

PIHOLE_PASSWD=[REDACTED]
TZ=Europe/London

DNS_1=192.168.1.1

COND_FORWARD=true
ROUTER_DOMAIN=lan
ROUTER_IP=192.168.1.1
ROUTER_REVERSE=192.168.1.0/24

SERVER_IP=192.168.1.101
SERVER_IP6=

V_HOST=pi-hole

with PIHOLE_PASSWD, SERVER_IP and V_HOST being the only 3 that change between the instances

[rdwebdesign]

I can't understand... I was expecting to see a difference here.
Your pi1 uses the same OS and same docker version than pi2, but just one is failing.

[hugalafutro]

Indeed. If anything, I'd expect pi3 making problems, as pi2 is a literal clone of pi1 with changed hostname, whereas pi3 is a flightradar24 install with pihole in docker slapped on top. Or maybe the issue relates to something we didn't think to compare.

[julhei]

I noticed PIHOLE_BASE has been updated from buster-slim to bullseye-slim here ee20e52.

As per debuerreotype/docker-debian-artifacts#106 (comment)

... bullseye is now using the 64bit time syscalls on 32bit userspace, so you need to make sure your Docker is sufficiently new and you'll need a newer libseccomp than is probably available in the stable distro you're running on the host.

I upgraded libseccomp2 to version 2.5.1 from backports following the approach described here under option 2, and more general here. After restarting the pihole container, CPU utilization was back to known, low values and the not permitted syscalls are gone.

Can you check your libseccomp2 versions on your systems?

$ apt list libseccomp2 -a
libseccomp2/now 2.5.1-1~bpo10+1 armhf [installed,local]
libseccomp2/oldstable 2.3.3-4 armhf

[hugalafutro]

That was it!
The offending pi had libseccomp2/oldstable,now 2.3.3-4 armhf [installed] while the 2 working ones had libseccomp2/buster-backports,now 2.5.1-1~bpo10+1 armhf [installed]

I followed the Option 2 from the link and all is well after container rebuild without privileged
I now recall having to do this for different container on the other pies, just completely forgot about it/not needed it for any container on pi1 yet.

[mhavelant]

Jackpot for me as well, installing the backport fixed the obscene CPU usage, thanks!

[mhavelant]

I think we can close the issue then. Or we could make a readme update to tell people about this, and close the issue with the PR.

[pralor-bot]

This issue has been mentioned on Pi-hole Userspace. There might be relevant details there:

https://discourse.pi-hole.net/t/in-docker-with-compose-there-is-a-issue-where-date-cannot-be-pulled/54837/1

[MayeulC]

I ran into this as well, but upgrading libseccomp2 didn't fix anything (I made sure the correct version is installed, and even rebooted the raspberrypi).
Upgrading libseccomp2 allowed me to get rid of the following log lines, though:

sudo: unexpected child termination condition: 0
sudo: error in event loop: Operation not permitted

However, to solve the cron issue, I decided to follow this and override the seccomp policy for now. I'll upgrade the distribution later. Doing this solved my issue with cron taking 100% of one core and strace -p $PID being full of EPERM.

For reference, here is how I launch the container:

docker run -d \
    --name pihole \
    -e DNS1=1.1.1.1 \
    -e DNS2=10.60.4.1 \
    -e ServerIP=10.60.4.3 \
    -e ServerIPv6=2a01:xxx:xxx:xxx::yyy \
    -e WEBPASSWORD=xxxxx \
    -e TZ='Europe/Paris' \
    -e PIHOLE_INTERFACE=eth0 \
    -e DNSMASQ_USER=root \
    -v "/root/docker-containers/pihole/conf-volumes/pihole/:/etc/pihole/" \
    -v "/root/docker-containers/pihole/conf-volumes/dnsmasq.d/:/etc/dnsmasq.d/" \
    --dns=127.0.0.1 --dns=1.1.1.1 \
    --restart=unless-stopped \
    --cap-add=NET_ADMIN \
    -p 8314:80 \
    -p 8315:443 \
    -p 67:67 \
    -p 53:53/tcp \
    -p 53:53/udp \
    pihole/pihole:latest

[lemmy04]

Updated my pi that runs pihole in docker from buster to bullseye, load is down to normal.