IPFS with cluster-follower run out of memory and got killed by OOM-Job

[RubenKelevra]

Version information:

go-ipfs version: 0.5.0-rc3-8cb67ab
Repo version: 9
System version: amd64/linux
Golang version: go1.14.2

Description:

I run an ipfs client and a cluster-follower on a VM on CentOS 7. This machine has two AMD EPYC 7551 32-Core Processor cores and 1 GB of memory. The machine has 8 GB of swap.

The machine run out of memory, so 8 GB of swap and 1 GB of RAM got filled by IPFS until IPFS got killed from the OOM job of the kernel.

Sadly journalctl dropped more than 100,000 lines from the stack trace, hope it's helpful anyway (it's attached).

The screen in which I had run the cluster-follower also printed a stack trace, I attach this as well.

The cluster I was following is the pacman.store collab cluster.

stacktrace_cluster_follower.txt
out_of_memory_stacktrace_ipfs.txt

[Stebalien]

Swap is basically useless to IPFS and is likely the cause of your problem. What's your swappiness set to?

[RubenKelevra]

What's your swappiness set to?

Swappiness is 90.

Swap is basically useless to IPFS and is likely the cause of your problem.

The machine just runs IPFS and one cluster follower, nothing else (except a sshd) - when IPFS is not running, the machine uses 60 MB of memory.

I understand that Go has a garbage collector which is probably not optimized for daemon like IPFS and might up cluttering the memory into small chunks. But I thought with enough memory pressure the kernel would clean up and no longer used space from go. How is the kernel unable to regain the space used by IPFS and completely fill up the 8 GB swap file with it?

Free swap = 52kB

[Stebalien]

TL;DR: go-ipfs can instantly fill up 1GiB of memory due to ipfs/go-ds-badger#86. When that happens, it's game over.

Also note, with a swappiness of 90 and 1GiB of memory, Linux will start swapping a 10% memory usage. You should probably set it to something like 1 or 10.

However, if you're only running go-ipfs, you should just disable swap.

---

Some context:

Anything that touches swapped out memory will slow to a crawl. Unfortunately, this applies more to closing connections than it does to forming connections because we need to touch a bunch of different state to close a connection while forming a connection usually touches shared state (keys, etc.) likely to be kept in-memory.

Basically, this issue is #1482.

[RubenKelevra]

In this case, we should just disable swapping for IPFS, right? 🤔