I ran out of disk space with systemdspwaner

[marouenbg]

I am using systemdspawner on Ubuntu 18.04 and after running the server for a while ~ few week,s I run out of disk space, I traced this back to the /var/lib/private folder, which seems to be inaccessible even after changing permissions.
Running ls shows that thiss folder contains user info named after token like 7f7564c7-0f38-40fa-a768-2eddd9c43571
I thought the private space is deleted when user is logged of.
I am using systemdspwaner with tmpauthenticator and the following parameters:

c.JupyterHub.spawner_class = systemdspawner.SystemdSpawner
c.JupyterHub.port = 80
c.SystemdSpawner.dynamic_users = True
c.Spawner.args = ['--NotebookApp.default_url=notebooks/Welcome_to_netBooks.ipynb']
c.SystemdSpawner.disable_user_sudo = True
c.SystemdSpawner.readonly_paths = ['/']
c.SystemdSpawner.isolate_tmp = True
c.SystemdSpawner.isolate_devices = True

I also cull idle sessions using idle-culler:

c.JupyterHub.services = [
{
"name": "jupyterhub-idle-culler-service",
"command": [
sys.executable,
"-m", "jupyterhub_idle_culler",
"--timeout=1200","--cull-users=True","--max-age=3600","--remove-named-servers=True","--cull-every=30"
],
"admin": True,
}
]

Any thoughts on how to parametrize Jupyter to remove user disk space after session ends?

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[yuvipanda]

@marouenbg thanks for reporting this. It looks like the home directories of the dynamic users aren't getting cleaned up when they are removed. Does that sound right?

This isn't something systemdspawner does by default, not sure how exactly do add this on...

[marouenbg]

Hey @yuvipanda :)
Yes that is exactly what is happening.
I will crontab a bash script to clean that folder once in a while because I do have some traffic on the server.
If you have any better idea, please let me know !

[behrmann]

Yes, you can solve this either with a timer (or a cronjob, but timer's are a bit more robust) yourself or you can hook up another unit to be called OnFailure=/OnSuccess= to clean up the directory when the server unit exits (that unfortunately needs a somewhat recent version of systemd, since OnSuccess= was only added in version 249) or you could add a tmpfiles snippet (see man tmpfiles.d) for the directory being added to /etc/tmpfiles.d, since that supports regular cleanup via systemd-tmpfiles-clean.timer.

[manics]

KubeSpawner already has a similar feature:
jupyterhub/kubespawner#475

[marouenbg]

Thank you all, I will run some tests and get back to you, closing for now!

[marouenbg]

For the record, I was able to set this up using a timer on files older than a certain time in /var/lib/private/
https://unix.stackexchange.com/questions/22674/shell-script-for-moving-oldest-files