OVA longevity issues

[ghost]

Harbor and Lifecycle teams,

I logged into a 1.3-rc4 dev ova this morning after ~3 weeks uptime after seeing out of memory errors in the ova ui – see attached photo.
Top shows memory usage of 96/7, though no process is above 1% usage.

Docker container names are way off – notice the repeating container id in the container name: https://gist.github.com/anonymous/74abaa2ad26ae2add6d0e897d7800dca
‘Docker compose down’ also fails to remove the containers’

Has anyone in harbor seen an error with docker compose prepending multiple container names?
Can anyone with more Linux experience help me diagnose the memory issue?

[reasonerjt]

You mentioned the env has been up for 3 weeks, but in the gist the containers were created 2 days ago?
And seems there are duplicated containers, and a few containers are gone, could you run docker ps -a and see if some of the containers exited due to error?

[ghost]

I've restarted docker and everything has come up successfully.
There were no additional containers reported by ps -a other than the ones in the gist.

[ywk253100]

What is the version of OVA you are using? The env that you setup is wired, some lines are missing in
the script /etc/vmware/psc/get_token.sh used to generate PSC token. The complete version is here https://github.com/vmware/vic-product/blob/master/installer/build/scripts/psc/get_token.sh. But the Copy harbor token to container mount path part disappears in your script.

[wy65701436]

As for the harbor.service, I notice that it's configured as on failure, and restart on 5 seconds.

This service repeats to restart Harbor in short interval from its log, and it may cause conflict on docker containers, and I guest docker somehow will append the container name if the container is not stop successfully, and be started again.

1,How does this service identify failure?
2,Should we enlarge the interval?
3,Did you do anything during the tests?
4,What's the test scenario?

[ghost]

@ywk253100 the build is off of pr #1300. Good catch with the missing line, let me add that back in. Interestingly, all services start without it.

@wy65701436 i will try increasing the interval for stability purposes and to give docker enough time to teardown services. There are no tests here - the longevity aspect is just that ive left the ova powered on for ~4 weeks now.