Add health check for SMT engine disk storage

[ddaspit]

There is no health check for low disk storage. If the disk storage gets too low, then SMT engines will be negatively impacted. We can use DiskStorageHealthCheck from the AspNetCore.HealthChecks.System library.

[johnml1135]

This is done through Alerts and Prometheus. It is currently working as of this week (the email was improperly configured). Errors come as emails and look like this:

This needs no further action at this time.

[ddaspit]

I would like to keep this open. It doesn't seem right to me that we return a "healthy" status from the health check endpoint when SMT suggestions do not work, because of low/no disk space.

[johnml1135]

One option is to have "degraded" if we are getting GRPC errors (which we were) and keep it on for 30 minutes or so...

[ddaspit]

It would be easier to just check the disk space, especially since there is already a health check library that can do it.

[johnml1135]

Ok - it will be a redundant check, but I can see the value.

[Enkidu93]

In order to use this health check, we have to reference a specific drive by string (e.g. "C://"). How will this work in containers? Is this workable, @johnml1135?

[johnml1135]

I am assuming that the main paths /var/lib/serval and /var/lib/machine would need to be monitored. They are defined in appsettings.json:

  "DataFile": {
    "FilesDirectory": "/var/lib/serval/files"
  }

and

  "SmtTransferEngine": {
    "EnginesDir": "/var/lib/machine/engines"
  }