Healthcheck not working with MongoDB (Panache)

[pschmidt88]

Describe the bug

It seems that the healthcheck is not working correctly with the mongodb-client and mongodb-panache(-kotlin) extension.

Given wrong credentials/wrong connection string the healthcheck is still UP until some action, endpoint, etc tries to actively access the database. That action will fail and only then the healthcheck will switch to "DOWN".

Expected behavior

When querying the (readiness) healthcheck, it should check the database connection and return a negative result if the connection to the mongodb could not be established.

Actual behavior

The (readiness) check returns a positive result although a wrong connection string / wrong credentials are configured

How to Reproduce?

1. Add mongodb-panache and smallrye-health extensions
2. Configure a wrong database connection (wrong credentials, wrong string, wrong host)
3. Start the quarkus application
4. Access readiness check at http://localhost:8080/q/health/ready

Output of uname -a or ver

Linux pop-os 5.15.11-76051511-generic #202112220937164018548121.10~b3a2c21 SMP Wed Dec 22 15:41:49 U x86_64 x86_64 x86_64 GNU/Linux

Output of java -version

OpenJDK 11.0.13

GraalVM version (if different from Java)

GraalVM CE 21.3.0 (build 11.0.13+7-jvmci-21.3-b05)

Quarkus version or git rev

2.6.2.Final

Build tool (ie. output of mvnw --version or gradlew --version)

6.9

Additional information

No response

[quarkus-bot]

/cc @FroMage, @evanchooly, @jmartisk, @loicmathieu, @xstefank

[loicmathieu]

@pschmidt88 unfortunatly, this is by design.

We instantiate MongoDB client lazilly, so when the rediness check is done but no use of the MongoDB client has been done yet, the readiness check will not find any MongoDB client instances inside our CDI container and respond OK.

The MongoDB health check is a readiness check so it's compatible with the usage of a readiness check (if it was a liveness check, checks should have been done eagerly).

Maybe you can force the intitialization of the MongoDB client using a method listening to the StartupEvent and doing a manyal ping using the MongoDB client, this should do the trick. See https://quarkus.io/guides/lifecycle#listening-for-startup-and-shutdown-events

[geoand]

@loicmathieu maybe we can provide some config option to force eager initialization?

[loicmathieu]

@geoand we discussed it and dismissed it at the time we move to lazy init, we can revisit it but it will make things a little harder.

[geoand]

@loicmathieu I am thinking that if we provide this sort of configuration, it could be a build time option that simply results in generating code with the StartupEvent and manual ping of the client that you mention.
If that approach works, it should be fairly easy to implement.

[loicmathieu]

@geoand there can be multiple clients, both reactive and sync, so it may be a little tricky.
And Agroal also is lazy so I prefere to keep things consistent, with Quarkus: database clients are lazily produced, that's all.

And it's compatible with the fact that we only provide readiness probes and not liveness probes (so a container on k8s will not be restarted if a connection to a database fails). I agree that having a container READY untill the first request triggers the init of the MongoDB client and the container moves to FAIL can be strange but the container is indeed ready to receive a request untill it discover that it cannot serves it ... This is like chicken and egg problem ... This is useful when you start both the application and the database in a distibuted env.

If you really want to eagerly validate the connection, you can create your own Readiness Probe (or a startup probe if you want to automatically restart the container), inject the client in it and do a ping (listDatabase for eg.).

[geoand]

And Agroal also is lazy so I prefere to keep things consistent, with Quarkus: database clients are lazily produced, that's all.

Sure, but does the health check work in the same way for Agroal datasources?
If so, then I think we can leave it as is. If not, we should reconsider.

If you really want to eagerly validate the connection, you can create your own Readiness Probe (or a startup probe if you want to automatically restart the container), inject the client in it and do a ping (listDatabase for eg.).

Do we have docs on how to do that? If not, then we should :)

[loicmathieu]

Not sure how the health check will works on Agroal, see https://github.com/quarkusio/quarkus/blob/main/extensions/agroal/runtime/src/main/java/io/quarkus/agroal/runtime/health/DataSourceHealthCheck.java#L29

This needs to be tested ...

[gsmet]

Could someone describe how it behaves exactly?

Is it not working because there is no usage at all of the client in the code? Or we have this issue even if there is some usage of the client in the code but it hasn't been called yet?

Because I could see how the former could be a desirable behavior but certainly not the latter.

[loicmathieu]

@gsmet as far as I understand, the first readiness check is OK because the application code didn't use the MongoDB client yet so there is no MongoDB client bean inside the CDI container.

After a call to the application, the MongoDB client is build.

Then a new call to the readiness check is NOK as the connection to MongoDB cannot be established.

[pschmidt88]

@gsmet As @loicmathieu already pointed out - there is a client (or in my case the panache repository), but it was not called yet.

[ayhanap]

Not directly related but I'm also looking for a way to initialize MongoDB connection pool at start. Lazy initialization mentioned here causes application to serve first requests slow. Doing all sort of things for fast serverless boot time but this hits.
Will probably fire some basic query in my custom application health check.

[loicmathieu]

@ayhanap to initialize eagerly, you can use a method listening to the startup event https://quarkus.io/guides/lifecycle#listening-for-startup-and-shutdown-events then in this method make a dumb query (or a ping command as we do in the health check).

[nomaster]

I'm also affected by this issue, but I feel unable to implement a fix.

Can anyone provide us with some example code?

[pschmidt88]

This should work

@Startup
@ApplicationScoped
class AppLifecycleBean(
    val mongoClient: MongoClient,
) {
  
    fun pingDatabase(@Observes event: StartupEvent) {
        val databaseName = ConfigProvider.getConfig()
            .getValue("quarkus.mongodb.database", String::class.java)

        mongoClient.getDatabase(databaseName).runCommand(Document("ping", 1))
    }
}

[wiebeck]

@ayhanap to initialize eagerly, you can use a method listening to the startup event https://quarkus.io/guides/lifecycle#listening-for-startup-and-shutdown-events then in this method make a dumb query (or a ping command as we do in the health check).

Well, in theory this works. In (our) practice we use MongoDB Atlas and Vault as credential provider. Vault is able to create credentials from MongoDB Atlas but these credentials are applied on project level and are not valid immediately since they need to get propagated to the cluster first. So introducing a database query on app startup won't help us here (we already failed using @Indexed annotations on fields for the same reason).

[wiebeck]

Addendum for the answer by @pschmidt88: It's sufficient to simply call mongoClient.getDatabase(databaseName) to initialize the mongoClient and with it the health check that properly determines whether the provided credentials (static or dynamic via Vault) are valid. You don't need to actually run a Mongo command. That's what the Quarkus MongoHealthCheck does.