You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem
If the importer encounters a deadlock or some other scenario that causes it to stop processing new files, it is unable to recover by itself.
Solution
Implement a Spring Boot HealthIndicator that checks if new stream files are processing. It can simply inject the MeterRegistry and ensure the counts for each of the stream types is increasing at some reasonable rate.
Add new health check to management.endpoint.health.group.liveness.include
Add new health check to management.endpoint.health.group.readiness.include
Note that liveness probes are disabled by default for importer since netty doesn't start until after Flyway finishes migrating. We will separately need to solve this issue.
Alternatives
Only use in readiness probe and allow another importer pod take over once leadership election race conditions are handled.
Additional Context
The text was updated successfully, but these errors were encountered:
Problem
If the importer encounters a deadlock or some other scenario that causes it to stop processing new files, it is unable to recover by itself.
Solution
HealthIndicator
that checks if new stream files are processing. It can simply inject theMeterRegistry
and ensure the counts for each of the stream types is increasing at some reasonable rate.management.endpoint.health.group.liveness.include
management.endpoint.health.group.readiness.include
Note that liveness probes are disabled by default for importer since netty doesn't start until after Flyway finishes migrating. We will separately need to solve this issue.
Alternatives
Only use in readiness probe and allow another importer pod take over once leadership election race conditions are handled.
Additional Context
The text was updated successfully, but these errors were encountered: