Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add file processing status to importer health checks #1892

Closed
steven-sheehy opened this issue Apr 26, 2021 · 0 comments · Fixed by #1993
Closed

Add file processing status to importer health checks #1892

steven-sheehy opened this issue Apr 26, 2021 · 0 comments · Fixed by #1993
Assignees
Labels
enhancement Type: New feature P2 parser Area: File parsing
Milestone

Comments

@steven-sheehy
Copy link
Member

steven-sheehy commented Apr 26, 2021

Problem
If the importer encounters a deadlock or some other scenario that causes it to stop processing new files, it is unable to recover by itself.

Solution

  • Implement a Spring Boot HealthIndicator that checks if new stream files are processing. It can simply inject the MeterRegistry and ensure the counts for each of the stream types is increasing at some reasonable rate.
  • Add new health check to management.endpoint.health.group.liveness.include
  • Add new health check to management.endpoint.health.group.readiness.include

Note that liveness probes are disabled by default for importer since netty doesn't start until after Flyway finishes migrating. We will separately need to solve this issue.

Alternatives
Only use in readiness probe and allow another importer pod take over once leadership election race conditions are handled.

Additional Context

@steven-sheehy steven-sheehy added enhancement Type: New feature parser Area: File parsing P2 labels Apr 26, 2021
@Nana-EC Nana-EC self-assigned this May 14, 2021
@steven-sheehy steven-sheehy added this to the Mirror 0.35.0 milestone May 19, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Type: New feature P2 parser Area: File parsing
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants