Enable importer leader election

[steven-sheehy]

Problem
The importer supports leadership election via configmap locks using Spring Cloud Kubernetes. Despite this, there are still probably race conditions that would cause the database to get out of sync if leader election was enabled. Consider the below scenario:

1. Last file in DB is T1
2. Pod A marked unready while downloading T4, T3 is in queue, and in the middle of parsing T2
3. B elected leader and sees T1 as last and starts downloading T2
4. A finishes parsing T2

Solution
Test multiple importer pods and ensure data processing remains consistent when primary pod is both killed and marked unready. Make changes as necessary to ensure consistency. This is what should occur:

• if A imports T2 successfully, then B downloads T2-T4, B discards T2, A discards T3-T4
• If A imports T2 unsuccessfully, then B downloads T2-T4, B imports T2, A discards T2-T4
• If A and B are processing T2 simultaneously during either scenario, one of them will insert first and the other will fail with an unique constraint violation and be discarded on the next poll.

Tasks:

• Add @Leader around parsers
• Add a stream file existence check inside each parser and discard if it exists
• Set values-prod.yaml to 2 importer replicas

Alternatives

Additional Context