Implement Postgres locking

[r4victor]

Closes #1632

This PR implements database-level locking for handling concurrent processing in multi-replica server with Postgres setting. More specifically, it:

• Implements resource locking via select for update.
• Refactors code to be suitable for select for update locking. This includes replacing joinedload's, refetching attributes, ensuring the locks held for the entire transaction duration, splitting submitted jobs processing into two steps (assign instance or no instance/provision instance).
• Refactors in-memory locking to replace custom per-table locks (*_LOCK) with a unified locker interface.
• Fixes bugs with in-memory locking.
• Adds contributing guide on locking implementation.

Tested on SQLite:

• Launch runs on new instances
• Launch runs on existing instances
• Stop runs
• Delete runs
• Create/delete cloud fleets
• Create/delete ssh fleets
• Create/attach/detach/delete volumes
• Create/delete gateways
• Run multi-replica services

Tested on Postgres:

• Launch runs on new instances (multi-replica server)
• Launch runs on existing instances (multi-replica server)
• Launch many concurrent runs (multi-replica server)
• Stop runs (multi-replica server)
• Create/delete cloud fleets (multi-replica server)
• Create/attach/detach/delete volumes (multi-replica server)
• Create/delete gateways (single replica server)
• Run multi-replica services (single replica server)

[jvstme]

@r4victor, thanks for this work. I found 3 unresolved TODOs/FIXMEs apart from compatibility TODOs. Are they all fine to merge?

[jvstme]

Why discard processing multiple instances at once? Isn't it much faster than waiting for the next scheduled process_instances task in 4-6 seconds?

[r4victor]

Before the PR, instances were processed concurrently with asyncio.as_completed which cannot be done when the processing happens in the scope of one transaction (it's not safe to share an sqlalchemy session between coroutines). Processing multiple instances both concurrently and sequentially is also problematic because instance processing can take time (e.g. remote instance waiting) which would lead to blocking processing. The current approach tolerates some instances being slow to process. It's consistent with other processing tasks.

If the processing throughput/latency becomes a problem, we could just decrease processing interval / increase max_instances.

[jvstme]

(nit) And jobs.instance_assigned

[r4victor]

3 unresolved TODOs/FIXMEs

Those left were present before the changes and are not critical.