[core] initialize OfflineDatabase asynchronously in DefaultFileSource

[kkaefer]

Sometimes, initialization can take hundreds of milliseconds, in particular when the database doesn't exist yet, or when the app/device is doing a lot of I/O already. Instead of synchronously initializing the OfflineDatabase objects, we're now asynchronously initializing the object through a self-sent message that by virtue of being the first message for this actor guarantees that the object is there when it's needed by other member functions.

For completely fresh app start with no previous database, I'm seeing the synchronous initialization time go down from ~200ms to ~1ms. Of course, the work still needs to happen, but instead it'll only block the DefaultFileSource::Impl thread rather than the main thread that initializes the DefaultFileSource.

General longer term fix is tracked in #9666.

[jfirebaugh]

👍

The main thing I considered in this review was how the change will affect the failure mode in cases like #7920. Prior to this change, if the OfflineDatabase constructor throws an exception, it goes uncaught in the DefaultFileSource thread, aborting the application. Because the initializeOfflineDatabase invocation introduced in this PR is a self-invocation, and invocations do not catch exceptions, that continues to be the case. The stack trace for DefaultFileSource thread at the time of the exception will be slightly different, but not in a material way, and the end result (application abort) should be the same.

There will be a difference in what the main thread is doing at the time of the abort: prior, it will always be waiting on a future. Afterward, it may be at some other arbitrary point in execution. I wouldn't expect this to matter though.

If in the future we change invocation to have built-in exception handling (a try/catch in Mailbox::receive, say), then we will need to consider the effects on code like this carefully. Likely, such a change would need to be accompanied by an actor supervisory system that allows the "owner" of the actor (DefaultFileSource in this case) to detect and recover (e.g. by re-initializing the actor, rather than attempting to send messages to an actor whose database connection is invalid).