WIP: Batch disk operations #7895
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hickeng
changed the title
5 tasks
1 task
hickeng
commented
Jun 16, 2018
| @@ -98,6 +99,22 @@ func main() { | |||
| extraconfig.SetLogLevel(log.DebugLevel) | |||
| } | |||
|
|
|||
| if vchConfig.Diagnostics.DebugLevel > 4 { | |||
There was a problem hiding this comment.
This is here solely to allow capture of the raw soap requests issued to vSphere.
This may be a useful capability to keep, but likely needs tidying up somewhat.
|
The disk-specific portions of this should be preserved, but the batching logic should be replaced with the general-purpose batching from #8034 . |
zjs
added
kind/defect/performance
This adds disk batching to the endpointVM so operations such as volume filesystem creation can occur in parallel. It also adjusts the volume cache to remove a lock that seralizes all creates. This is NOT complete as of this commit - it requires singleflight or similar wrapping the volume create path, and also consideration as to the other inspect/delete operations as they apply to a partially completed volume. This should also allow an operation to specify if it can tolerate some latency. The desire behind this is so that disk detach operations for pull can be deferred and batched with the attach operation for the subsequent child disk which should halve the number of reconfigure operations associated with a pull request.
This adds singleflight for device listing to avoid large numbers of duplicate calls after having batched disk reconfigure operations. This also adds debug logic for capturing the govmomi soap output at debug > 4
This was a minor change to resolve a compile error from failure to reference the `shared` return value from singleflight. I don't know if that's actually needed, but adjusted to preserve both the TODO comment and a separate return path based on shared vs unique result. Corrects a missing package name in a type reference, and a spelling mistake to allow for compilation. A basic test at this point shows we can pull images, create containers with volumes attached, and parallel creation of volumes (via multiple concurrent `docker volume create` commands) shows batching of reconfigure operations occurring.
hickeng
force-pushed
the
disk-batching
branch
2 times, most recently
from
15ba6ec
to
41de10a
Compare
Observed an outofspace question causing a stun of the VCH until answering the question via the UI. In this case the question was msg.hbacommon.outofspace That has NOT been fully resolved. The following steps have been taken: 1. switch to retry on transient errors in vm.reconfigure in disk manager 2. added Retry answer to the question so the operation responds in a predictable manner instead of stunning the VCH indefinitely. Additionally async error handlers have been added on some disk management paths to allow for increased parallelization of attach/detact.
hickeng
force-pushed
the
disk-batching
branch
from
41de10a
to
90ec04e
Compare
hickeng
added
component/persona/docker
component/portlayer/storage
area/storage
hickeng
force-pushed
the
disk-batching
branch
from
8e38c19
to
8169ff9
Compare
hickeng
changed the title
hickeng
force-pushed
the
disk-batching
branch
from
8169ff9
to
b5abe95
Compare
Prior test was unnecessarily complex and resulting in a non-useful message in error case The automation token can be added into drone using the drone cli: * drone secret add or * login to the drone webui, go to profile->secrets, and update via the box in the top right https://ci-vic.vmware.com/vmware/vic/settings/secrets at time of writing
hickeng
force-pushed
the
disk-batching
branch
from
b5abe95
to
ee23edb
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds disk batching to the endpointVM so operations such as volume
filesystem creation can occur in parallel.
It also adjusts the volume cache to remove a lock that seralizes all
creates. This is NOT complete as of this commit - it requires singleflight
or similar wrapping the volume create path, and also consideration as to
the other inspect/delete operations as they apply to a partially completed
volume.
This should also allow an operation to specify if it can tolerate some
latency. The desire behind this is so that disk detach operations for pull
can be deferred and batched with the attach operation for the subsequent
child disk which should halve the number of reconfigure operations
associated with a pull request.
This has additional modifications to the docker personality processing for
multiple anonymous volumes to parallelize the volume creation.
This means that most concurrent volume operations batch, reducing
container creation time by ~50% with 10 anonymous volumes in my tests.
Related: #5677
Fixes: #7929