Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support concurrent creation of volumes (vmdks) #7929

Open
5 tasks
hickeng opened this issue May 8, 2018 · 2 comments · May be fixed by #7895
Open
5 tasks

Support concurrent creation of volumes (vmdks) #7929

hickeng opened this issue May 8, 2018 · 2 comments · May be fixed by #7895
Labels
area/storage Storage-related functionality component/portlayer/storage kind/defect/performance Behavior that is functionally correct, but performs worse than intended kind/enhancement Behavior that was intended, but we want to make better severity/3-moderate Medium usability or functional impact. Potentially has an inconvenient workaround. source/customer Reported by a customer, directly or via an intermediary team/foundation

Comments

@hickeng
Copy link
Member

hickeng commented May 8, 2018

Story
As a user of VIC I expect to be able to concurrently create containers that use volumes

Details
There are two primary issues that are causing undue serialization of volume creation currently:
a. a lock in the volume cache
b. disk attach/detach reconfigure operations

(a) causes strict serialization and should simply be removed in favour of a singleflight call or similar.
(b) causes partial serialization, but the latency can be high enough that its effect is close to full serialization - this can be addressed with batching logic (see #7895)

The state seen by callers of the API should remain consistent, i.e. list should not show a volume that cannot be used or deleted, and only one create for a volume with a given name should succeed.

This impact concurrent container creation performance when anonymous volumes are part of the container specification.

Acceptance

  • add performance test to ensure no regression in volume create after this has merged
  • add CI test for concurrent create of at least 40 containers (SCSI bus limits us to 15 attached at once and there's a concurrent operation limit of 35 so this forces us to exercise queue in both locations).
  • concurrent creation of volume with the same name should have exactly one return success/fail and others should return with already-exists.
  • concurrent use or deletion of volume that's in the process of creation should return "does-not-exist"
  • listing of volumes while volume is being created should not including inflight volume creation.
@hickeng hickeng added kind/enhancement Behavior that was intended, but we want to make better kind/defect/performance Behavior that is functionally correct, but performs worse than intended component/portlayer/storage team/foundation labels May 8, 2018
@zjs zjs added the source/customer Reported by a customer, directly or via an intermediary label Jul 31, 2018
@zjs
Copy link
Member

zjs commented Jul 31, 2018

Outstanding question: what should docker volume ls show during creation?

@zjs zjs added area/storage Storage-related functionality severity/3-moderate Medium usability or functional impact. Potentially has an inconvenient workaround. and removed priority/p1 labels Jul 31, 2018
@hickeng hickeng linked a pull request Aug 8, 2018 that will close this issue
@hickeng
Copy link
Member Author

hickeng commented Aug 8, 2018

what should docker volume ls show during creation?

Yeah, that's the difficult case. The other operations are easy - they block when addressing that specific volume that's in creating.

Any option I considered has to potential to confuse clients as there's no "state" recorded for volumes.

My WIP thoughts are to include it in the list, but block on any use (including delete/inspect) if it's not fully created.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/storage Storage-related functionality component/portlayer/storage kind/defect/performance Behavior that is functionally correct, but performs worse than intended kind/enhancement Behavior that was intended, but we want to make better severity/3-moderate Medium usability or functional impact. Potentially has an inconvenient workaround. source/customer Reported by a customer, directly or via an intermediary team/foundation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants