Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add UCX Comm #2591

Merged
merged 85 commits into from
May 31, 2019
Merged

Add UCX Comm #2591

merged 85 commits into from
May 31, 2019

Commits on Feb 19, 2019

  1. ENH: UCX-based Comms 

    * Stubs for classes
    Tom Augspurger committed Feb 19, 2019
    Configuration menu
    Copy the full SHA
    a9c17f8 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2019

  1. CUDA failing

    Tom Augspurger committed Feb 22, 2019
    Configuration menu
    Copy the full SHA
    65bd972 View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2019

  1. fixups

    Tom Augspurger committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    4fc6acd View commit details
    Browse the repository at this point in the history

  2. wip

    Tom Augspurger committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    d64c5cc View commit details
    Browse the repository at this point in the history

  3. zero copy

    Tom Augspurger committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    b28668b View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2019

  1. wip

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    4fcafae View commit details
    Browse the repository at this point in the history

  2. BUG: Ensure proper cleanup in comm_pair tests

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    f33ba29 View commit details
    Browse the repository at this point in the history

  3. Reset reader_added before listening

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    b61e56d View commit details
    Browse the repository at this point in the history

  4. CUDA failing

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    89df2bd View commit details
    Browse the repository at this point in the history

  5. fixups

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    5b51716 View commit details
    Browse the repository at this point in the history

  6. wip

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    a71c896 View commit details
    Browse the repository at this point in the history

  7. zero copy

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    fb862bf View commit details
    Browse the repository at this point in the history

  8. Merge branch 'ucx+data-handling' of https://github.com/TomAugspurger/… 

    …distributed into ucx+data-handling
    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    6a194a3 View commit details
    Browse the repository at this point in the history

  9. fix registration

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    594028c View commit details
    Browse the repository at this point in the history

  10. all passing

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    120bc2f View commit details
    Browse the repository at this point in the history

  11. cleanup

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    2def137 View commit details
    Browse the repository at this point in the history

  12. move override to the test

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    fcb800a View commit details
    Browse the repository at this point in the history

  13. rename

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    8745a19 View commit details
    Browse the repository at this point in the history

  14. remove old tests

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    1339a3e View commit details
    Browse the repository at this point in the history

  15. todos

    Tom Augspurger committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    5d0d993 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2019

  1. Send headers

    Tom Augspurger committed Feb 27, 2019
    Configuration menu
    Copy the full SHA
    8bc2dbb View commit details
    Browse the repository at this point in the history

  2. use nbytes

    Tom Augspurger committed Feb 27, 2019
    Configuration menu
    Copy the full SHA
    3e998ce View commit details
    Browse the repository at this point in the history

Commits on Apr 2, 2019

  1. let internal protocol machiner set lengths

    @quasiben
    quasiben committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    d4b3501 View commit details
    Browse the repository at this point in the history

  2. clean up

    @quasiben
    quasiben committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    b89717e View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' into tom-ucx

    @mrocklin
    mrocklin authored Apr 2, 2019
    Configuration menu
    Copy the full SHA
    b274699 View commit details
    Browse the repository at this point in the history

  4. numba and ucx updates from @tom

    @quasiben
    quasiben committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    3906d56 View commit details
    Browse the repository at this point in the history

  5. Merge branch 'tom-ucx' of github.com:quasiben/distributed into tom-ucx

    @quasiben
    quasiben committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    66ac283 View commit details
    Browse the repository at this point in the history

  6. remove length settings in cupy/numba protocols

    @quasiben
    quasiben committed Apr 2, 2019
    Configuration menu
    Copy the full SHA
    4eb5860 View commit details
    Browse the repository at this point in the history

Commits on Apr 11, 2019

  1. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @quasiben
    quasiben committed Apr 11, 2019
    Configuration menu
    Copy the full SHA
    54e99a5 View commit details
    Browse the repository at this point in the history

Commits on Apr 16, 2019

  1. partially working localcluster -- need to cleanup delete

    @quasiben
    quasiben committed Apr 16, 2019
    Configuration menu
    Copy the full SHA
    6dcd192 View commit details
    Browse the repository at this point in the history

  2. merge with master

    @quasiben
    quasiben committed Apr 16, 2019
    Configuration menu
    Copy the full SHA
    7e9fd35 View commit details
    Browse the repository at this point in the history

  3. merge with master

    @quasiben
    quasiben committed Apr 16, 2019
    Configuration menu
    Copy the full SHA
    8a7f9e6 View commit details
    Browse the repository at this point in the history

Commits on Apr 17, 2019

  1. yield comm closing and parameterize ucx test

    @quasiben
    quasiben committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    d332b07 View commit details
    Browse the repository at this point in the history

  2. Use random port when given 0 

    This should be fixed upstream and should retry if the port is taken

Also handle AttributeError on Python 3.6 for asyncio
    @mrocklin
    mrocklin committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    6fdcd2b View commit details
    Browse the repository at this point in the history

  3. Allow for async def comm.close functions 

    Previously we called `comm.close()`, a coroutine, assuming that this would
start running the function.  This is true in tornado, but not in asyncio. Now
we explicitly yield or add them to the IOLoop
    @mrocklin
    mrocklin committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    c0ffeff View commit details
    Browse the repository at this point in the history

  4. Coerce the small bytes header of a message to bytes 

    UCX was producing memoryviews here, and they seemed to be malformed.
Coercing to bytes seems foolproof and cheap (these should always be small)
    @mrocklin
    mrocklin committed Apr 17, 2019
    Configuration menu
    Copy the full SHA
    44c1d5c View commit details
    Browse the repository at this point in the history

Commits on Apr 25, 2019

  1. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    edc094c View commit details
    Browse the repository at this point in the history

  2. Correcting ucp import statement to replect ucx-py PR#85 -- Also addin… 

    …g simple error message for failed ucx import
    @mrocklin
    Richard Zamora authored and mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    69250f7 View commit details
    Browse the repository at this point in the history

  3. Removing print statement

    @rjzamora @mrocklin
    rjzamora authored and mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    bf6b6b9 View commit details
    Browse the repository at this point in the history

  4. remove print statements

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    2c9cc2b View commit details
    Browse the repository at this point in the history

  5. skip ucx tests if not available

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    c5d669d View commit details
    Browse the repository at this point in the history

  6. cleanup ucx comm tests

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    f2925bf View commit details
    Browse the repository at this point in the history

  7. black

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    c78eb51 View commit details
    Browse the repository at this point in the history

  8. downgrade logging to debug level

    @mrocklin
    mrocklin committed Apr 25, 2019
    Configuration menu
    Copy the full SHA
    477a913 View commit details
    Browse the repository at this point in the history

Commits on Apr 26, 2019

  1. raised CommClosedError on closed UCX comm

    @mrocklin
    mrocklin committed Apr 26, 2019
    Configuration menu
    Copy the full SHA
    52f2f8c View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2019

  1. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    329b590 View commit details
    Browse the repository at this point in the history

  2. Clean up UCX tests

    @mrocklin
    mrocklin committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    4a30f17 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    ac1c8d0 View commit details
    Browse the repository at this point in the history

  4. Add larger cupy array into test 

    Currently this fails.  An array slightly smaller than this is fine
    @mrocklin
    mrocklin committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    25b1327 View commit details
    Browse the repository at this point in the history

  5. Send all frame metadata as a single message 

    This should reduce back and forth a bit
    @mrocklin
    mrocklin committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    ff5ca77 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2019

  1. add environment variables at ucx import time

    @mrocklin
    mrocklin committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    f0c51a5 View commit details
    Browse the repository at this point in the history

  2. black

    @mrocklin
    mrocklin committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    ce46bc3 View commit details
    Browse the repository at this point in the history

  3. add comments about byte offsets

    @mrocklin
    mrocklin committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    ecdf3cb View commit details
    Browse the repository at this point in the history

Commits on May 2, 2019

  1. Add comm.close call to IOLoop 

    This is important if comm.close is defined as an async function rather than a
tornado coroutine
    @mrocklin
    mrocklin committed May 2, 2019
    Configuration menu
    Copy the full SHA
    f557766 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2019

  1. Merge branch 'master' into tom-ucx

    @mrocklin
    mrocklin committed May 6, 2019
    Configuration menu
    Copy the full SHA
    152be22 View commit details
    Browse the repository at this point in the history

Commits on May 9, 2019

  1. Merge branch 'master' into tom-ucx

    @mrocklin
    mrocklin committed May 9, 2019
    Configuration menu
    Copy the full SHA
    e809d06 View commit details
    Browse the repository at this point in the history

  2. set UCX_TLS default environment variable

    @mrocklin
    mrocklin committed May 9, 2019
    Configuration menu
    Copy the full SHA
    ec3f77f View commit details
    Browse the repository at this point in the history

Commits on May 17, 2019

  1. await endpoint

    @mrocklin
    mrocklin committed May 17, 2019
    Configuration menu
    Copy the full SHA
    b21a954 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2019

  1. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed May 20, 2019
    Configuration menu
    Copy the full SHA
    57d4f69 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2019

  1. Merge branch 'master' into tom-ucx

    @mrocklin
    mrocklin committed May 22, 2019
    Configuration menu
    Copy the full SHA
    183a0ce View commit details
    Browse the repository at this point in the history

Commits on May 23, 2019

  1. Add --protocol flag to CLI

    @mrocklin
    mrocklin committed May 23, 2019
    Configuration menu
    Copy the full SHA
    b319a37 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'cli-protocol' into tom-ucx

    @mrocklin
    mrocklin committed May 23, 2019
    Configuration menu
    Copy the full SHA
    939769b View commit details
    Browse the repository at this point in the history

Commits on May 27, 2019

  1. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    f7afac7 View commit details
    Browse the repository at this point in the history

  2. Create separate cuda serialization family 

    Now we can still move GPU objects on non-UCX comms
    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    9ba163f View commit details
    Browse the repository at this point in the history

  3. Merge branch 'master' of github.com:dask/distributed into tom-ucx

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    b969368 View commit details
    Browse the repository at this point in the history

  4. workers -> workers.values()

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    2b2b459 View commit details
    Browse the repository at this point in the history

  5. lint

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    7532265 View commit details
    Browse the repository at this point in the history

  6. allow cupy arrays as frames 

    (perhaps not the best idea, but it allows a sensible test until we have a
device-memoryview)
    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    8babb8e View commit details
    Browse the repository at this point in the history

  7. numba fixed an issue upstream. remove a hack

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    6696466 View commit details
    Browse the repository at this point in the history

  8. Don't special-case MemoryPointer objects 

    Lets just not split them into pieces in the first place hopefully?
    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    23d0b0d View commit details
    Browse the repository at this point in the history

  9. Handle compression in dask_dumps, not sub serializers

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    4c94e8d View commit details
    Browse the repository at this point in the history

  10. add failing test for large messages in UCX

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    5e49409 View commit details
    Browse the repository at this point in the history

  11. cleanup close comments

    @mrocklin
    mrocklin committed May 27, 2019
    Configuration menu
    Copy the full SHA
    1457011 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2019

  1. Cleanup UCX comm

    @mrocklin
    mrocklin committed May 28, 2019
    Configuration menu
    Copy the full SHA
    d89e654 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' into tom-ucx

    @mrocklin
    mrocklin authored May 28, 2019
    Configuration menu
    Copy the full SHA
    7abf6b7 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'tom-ucx' of github.com:quasiben/distributed into tom-ucx

    @mrocklin
    mrocklin committed May 28, 2019
    Configuration menu
    Copy the full SHA
    b3d1a39 View commit details
    Browse the repository at this point in the history

Commits on May 29, 2019

  1. add logging for the preload module

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    23e4b4c View commit details
    Browse the repository at this point in the history

  2. Verify that we use cuda to serialize

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    a9478e4 View commit details
    Browse the repository at this point in the history

  3. black

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    0a71051 View commit details
    Browse the repository at this point in the history

  4. Fix test_comm test

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    51d4c03 View commit details
    Browse the repository at this point in the history

  5. flake8

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    3a28ea0 View commit details
    Browse the repository at this point in the history

  6. py35 compat

    @mrocklin
    mrocklin committed May 29, 2019
    Configuration menu
    Copy the full SHA
    3a0e25d View commit details
    Browse the repository at this point in the history

Commits on May 30, 2019

  1. _ucp_init -> ucp.init

    @mrocklin
    mrocklin committed May 30, 2019
    Configuration menu
    Copy the full SHA
    90f8eeb View commit details
    Browse the repository at this point in the history

  2. remove _frames suffix

    @mrocklin
    mrocklin committed May 30, 2019
    Configuration menu
    Copy the full SHA
    850b9ad View commit details
    Browse the repository at this point in the history

  3. Remove unnecessary comments

    @mrocklin
    mrocklin committed May 30, 2019
    Configuration menu
    Copy the full SHA
    88afada View commit details
    Browse the repository at this point in the history