Adding profiling to dask shuffle

[arunraman]

Adding benchmark profile for dask shuffle.

[GPUtester]

Can one of the admins verify this patch?

[GPUtester]

Can one of the admins verify this patch?

[ajschmidt8]

add to allowlist

[jakirkham]

The CI failures here should be resolved by PR ( #623 ) and PR ( rapidsai/cudf#8342 )

[arunraman]

The CI failures here should be resolved by PR ( #623 ) and PR ( rapidsai/cudf#8342 )

Thanks, do I need to re-trigger the test?

[jakirkham]

No we can do that. We just have to wait for those PRs to be merged

[madsbk]

Thanks @arunraman, good idea including profiling of the explicit-comms execution!

Hope you don't mind, I took the liberty to move the profiling to include both the regular and the explicit-comms execution.

[quasiben]

rerun tests

[jakirkham]

Still waiting on PR ( #623 ) before CI is fixed

[pentschev]

rerun tests

[jakirkham]

@gpucibot merge

[madsbk]

BTW @arunraman, Dask's performance profile might not include the communication within the explict-comms tasks. @jakirkham do you know what the performance profile capture exactly?

[quasiben]

Performance profile will capture:

• Task Stream
• Worker Profile Compute
• Worker Profile Admin
• Scheduler Profile Admin
• Communication Bandwidth by worker
• Communication Bandwidth by type

[madsbk]

@quasiben, what about manually created endpoints:

ep = await distributed.comm.connect(address)