-
-
Notifications
You must be signed in to change notification settings - Fork 718
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add nanny logs #2744
Add nanny logs #2744
Conversation
Other functions like |
So In fact, I suspect that on the scheduler you could just pass through the |
Thanks @mrocklin that works out quite nicely. I updated the docstring for |
As an aside, having timestamps in logs is quite helpful for collation across the cluster (assuming the clocks synced well enough). What's our backwards compatibility story for configuration things like that? |
Actually, this may not be ready. I thought I had sorted out the duplicate handler, but apparently not. |
Other than the possible duplicate handler issue the implementation here looks great to me. |
I don't have any particular thoughts here. You may want to take a look here though: |
Can you explain this a bit more? This seems to be the same behavior as before. Is this correct? |
I'm not too sure what's going on. In [1]: from distributed import Client
...: import logging
...:
...:
...: client = Client(n_workers=2, threads_per_worker=1)
...:
...: logger = logging.getLogger('distributed.nanny')
...: logger.handlers
Out[1]:
[<StreamHandler <stderr> (WARNING)>,
<DequeHandler (NOTSET)>,
<DequeHandler (NOTSET)>] So if I get the Oh... but I just realized this is the same behavior as ...: client = Client(n_workers=2, threads_per_worker=1, processes=False)
...:
...: logger = logging.getLogger('distributed.worker')
...: logger.handlers
Out[2]: [<DequeHandler (NOTSET)>, <DequeHandler (NOTSET)>] so perhaps we're OK? |
Looks good to me. Merging. |
* upstream/master: (58 commits) Add unknown pytest markers (dask#2764) Delay lookup of allowed failures. (dask#2761) Change address -> worker in ColumnDataSource for nbytes plot (dask#2755) Remove module state in Prometheus Handlers (dask#2760) Add stress test for UCX (dask#2759) Add nanny logs (dask#2744) Move some of the adaptive logic into the scheduler (dask#2735) Add SpecCluster.new_worker_spec method (dask#2751) Worker dashboard fixes (dask#2747) Add async context managers to scheduler/worker classes (dask#2745) Fix the resource key representation before sending graphs (dask#2716) (dask#2733) Allow user to configure whether workers are daemon. (dask#2739) Pin pytest >=4 with pip in appveyor and python 3.5 (dask#2737) Add Experimental UCX Comm (dask#2591) Close nannies gracefully (dask#2731) add kwargs to progressbars (dask#2638) Add back LocalCluster.__repr__. (dask#2732) Move bokeh module to dashboard (dask#2724) Close clusters at exit (dask#2730) Add SchedulerPlugin TaskState example (dask#2622) ...
* upstream/master: (43 commits) Add unknown pytest markers (dask#2764) Delay lookup of allowed failures. (dask#2761) Change address -> worker in ColumnDataSource for nbytes plot (dask#2755) Remove module state in Prometheus Handlers (dask#2760) Add stress test for UCX (dask#2759) Add nanny logs (dask#2744) Move some of the adaptive logic into the scheduler (dask#2735) Add SpecCluster.new_worker_spec method (dask#2751) Worker dashboard fixes (dask#2747) Add async context managers to scheduler/worker classes (dask#2745) Fix the resource key representation before sending graphs (dask#2716) (dask#2733) Allow user to configure whether workers are daemon. (dask#2739) Pin pytest >=4 with pip in appveyor and python 3.5 (dask#2737) Add Experimental UCX Comm (dask#2591) Close nannies gracefully (dask#2731) add kwargs to progressbars (dask#2638) Add back LocalCluster.__repr__. (dask#2732) Move bokeh module to dashboard (dask#2724) Close clusters at exit (dask#2730) Add SchedulerPlugin TaskState example (dask#2622) ...
I had a need to collate logs from the cluster, and would like to include logs from the Nanny. This adds a method to get logs from the nanny as well.
This isn't quite ready.
distribted.nanny.logger
multiple times, resulting in duplicate logs. Need to verify what's going on.