Cannot maintain effective debug-level on distributed (parent) logger with local scheduler/cluster

[javabrett]

With config.yaml including simple-form logging config:

logging:
  distributed: debug

>>> import logging
>>> import distributed
>>> logging.getLogger("distributed")
<Logger distributed (DEBUG)>
>>> logging.getLogger("distributed").handlers
[<StreamHandler <stderr> (NOTSET)>]
>>> logging.getLogger("distributed").debug("debug")
distributed - DEBUG - debug
>>> 

All looks good. Start a Client:

>>> from distributed import Client
>>> c = Client()
>>> logging.getLogger("distributed")
<Logger distributed (DEBUG)>
>>> logging.getLogger("distributed").handlers
[<StreamHandler <stderr> (WARNING)>]
>>> logging.getLogger("distributed").debug("debug")
>>> 

No logging output, as the distributed logger has had its Handler (a StreamHandler) set to WARNING threshold.

Once a Client is started and fires-up a local cluster/scheduler, I suspect a worker:

distributed/distributed/worker.py

Lines 424 to 425 in 09b959a

	if silence_logs:
	silence_logging(level=silence_logs)

... runs silence_logging, which does just as it is named:

distributed/distributed/utils.py

Lines 728 to 742 in 09b959a

	def silence_logging(level, root="distributed"):
	"""
	Change all StreamHandlers for the given logger to the given level
	"""
	if isinstance(level, str):
	level = getattr(logging, level.upper())
	old = None
	logger = logging.getLogger(root)
	for handler in logger.handlers:
	if isinstance(handler, logging.StreamHandler):
	old = handler.level
	handler.setLevel(level)
	return old

See also #2659 . It might be argued that nothing important is logging to the distributed logger, and everything is logging to a child e.g. distributed.client (even through the loggers are flattened with the simple config), which might be true, but it further erodes the notion of hierarchical logging.

[mrocklin]

@jcrist might have thoughts here

[GenevieveBuckley]

Cross referencing this broader discussion on logging: #2659