Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: disable usage of root logger #7296

Merged
merged 7 commits into from
Apr 5, 2022
Merged

fix: disable usage of root logger #7296

merged 7 commits into from
Apr 5, 2022

Conversation

maxstrobel
Copy link
Contributor

@maxstrobel maxstrobel commented Apr 5, 2022

logging.basicConfig configures Python's root logger. This prohibits
fine control of logging, overwrites logging configuration done outside
the package, and is not best practice. Instead, the used logger is now
configured directly, and the root logger is untouched.

Example:
If yolov5 is used as part of another project with some sophisticated
logging, the internal logging.basicConfig call overwrites all the
external configuration.

Closes #6060

πŸ› οΈ PR Summary

Made with ❀️ by Ultralytics Actions

🌟 Summary

Improved logging setup in the YOLOv5 codebase.

πŸ“Š Key Changes

  • Standardized whitespace in Objects365.yaml.
  • Enhanced logging configuration in general.py by:
    • Setting the logging level based on the process rank and verbosity.
    • Adding a stream handler explicitly with a simple message format.

🎯 Purpose & Impact

  • 🧹 The removal of superfluous whitespace keeps configuration files clean and consistent.
  • πŸ“ˆ The updated logging mechanism allows for more controlled and formatted logging output.
    • For users and developers, this means clearer and more informative log messages while training models, running detections, or performing validations.
    • It may also facilitate easier debugging by ensuring that the right amount of information is being logged.

`logging.basicConfig` configures Python's root logger. This prohibits
fine control of logging, overwrites logging configuration done outside
the package, and is not best practice. Instead, the used logger is now
configured directly, and the root logger is untouched.

Example:
    If yolov5 is used as part of another project with some sophisticated
    logging, the internal `logging.basicConfig` call overwrites all the
    external configuration.
@glenn-jocher
Copy link
Member

@maxstrobel thanks for the PR! Is there a way to reduce the added lines while accomplishing the same result? The PR replaces 2 lines with 9 currently. We'd like to minimize/simplify where possible.

@maxstrobel
Copy link
Contributor Author

maxstrobel commented Apr 5, 2022

@glenn-jocher - Thanks for the quick response.

Maybe, we could save one or two lines, but I think it will always be a bit more.


Some explanations about logging:

Actually the function did two things, which I tried to replicate:

  1. Setup logging - set the log level & format for the root logger (& indirectly for all other loggers, because they are connected through hierarchy to the root logger).
  2. Return the logger that is used to record the actual log messages.

I think a better approach (but also slightly more changes) would be to split this up in two functions.

  1. Setup the logging - basically what I did in the function without the last line. This function is called at the top of your scripts, e.g. for training / inference.
  2. Provide the logger to your package as until now - this would be as simple as LOGGER = logging.getLogger("yolov5").

@glenn-jocher
Copy link
Member

@maxstrobel oh I see. So the current function would not return anything, and each independent file would get the logger on it's own with LOGGER = logging.getLogger("yolov5")?

Would all these instances be connected as they are now (the same LOGGER essentially?)

LOGGER is imported now in quite a few places with from utils.general import LOGGER, would we replace each one of those?

@glenn-jocher
Copy link
Member

glenn-jocher commented Apr 5, 2022

@glenn-jocher would this be a middle ground?

def set_logging(name=None, verbose=VERBOSE):
    # Sets level and returns logger
    if is_kaggle():
        for h in logging.root.handlers:
            logging.root.removeHandler(h)  # remove all handlers associated with the root logger object
    rank = int(os.getenv('RANK', -1))  # rank in world for Multi-GPU trainings
    level = logging.INFO if (verbose and rank in (-1, 0)) else logging.WARNING
    log = logging.getLogger(name)
    log.setLevel(level)
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(message)s"))
    handler.setLevel(level)
    log.addHandler(handler)


set_logging()  # EDIT: I think we need to run this before defining LOGGER
LOGGER = logging.getLogger("yolov5")  # define globally (used in train.py, val.py, detect.py, etc.)

@glenn-jocher
Copy link
Member

glenn-jocher commented Apr 5, 2022

@maxstrobel lastly, since you seem to know your way around logging, do you know what is happening in Kaggle? Without removing the root loggers in kaggle no output is observed in kaggle notebooks, so I was forced into this special workaround in set_logging(). The same notebook in Colab functions correctly:

@maxstrobel
Copy link
Contributor Author

maxstrobel commented Apr 5, 2022

@maxstrobel oh I see. So the current function would not return anything, and each independent file would get the logger on it's own with LOGGER = logging.getLogger("yolov5")?

Would all these instances be connected as they are now (the same LOGGER essentially?)

LOGGER is imported now in quite a few places with from utils.general import LOGGER, would we replace each one of those?

logging.getLogger(name) returns the logger with that specific name - all calls will return exactly the same instance, see https://docs.python.org/3/library/logging.html#logging.getLogger.

So, if you call LOGGER = logging.getLogger("yolov5") at the top of your scripts, or somewhere centrally & import the object makes no difference.
I prefer the first option, since it gives you the chance to setup different loggers on a file level & control them finer - but I think here both options are okay and we can stay with your proposal.


@glenn-jocher would this be a middle ground?

def set_logging(name=None, verbose=VERBOSE):
    # Sets level and returns logger
    if is_kaggle():
        for h in logging.root.handlers:
            logging.root.removeHandler(h)  # remove all handlers associated with the root logger object
    rank = int(os.getenv('RANK', -1))  # rank in world for Multi-GPU trainings
    level = logging.INFO if (verbose and rank in (-1, 0)) else logging.WARNING
    log = logging.getLogger(name)
    log.setLevel(level)
    handler = logging.StreamHandler()
    handler.setFormatter(logging.Formatter("%(message)s"))
    handler.setLevel(level)
    log.addHandler(handler)


set_logging()  # EDIT: I think we need to run this before defining LOGGER
LOGGER = logging.getLogger("yolov5")  # define globally (used in train.py, val.py, detect.py, etc.)
set_logging()  # EDIT: I think we need to run this before defining LOGGER

Maybe you want to have a look at the Logging Flow, https://docs.python.org/3/howto/logging.html#logging-flow.
image

Basically logging can be separated in those two blocks. Simply speaking, the logger flow collects & records the log messages from all your LOGGER.debug / LOGGER.info / ... calls, & the Handler flow prints the log messages to the command line or to a file.
What you want to configure or setup is probably rather the Handler flow, i.e. the log level & format of the messages that are printed to the command line.


@maxstrobel lastly, since you seem to know your way around logging, do you know what is happening in Kaggle? Without removing the root loggers in kaggle no output is observed in kaggle notebooks, so I was forced into this special workaround in set_logging(). The same notebook in Colab functions correctly:

Probably those changes would also resolve the Kaggle issue, since the specifically mention to not use logging.basicConfig:
https://www.kaggle.com/code/residentmario/notes-on-python-logging/notebook#Object-configuration

@glenn-jocher
Copy link
Member

@maxstrobel ok got it. I'll do some testing in Kaggle to see if PR resolves issue there. That would be a nice bonus.

@maxstrobel
Copy link
Contributor Author

@glenn-jocher, sounds like a plan. If you want to avoid too much changes, I'd go for your "middle ground" solution (+ if applicable remove the Kaggle stuff).

@glenn-jocher
Copy link
Member

@maxstrobel tested with a5f8969 and get the original problem of logging output not showing in Kaggle (only the torch download print() statements are showing). Strange. Anyway that's outside the scope of the original changes so I will merge PR as is now.

Screenshot 2022-04-05 at 17 27 40

@glenn-jocher glenn-jocher merged commit 741fac8 into ultralytics:master Apr 5, 2022
@maxstrobel maxstrobel deleted the bug/global-logging branch April 5, 2022 15:38
@glenn-jocher
Copy link
Member

@maxstrobel PR is merged. Thank you for your contributions to YOLOv5 πŸš€ and Vision AI ⭐

BjarneKuehl pushed a commit to fhkiel-mlaip/yolov5 that referenced this pull request Aug 26, 2022
* fix: disable usage of root logger

`logging.basicConfig` configures Python's root logger. This prohibits
fine control of logging, overwrites logging configuration done outside
the package, and is not best practice. Instead, the used logger is now
configured directly, and the root logger is untouched.

Example:
    If yolov5 is used as part of another project with some sophisticated
    logging, the internal `logging.basicConfig` call overwrites all the
    external configuration.

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Update general.py

* Update general.py

* Comment kaggle

* Uncomment kaggle

Co-authored-by: Maximilian Strobel <Maximilian.Strobel@infineon.com>
Co-authored-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Celery logger stops working using torch.hub.load
2 participants