Capture LGBM model's .fit messages and redirect them to python logger

[MTDzi]

I'm using the logging Python module in my project and would like to be able to capture the messages produced by the lightgbm.sklearn.LGBMClassifier.fit method.

So far, I've re-directed the STDOUT to the logging file:

logging.basicConfig(filename=log_filename, level=logging.INFO)
channel = logging.StreamHandler(sys.stdout)
sys.stdout = open(log_filename, 'a', 1)
channel.setLevel(logging.INFO)
root = logging.getLogger()
root.setLevel(logging.INFO)
root.addHandler(channel)

but in doing so, I'm also re-directing output from the debugger -- not good.

How can I do it better?

[StrikerRUS]

At present practically all messages (except a few warnings) come directly from the C++ code to the stdout/stderr. So, there is no way to manage them by Python's setLevel() functionality. As a workaround you can write a function that will filter messages you don't want to write into a log and re-redirect them back to the console. The code of C++ logger may help you: https://github.com/Microsoft/LightGBM/blob/master/include/LightGBM/utils/log.h

[StrikerRUS]

@guolinke @chivee
Maybe we can have something similar?
dmlc/xgboost#3438

[MTDzi]

Hey,

Thanks for the reply!

As a work-around (because I was interested in the messages related to early stopping), I just took the LGBMClassifier.evals_result_ and passed that as a message to the logger.

But, for anyone else interested in the matter, I highly discourage from using my previous work-around, the one with the

...
sys.stdout = open(log_filename, 'a', 1)
...

hack.

[StrikerRUS]

@MTDzi Thank you for sharing your experience!

[StrikerRUS]

@guolinke @chivee
Is it possible to implement #1493 (comment) in LightGBM?
Such thing will allow, for example, to log right into Jupyter Notebook cell output.

[StrikerRUS]

ping @guolinke @chivee

[guolinke]

@StrikerRUS
Only need to redirect the output in training, without all other logs?

[StrikerRUS]

@guolinke I mean, create a callback at cpp side for all logs, which can be used at Python/R/whatever_language side:
#1493 (comment)

[StrikerRUS]

@guolinke Is it possible?

[guolinke]

@StrikerRUS I think it is easy to implement. But I don't have time recently...

[StrikerRUS]

@guolinke

I think it is easy to implement.

It's great!

But I don't have time recently...

… and it's a pity.

BTW, will this also help to fix #1440?

[guolinke]

@StrikerRUS It seems different. The problem in R is to redirect the log to R's console, and need to interact with R's library.
I think this issue is only redirecting to file.

[StrikerRUS]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.