[FEA] Improved RMM internal logging control

[jlowe]

Is your feature request related to a problem? Please describe.
Recently RMM has started emitting an rmm_log.txt file that logs internal RMM messages such as this (which occurred in an application that can recover from many RMM OOM situations and continue):

[ 32522][09:32:03:171617][info  ] ----- RMM LOG BEGIN [PTDS DISABLED] -----
[ 32522][09:32:03:171645][error ] [A][Stream 0x1][Upstream 3458764513820540928B][FAILURE max pool size exceeded]

There are cases where either the current working directory may not be writable or will not persist past the application lifetime, and thus the user would never see the logs. Therefore we need to be able to control where these logging messages are written, including the possibility of using existing file descriptors (e.g.; stdout or stderr).

Describe the solution you'd like
RMM could provide an API to set where internal log messages will be written. The initial implementation could just take a file descriptor to use. Being able to provide a spdlog sink or something similar would provide the most flexibility, as the application could implement some dynamic control behavior during logging if desired.

Describe alternatives you've considered
Compile-time flags would be less ideal since there is no flexibility. For example, if we need to compute the appropriate logging path at runtime, compile-time flags aren't going to work.

[jrhemstad]

Similar to logging_resource_adaptor, we can just add an optional spdlog sink to the pool_memory_resource ctor.

[harrism]

Similar to logging_resource_adaptor, we can just add an optional spdlog sink to the pool_memory_resource ctor.

The log is not specific to pool_memory_resource, so that won't be sufficient. It should probably be through the rmm::logger() interface.

[rongou]

@jlowe can this be considered fixed now?

[jlowe]

While we can workaround the undesired rmm_log.txt file by forcing RMM logging off during the build, I think it would be more ideal to not completely suppress these messages but instead direct them to a more appropriate location (e.g.: stdout, stderr, or some application-specific handling). Being able to provide a custom spdlog sink for RMM's internal logging should cover those cases and hopefully not be burdensome to implement.

[harrism]

Yeah planning to add config options in 0.17

[github-actions]

This issue has been marked rotten due to no recent activity in the past 90d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

[github-actions]

This issue has been marked stale due to no recent activity in the past 30d. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be marked rotten if there is no activity in the next 60d.

[jlowe]

This is still desired. It was originally targeted for 0.17, but I do not think it was implemented.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[jlowe]

Still desired.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[jlowe]

Still desired.

[harrism]

@jlowe spdlog provides a way to add and remove sinks from already running loggers. We could provide an RMM specific way of specifying a new sink, but it would be less general than the API that spdlog already provides. Currently you can do this (not thread safe):

auto sinks_vector = rmm::logger()->sinks();
sinks_vector.erase(sinks_vector.begin()); // remove the first sink (assuming it is the default
sinks_vector.push_back(std::make_shared<new_sink>(...));

You can also add new sinks without deleting the existing sink(s). See gabime/spdlog#820

Is this sufficient, or do you think we need to provide RMM-specific wrappers for this functionality?

[jlowe]

Thanks, @harrism! I didn't realize we could replace a sink on an existing log. I think that should be sufficient to do what we need.

[harrism]

Thanks, @harrism! I didn't realize we could replace a sink on an existing log. I think that should be sufficient to do what we need.

I didn't either, but when I started to look at fixing this, I discovered it.