log_vc_info: handle long command output #5821
Conversation
| def pipe_poller(proc, *files, chunk_size=4096): | ||
| """Read from a process without hitting buffer issues. | ||
|
|
||
| Standin for subprocess.Popen.communicate. | ||
|
|
||
| When PIPE'ing from subprocesses, the output goes into a buffer. If the | ||
| buffer gets full, the subprocess will hang trying to write to it. | ||
|
|
||
| This function polls the process, reading output from the buffers into | ||
| memory to prevent them from filling up. |
There was a problem hiding this comment.
In what cases might you call this function that isn't the case of pipe_poller(proc, proc.stdout, proc.stderr)? I.e. when would files be something other than the output of proc?
There was a problem hiding this comment.
You might only be piping one of stdout/err, so you might want:
pipe_poller(proc, proc.stdout)Or:
pipe_poller(proc, proc.stderr)You might also be piping one function into another, redirecting stdout/err into some other kind of buffer or fancy polling other other files/buffers associated with the process?
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
* Log the commands run by log_vc_info (debug level). * Handle the risk of large command output clogging up the buffer causing commands to hang. ..
Co-authored-by: Ronnie Dutta <61982285+MetRonnie@users.noreply.github.com>
oliver-sanders
force-pushed
the
pipe_poller
branch
from
492d3f0
to
e7dd6cf
Compare
|
I don't think using a Bash timeout will replicate the problem being fixed here? You need to get a process to write so much to stdout/err that it fills Python's buffers. |
Co-authored-by: Tim Pillinger <26465611+wxtim@users.noreply.github.com>
|
Have approved - the MacOs Tests look to be failing in a standard way. I've kicked flaky and the tutorial tests just to be sure. |
|
The tutorials tests are failing here as on upstream/8.2.x at the install step. |
…at_runtime * upstream/master: log_vc_info: handle long command output (cylc#5821) GH Actions: limit tutorial workflow to Py3.11 remove cylc task dependencies env var (cylc#5836) Refactor.lint (cylc#5718) protobuf 4.24.4 upgrade (cylc#5828) made reinstall work on multiple workflows Fix `IndepQueueManager` test (cylc#5832) Lint: Add a check for indentation being 4N spaces. (cylc#5772)
Check List
CONTRIBUTING.mdand added my name as a Code Contributor.setup.cfg(andconda-environment.ymlif present).CHANGES.mdentry included if this is a change that can affect users?.?.xbranch.