shell: make kvs output limit configurable and default single-user jobs to unlimited #5732
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Problem: The job shell doesn't capture the owner of the enclosing instance, so there is no way to determine if the job is running in single vs multiuser mode. Capture the local broker security.owner attribute.
Problem: The shell fetches the local instance owner, but does not expose this value to plugins. Add an instance_owner key to the shell info object.
Problem: Presence of the instance_owner value in the shell info object is not documented. Add it to flux_shell_get_info(3).
garlick
approved these changes
Feb 12, 2024
grondo
force-pushed
the
shell-output-limit-relax
branch
from
34f709b
to
2299dfe
Compare
Yes, sorry that was missed. I've added a |
|
The 'system' instance container that runs systemd under docker has apparently stopped working. We may be hitting this issue: Though I'm not sure exactly when the underlying version of systemd would have been updated, since the system tests were working as of last Friday afaict. Anyway, the symptoms are the same as in that issue. As suggested, the tests do work under podman, so maybe that is one path forward if available on Github actions. I'll move this discussion to a separate issue, but until we figure this out we may have to disable the requirement for the |
Problem: The job shell truncates output at 10M to workaround current KVS limitations with very large eventlogs in possibly long-running system instances. However, this is unnecessarily limiting in single-user subinstances where it is most convenient to have all job output folded into the output of a batch job. Convert the KVS output limit to a configurable value set via the job shell option `output.limit`. Keep the default limit of 10M if the userid of the enclosing instance broker does not match the current userid, and disable the limit completely if the broker and shell userids match. Update shell output truncation tests that would now fail with the new behavior. Ensure output limit is not enforced in the single user test instance. Note that this allows a user to disable the limit in a multiuser instance as well via `-o output.limit=0`, but this was possible anyway since the limit is not really enforced by the instance (a user could build a custom job shell for instance).
Problem: The job shell output.limit option is not documented. Document this option in flux-shell(1) as well the common job shell options table for the cli submission commands.
grondo
force-pushed
the
shell-output-limit-relax
branch
2 times, most recently
from
00ab813
to
26194c3
Compare
Problem: There is no test that ensures the job shell truncates output to 10M by default for guest jobs in a multiuser instance. Add a test to t9000-system.t that ensures output is truncated in this case.
Problem: The use of `source` in t9000-system.t is non-POSIX and thus not supported by POSIX sh. Replace `source` with the POSIX `.`.
grondo
force-pushed
the
shell-output-limit-relax
branch
from
26194c3
to
74b4faa
Compare
|
Ok, I figured out why the system image tests were failing (my fault). I'll set MWP here now. |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## master #5732 +/- ##
==========================================
- Coverage 83.46% 83.45% -0.02%
==========================================
Files 487 487
Lines 82984 83028 +44
==========================================
+ Hits 69265 69288 +23
- Misses 13719 13740 +21
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds support for a new
output.limitjob shell option which overrides the default KVS output limit.The limit remains
10Mfor multiuser instance jobs (when broker userid != job shell userid), but the limit is removed for single-user jobs (broker userid == shell userid).This should resolve some of the issues users have hit with the KVS output limit inside batch jobs, where it is most convenient to have jobs output folded in with that of the batch output file.