Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store and stream task execution logs #286

Open
Tracked by #58
zaychenko-sergei opened this issue Nov 14, 2023 · 0 comments
Open
Tracked by #58

Store and stream task execution logs #286

zaychenko-sergei opened this issue Nov 14, 2023 · 0 comments
Labels
enhancement New feature or request

Comments

@zaychenko-sergei
Copy link
Contributor

zaychenko-sergei commented Nov 14, 2023

Introduce TaskExecutionLogStore component:
- Associates a log file with a Task ID
- Each file is a stream of structured log entry records (perhaps, in a format like NDJSON) - each record contains information about the components issuing it, traces/spans …
- Import and save log files as task progresses / completes

Initial simplest implementation:
- storage an in-memory hash table with task ID as a key
- store log files collection as an in-memory file in a format convenient for quick skipping (like NDJSON)
lost after server restart
- Have a time-based automatic purging of old logs - i.e. N=30 days max
- Future: Store logs as S3
- Future: Maybe something like ELK stack, if further analytics will be necessary

GraphQL API:
- Allow downloading / viewing full log files / viewing parts (paging by log entries starting from the latest)
- Event time or log entry ID arebetter boundary than index for this type of pagination
- Filter log entries by log level
- A way to scroll quickly to given or relative time (1 hour ago, 3 hours ago, ..)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant