You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Introduce TaskExecutionLogStore component:
- Associates a log file with a Task ID
- Each file is a stream of structured log entry records (perhaps, in a format like NDJSON) - each record contains information about the components issuing it, traces/spans …
- Import and save log files as task progresses / completes
Initial simplest implementation:
- storage an in-memory hash table with task ID as a key
- store log files collection as an in-memory file in a format convenient for quick skipping (like NDJSON)
lost after server restart
- Have a time-based automatic purging of old logs - i.e. N=30 days max
- Future: Store logs as S3
- Future: Maybe something like ELK stack, if further analytics will be necessary
GraphQL API:
- Allow downloading / viewing full log files / viewing parts (paging by log entries starting from the latest)
- Event time or log entry ID arebetter boundary than index for this type of pagination
- Filter log entries by log level
- A way to scroll quickly to given or relative time (1 hour ago, 3 hours ago, ..)
The text was updated successfully, but these errors were encountered:
Introduce TaskExecutionLogStore component:
- Associates a log file with a Task ID
- Each file is a stream of structured log entry records (perhaps, in a format like NDJSON) - each record contains information about the components issuing it, traces/spans …
- Import and save log files as task progresses / completes
Initial simplest implementation:
- storage an in-memory hash table with task ID as a key
- store log files collection as an in-memory file in a format convenient for quick skipping (like NDJSON)
lost after server restart
- Have a time-based automatic purging of old logs - i.e. N=30 days max
- Future: Store logs as S3
- Future: Maybe something like ELK stack, if further analytics will be necessary
GraphQL API:
- Allow downloading / viewing full log files / viewing parts (paging by log entries starting from the latest)
- Event time or log entry ID arebetter boundary than index for this type of pagination
- Filter log entries by log level
- A way to scroll quickly to given or relative time (1 hour ago, 3 hours ago, ..)
The text was updated successfully, but these errors were encountered: