- Main thread
- Filter blocks (by timestamp and level filters)
- Sort blocks in chronological order by first record timestamp
- Pusher thread
- Read and push blocks one by one
- Worker threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records
- If there are no records, go to (1)
- Format records into a preallocated buffer
- Collect formatted records ranges
- Push the formatted block
- Merger thread
- Pull next formatted block
- Put the block into workspace
- Sort all blocks in a workspace by next record timestamp
- Copy the first record of the first block
- If its timestamp is >= than timestamp of recently pulled formatted block, go to (1), else go to (3)
- Reader thread
- Read input splitting into blocks
- Archive each block and store into memory
- Indexer threads
- Index each block
- Pusher thread
- Filter blocks (by timestamp and level filters)
- Sort blocks in chronological order by first record timestamp
- Read and push blocks one by one
- Worker threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records
- If there are no records, go to (1)
- Format records into a preallocated buffer
- Collect formatted records ranges
- Push the formatted block
- Merger thread
- Pull next formatted block
- Put the block into workspace
- Sort all blocks in a workspace by next record timestamp
- Copy the first record of the first block
- If its timestamp is >= than timestamp of recently pulled formatted block, go to (1), else go to (3)
- Reader thread
- Read input splitting into blocks
- Parser threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records (if no filter, (3)..(6) may be skipped)
- If there are no records, go to (1)
- Copy records to a new block
- Collect ranges of the records
- Archive block and store it into memory
- Push the metadata of the block down to the pipeline
- Go to (1)
- Pusher thread
- Collect all incoming blocks
- Filter blocks (by timestamp and level filters) (? - duplicates 2.3)
- Sort blocks in chronological order by first message timestamp
- Read and push blocks one by one
- Worker threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records (? - duplicates 2.3)
- If there are no records, go to (1)
- Format records into a preallocated buffer
- Collect formatted records ranges
- Push the formatted block
- Merger thread
- Pull next formatted block
- Put the block into workspace
- Sort all blocks in a workspace by next record timestamp
- Copy the first record of the first block
- If its timestamp is >= than timestamp of recently pulled formatted block, go to (1), else go to (3)
- Main thread
- Filter blocks (by timestamp and level filters)
- Sort blocks in chronological order by first record timestamp
- Evaluate block lifetimes along the chronological stream
- Reader thread
- Iterate over the filtered and sorted blocks
- If next block does not go in order
- For each block before it
- Drop it if filtered out by (1.1)
- Push it to the archiver threads otherwise
- For each block before it
- Read the block
- Push it to the worker threads
- If next block does not go in order
- Iterate over the filtered and sorted blocks
- Archiver threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records (if no filter, (3)..(4) may be skipped)
- Copy records to a new block
- If there are no records left, go to (1)
- Archive the block and store into memory
- Go to (1)
- Worker threads
- Take next incoming block and split it into lines
- Parse lines as records
- Filter records (? - duplicates 3.3)
- If there are no records, go to (1)
- Format records into a preallocated buffer
- Collect formatted records ranges
- Push the formatted block
- Merger thread
- Pull next formatted block
- Put the block into workspace
- Sort all blocks in a workspace by next record timestamp
- Copy the first record of the first block
- If its timestamp is >= than timestamp of recently pulled formatted block, go to (1), else go to (3)