File with messages which swap a variable for a variable placeholder aren't compressed correctly

[kirkrodrigues]

Bug

If a file contains two messages which are identical except one message contains a variable and the other contains its corresponding variable placeholder, the file will not be compressed correctly such that decompressing it either stops prematurely or crashes.

For example, these log messages (where ^Q is an integer-variable placeholder)...

2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties ^Q from
2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties 123 from
2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties 123 from

... will cause the following exception:

2023-09-17 21:36:24,750 [error] FileWriter not closed before being destroyed - may cause data loss                                                                                                               2023-09-17 21:36:24,750 [error] Decompression failed: src/DictionaryReader.hpp:208 DictionaryReader operation failed, error_code=3

If instead we reorder the messages...

2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties 123 from
2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties ^Q from
2015-03-23 07:29:48,942 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties 123 from

... decompression will stop after the 2nd message, where the 2nd message is also incorrect.

CLP version

5d0f7b3

Environment

Ubuntu 20.04

Reproduction steps

• Compress the file described in the bug report: ./clp c archives log.txt
• Decompress the file described in the bug report: ./clp x archives decomp
• Observe a crash or incorrect decompressed output: diff log.txt decomp/log.txt