A Token Filter used to concatenate one or more tokens into a single token within a token stream. You can specify a token separator, a token at which to begin concatenation, a token at which to end concatenation, and whether you want those marker tokens to be separated, combined, or dropped from the concatenated token. By default, it concatenates all tokens in the token stream with a space.
With default settings:
['the', 'quick', 'brown', fox'] => ['the quick brown fox']With startToken='<concat>', endToken='</concat>'
['the', '', 'quick', 'brown', '', fox'] => ['the', 'quick brown', 'fox']Configurable parameters:
- separator: the text to insert between each concatenated token. Defaults to space.
- startToken: if set, only tokens after the startToken and prior to the next endToken will be concatenated. If unset, concatenation starts at the beginning of the token stream.
- endToken: if set, stops concatenating tokens after the immediately preceding token.
- startTokenHandling: Supported options: 'separate', 'combine', 'drop' (the default). If set to separate, the start token will not be included in the subsequent concatenated token. If set to combine, the start token will be included in the subsequent concatenated token. If set to drop, the start token will be removed from the token stream.
- endTokenHandling: Supported options: 'separate', 'combine', 'drop' (the default). If set to separate, the end token will not be included in the preceding concatenated token. If set to combine, the end token will be included in the preceding concatenated token. If set to drop, the start token will be removed from the token stream.