Relations between words

[jan-niestadt]

Allows searching for (dependency) relations between words.

More details in plan-relations.md.

Done (relations):

• Support for indexing and searching relations (such as dependency relations) was added. Relations have a class (e.g. "dep" for dependency relations, also the default class), a type (e.g. "det" for a determiner relation), a source and target (both spans of adjacent tokens), and may have arbitrary attributes (just as inline tags can have attributes).
• Both relations and inline tags are indexed in the _relation annotation (which replaces the ill-named starttag annotation)
• Inline tags are now indexed as a special relation type pointing from the start to the end tag position, both 0-length).
• The API for indexing inline tags from a DocIndexer was abstracted, so no implementation details need to live in the DocIndexers.
• Both low-level relations functions and a high-level dependency relations operator were added to our CQL implementation. See CQL extension for dependency relations
• A DocIndexer for the CoNLL-U format was added, which can include dependency relations
• Capture groups were generalized to "match info", also capturing any inline tags or relations that were matched, including their type and attributes. BLS will return matched capture groups, relations and inline tags separately in the response. Use new matchinfo parameter with value captures to get the old behaviour (for compatibility).
• Match info is now taken into account for hit uniqueness. This means that several hits with the same start and end position may be returned, if they have different match info (e.g. several different ways groups can be captured in this same hit span)
• When matching relations, there is always one "active" relation, which is the one used in subsequent relation matching. You can adjust the current span to the source, target, or full span of the active relation. You can also adjust it to cover all matched relations if needed.
• The rmatch() operation is used in place of & (AND) when matching relations clauses. rmatch() ensures that all matched relations are unique, so we can find e.g. two different "det" relations with the same source, and not get a result with the same "det" relation twice. It also ensures that sets of matched relations are unique, so we don't get separate hits matching A;B and B;A. All this ensures that things work as expected for users.
• Constraints (after :: operator) may now use e.g. start(A) > start(B) to check the ordering of captures. end(A) also works. Previously :: constraints were only allowed at the top-level of the query, but they may now also occur in parenthesized subqueries, as long as they don't refer to groups captured outside that subquery.
• Relation queries are optimized where possible. When searching for relations where the source or target is only specified as []{m,n}, e.g. [] for one word, are resolved using a new length filter instead of finding all n-grams (this is used for any applicable AND(NOT) query). If you capture the source/target specified as an n-gram, this should be optimized as well by lifting the capture operation so the AND can be optimized as described.
• group by capture was improved as relations queries may capture spans outside of the requested hit+context. This is now no longer a problem.
• You can change the name of captured relations and inline tags to avoid collisions if you're capturing the same relation or tag twice in a query. Use e.g. _ A:-det-> _ to capture all det relations as A and A:<s/> to capture all sentences as A. The latter is a slightly incompatible change as this would formerly have been captured as a regular span, but is now captures as the inline tag (with tag name and attributes, in inlineTags section in BLS). If you really need the old behaviour, you can use A:_ident(<s/>). _ident() is a debug function that will not affect the meaning of the query, just prevent certain query rewrites from being applied. The third parameter of rel() can now be used to name the capture group as well. If the empty string is used, a name is automatically assigned (default behaviour). The 4th param is the direction filter that used to be the 3rd param.
• capture all relations for matched sentence with e.g. rcapture(sen:<s/>, 'sen', 'sen-rels')
• _ is now the "don't care" / "default" value for the relation operator(s) as well as for function calls. So _ --> _ now means "all relations. That is, _ is interpreted as []* here. /snippet operation does this automatically if you specify context=s.
• if a query matches several relations with same regex (e.g. multiple -->), each will get a unique capture name if not assigned explicitly in the query.
• support Parlamint TEI with dependency relations
• If relations query doesn't have rspan() call as root, automatically add rspan(..., 'all')

Done (other):

• context=s will return a whole sentence as context; context=5:10 will return 5 words before and 10 after the hit (context replaces wordsaroundhit, which still works for now). Also works for /snippet.
• /snippet response is now same as a hit in hits results
• Tests were switched to API v4
• Weight.isCachable() was implemented for various Weight classes to allow a bit more caching.
• inconsistent Unicode character normalization (canonical vs decomposed) will be corrected during indexing. Also soft hyphen and zero-width space are automatically removed. This prevents problems with sorting/grouping later.
• proxy supports POST requests as well, useful for very long queries
• HitProperties that use context from the forward index now manage their own context instead of the complicated system with needsContext, context indices, etc. They only fetch the context tokens they actually need, which could be slightly faster. The context is be disposed when it's no longer needed to save memory.
• API v4 is transitional (98% compatible with v3, adds stuff like new /corpora/NAME endpoints), experimental API v5 is new API (removes deprecated v3 stuff, changes some keys, some structures, XML more in line with JSON). Use api=3|4|exp to select API to use. Don't rely on any of the new API stuff until BlackLab 4.0 is released.
• CI saves responses for external and integrated separately (mainly because matchInfo differs; there may be more differences in the future)
• CorpusQL query can be translated to/from a JSON textpattern. You may pass such a JSON structure as the patt parameter, and the JSON structure for CQL query will be returned in the response as well. There's also a new endpoint /parse-pattern that can parse a CQL query into a JSON response and vice versa. Useful for implementing a query builder.
• Unify VTD and Saxon indexers, give Saxon feature parity
• Refactored DocumentFormats to simplify the code and make it easier to track errors.

TODO (see issue #449):

• Add proper relations tests (include some lassy data for tests? or something synthetic?)
• write proper documentation for relations (once features and syntax finalized)

MAYBE

• Include rcapture alternative? E.g. "de" within (<s/> collect -->)
• OPT: maybe drop the distinct relations requirement for rmatch if relationtypes in query are already distinct?
• Relation descendants: what operations would be needed? would indexing extra "multi-relations" help? (you could index all paths from the root to a leaf, with the relation types included in the indexed term, and the positions included in the payload) or do we need to reconstruct the full tree structure and perform the entire operation on that?
• it might be nice if you could do something like rspan but for any match info, i.e. "give me the span that matches group A" or "give me the span for the sentence tag".
• MAYBE: syntax to capture different spanMode without changing span? So instead of e.g. rspan(A:rel('flat', _, 'target'), 'source') you could write rcap(rel('flat'), 'A', 'target') ("capture target of rel('flat') as A, but resulting span is still the same as rel('flat')"), so rcap(query, name, spanMode='target')

Closes #405 and #201.