Highlighting in documents doesn't handle CDATA sections correctly

[jan-niestadt]

"Tags" inside a CDATA are seen as actual (unbalanced) XML open tags, and closing tags are added at the end of the document.

Example:

https://portal.clarin.ivdnt.org/blacklab-server-new/opensonar/docs/WR-P-E-C-0000000129/contents?query=%5Bword%3D%22schip%22%5D&wordstart=7000

[jan-niestadt]

Attempted fix in 981b8a5, but this causes a StackOverflowError in the regex evaluation (even though small-scale test succeeds). Possibly the document is too large for handling this way. See https://stackoverflow.com/a/7510006

[jan-niestadt]

It might be time to consider rewriting highlighting of document fragments using something like https://jsoup.org/

If that doesn't seem practical for whatever reason, another alternative is to loop through the document character by character, only using regexes whenever we find a < (and possibly not using them at all for comments or CDATA, which can get large, unlike tags).

[jan-niestadt]

@KCMertens mentioned that Saxon's parsing can be customized as well, including how to deal with unbalanced tags; maybe this could be a good solution

[KCMertens]

Here is how I've done it in the past using commandline arguments: https://github.com/INL/vws-conversie/blob/master/saxon/run-xslt-tagsoup.sh#L28
That's different of course, but docs for doing it programatically are here: https://saxonica.com/html/documentation9.6/sourcedocs/controlling-parsing.html

Tagsoup specifically is written to be lenient.