Illegal Unicode Escaping

[brcolow]

This is kind of a "gotcha" type issue but I think my reasoning is sound.
nanojson parses Java character literals correctly (and goes to great
lengths to make sure the UTF-8 is valid) in the consumeTokenStringUtf8Char
method. But when the characters are escaped (not Java character literals)
it uses the following simplified logic:

int escaped = 0;

for (int i = 0; i < 4; i++) {
    escaped <<= 4;
    int digit = buffer[index++];
    if (digit >= '0' && digit <= '9') {
        escaped |= (digit - '0');
    } else if (digit >= 'A' && digit <= 'F') {
        escaped |= (digit - 'A') + 10;
    } else if (digit >= 'a' && digit <= 'f') {
        escaped |= (digit - 'a') + 10;
    } else {
        throw createParseException(null, "Expected unicode hex escape character: " +
                (char) digit + " (" + digit + ")", false);
    }
}

reusableBuffer.append((char) escaped);

which is not sufficient because there are illegal combinations of encoded
Unicode characters (as well as illegal single characters) which I will now attempt
to show/describe.

The JSON spec states that any character may be escaped. The method
to escape characters outside of the Basic Multilingual Plane (U+0000
through U+FFFF) is as follows:

`To escape an extended character that is not in the Basic Multilingual
Plane, the character is represented as a 12-character sequence,
encoding the UTF-16 surrogate pair. So, for example, a string
containing only the G clef character (U+1D11E) may be represented as
"\uD834\uDD1E".

In other words, characters above U+FFFF are encoded as surrogate pairs.
Thus, the following JSON: "\uD800\uDC00" should decode to the Unicode
character: U+10000.

The following JSON Strings:

1.) "\uD800"
2.) "\uD800\uCC00"

Should fail parsing because a high surrogate character (U+D800) is:

in 1.) not followed by another Unicode character
in 2.) not followed by a low surrogate character

In other words, the following tests should not fail:

@Test
public void testFailBustedString8() {
    try {
        // High-surrogate character not followed by another Unicode character
        JsonParser.any().from("\"\\uD800\"");
        fail();
    } catch (JsonParserException e) {
        testException(e, 1, 7); // 7 may be the wrong char, but is irrelevant for the issue
    }
}

@Test
public void testFailBustedString9() {
    try {
        // High-surrogate character not followed by a low-surrogate character
        JsonParser.any().from("\"\\uD800\\uCC00\"");
        fail();
    } catch (JsonParserException e) {
        testException(e, 1, 7); // 7 may be the wrong char, but is irrelevant for the issue
    }
}

Thanks for your time, let me know if anything is not clear.

[mmastrac]

Great explanation. You are correct - the parser probably not accept these. However, as Javascript's JSON methods do support stringify and parse illegal surrogates (at least Firefox did in my quick test), this should probably be gated behind a parser flag that isn't set by default.

Ideally the parser and writer would have a "strict utf16 surrogate" mode, ie:

JsonWriter.strictSurrogateSupport().indent("  ") ...