Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(tokenizer): Introduce events #404

Merged
merged 19 commits into from
Feb 27, 2022
Merged

refactor(tokenizer): Introduce events #404

merged 19 commits into from
Feb 27, 2022

Commits on Feb 7, 2022

  1. refactor(tokenizer): Introduce events

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    aa8bf73 View commit details
    Browse the repository at this point in the history

  2. Move EOF creation & emission to _emitEOFToken

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    6efaa6d View commit details
    Browse the repository at this point in the history

  3. Port SAX parser to use tokenizer events

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    9249638 View commit details
    Browse the repository at this point in the history

  4. Rename tokenizer classes

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    9b59041 View commit details
    Browse the repository at this point in the history

  5. Flatten text & eof events 

    No need to construct objects, when flat parameters suffice
    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    764498e View commit details
    Browse the repository at this point in the history

  6. Shorten callback names

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    6c9e816 View commit details
    Browse the repository at this point in the history

  7. Simplify _emitCurrentCharacterToken

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    5b7b487 View commit details
    Browse the repository at this point in the history

  8. refactor(tokenizer): Move onParseError to handler

    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    4a16907 View commit details
    Browse the repository at this point in the history

  9. Use tokenizer events for generateTokenizationTests 

    This allows us to start asserting expected values as we walk through the document. This will give us stack traces for all future errors.

Improve generate tokenization tests imports
    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    a85f632 View commit details
    Browse the repository at this point in the history

  10. Use tokenizer events for location-info tests 

    Once again, stack traces will make errors easier to identify
    @fb55
    fb55 committed Feb 7, 2022
    Configuration menu
    Copy the full SHA
    f212403 View commit details
    Browse the repository at this point in the history

Commits on Feb 8, 2022

  1. Directly emit by token type

    @fb55
    fb55 committed Feb 8, 2022
    Configuration menu
    Copy the full SHA
    3f36ff8 View commit details
    Browse the repository at this point in the history

Commits on Feb 11, 2022

  1. Reintroduce character & eof tokens

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    a82d4c4 View commit details
    Browse the repository at this point in the history

  2. Add abstract class SinglePathHandler

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    ff0efa0 View commit details
    Browse the repository at this point in the history

  3. Remove unused queued tokenizer properties

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    a28d45d View commit details
    Browse the repository at this point in the history

  4. Move queued handler to its own file

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    936a523 View commit details
    Browse the repository at this point in the history

  5. Revert unwrapping of currentCharacterToken

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    be2ab48 View commit details
    Browse the repository at this point in the history

  6. Add some comments

    @fb55
    fb55 committed Feb 11, 2022
    Configuration menu
    Copy the full SHA
    8631aa5 View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2022

  1. Merge remote-tracking branch 'upstream/master' into refactor/tokenize… 

    …r-events
    @fb55
    fb55 committed Feb 26, 2022
    Configuration menu
    Copy the full SHA
    2c411d3 View commit details
    Browse the repository at this point in the history

  2. Fix comments

    @fb55
    fb55 committed Feb 26, 2022
    Configuration menu
    Copy the full SHA
    d65af65 View commit details
    Browse the repository at this point in the history