Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEA] Adds option to recover from invalid JSON lines in JSON tokenizer #13344

Merged
merged 28 commits into from
Jul 14, 2023
Merged

[FEA] Adds option to recover from invalid JSON lines in JSON tokenizer #13344

merged 28 commits into from
Jul 14, 2023

Commits on Jun 13, 2023

  1. adds option to recover from invalid JSON lines

    @elstehle
    elstehle committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    357eca8 View commit details
    Browse the repository at this point in the history

  2. fixes format

    @elstehle
    elstehle committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    5be6c14 View commit details
    Browse the repository at this point in the history

  3. fixes namespace

    @elstehle
    elstehle committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    7e48c35 View commit details
    Browse the repository at this point in the history

  4. resolves merge conflicts

    @elstehle
    elstehle committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    31c5cbe View commit details
    Browse the repository at this point in the history

  5. adds fst for token post-processing

    @elstehle
    elstehle committed Jun 13, 2023
    Configuration menu
    Copy the full SHA
    9b89685 View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jun 20, 2023
    Configuration menu
    Copy the full SHA
    16ca651 View commit details
    Browse the repository at this point in the history

Commits on Jun 27, 2023

  1. refactors fst and lookup tables, adds recovery mode

    @elstehle
    elstehle committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    9cacebd View commit details
    Browse the repository at this point in the history

  2. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    6cc468e View commit details
    Browse the repository at this point in the history

  3. fixes format

    @elstehle
    elstehle committed Jun 27, 2023
    Configuration menu
    Copy the full SHA
    9eed9c6 View commit details
    Browse the repository at this point in the history

Commits on Jul 4, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    346c05b View commit details
    Browse the repository at this point in the history

  2. makes test data prep run on correct stream

    @elstehle
    elstehle committed Jul 4, 2023
    Configuration menu
    Copy the full SHA
    97448ad View commit details
    Browse the repository at this point in the history

Commits on Jul 6, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    926b010 View commit details
    Browse the repository at this point in the history

  2. enum instead of bool for resetting stack on delimiter

    @elstehle
    elstehle committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    65dee53 View commit details
    Browse the repository at this point in the history

  3. adds comments on token post-processing

    @elstehle
    elstehle committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    4ad53b0 View commit details
    Browse the repository at this point in the history

  4. corrects comment in agent dfa

    @elstehle
    elstehle committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    0ec1957 View commit details
    Browse the repository at this point in the history

  5. uses new enum value for stack behavior

    @elstehle
    elstehle committed Jul 6, 2023
    Configuration menu
    Copy the full SHA
    37186f8 View commit details
    Browse the repository at this point in the history

Commits on Jul 7, 2023

  1. adds documentation on lookup table factories

    @elstehle
    elstehle committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    2391fa5 View commit details
    Browse the repository at this point in the history

  2. uses switch/case instead of if/else

    @elstehle
    elstehle committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    5c9eccc View commit details
    Browse the repository at this point in the history

  3. makes recovery_mode option an enum instead of bool

    @elstehle
    elstehle committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    b3656bb View commit details
    Browse the repository at this point in the history

  4. addresses review comments

    @elstehle
    elstehle committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    bc78ff8 View commit details
    Browse the repository at this point in the history

  5. removes raw_ptr_cast

    @elstehle
    elstehle committed Jul 7, 2023
    Configuration menu
    Copy the full SHA
    446ddbb View commit details
    Browse the repository at this point in the history

Commits on Jul 10, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    4b82e1f View commit details
    Browse the repository at this point in the history

  2. renames enum option to recover from invalid lines

    @elstehle
    elstehle committed Jul 10, 2023
    Configuration menu
    Copy the full SHA
    7e8d142 View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    777a940 View commit details
    Browse the repository at this point in the history

  2. clarifies that post-process requirements

    @elstehle
    elstehle committed Jul 12, 2023
    Configuration menu
    Copy the full SHA
    442cb11 View commit details
    Browse the repository at this point in the history

Commits on Jul 14, 2023

  1. Merge remote-tracking branch 'upstream/branch-23.08' into feature/jso… 

    …n-lines-recovery
    @elstehle
    elstehle committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    19f231b View commit details
    Browse the repository at this point in the history

  2. removes premap_op from translation table

    @elstehle
    elstehle committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    34da916 View commit details
    Browse the repository at this point in the history

  3. removes premap_op from transition table

    @elstehle
    elstehle committed Jul 14, 2023
    Configuration menu
    Copy the full SHA
    154ee00 View commit details
    Browse the repository at this point in the history