Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support for HF Tokenizers #408

Merged
merged 8 commits into from
Aug 13, 2023
Merged

Support for HF Tokenizers #408

merged 8 commits into from
Aug 13, 2023

Commits on Aug 5, 2023

  1. tokenizers output type conversion

    @guillaume-be
    guillaume-be committed Aug 5, 2023
    Configuration menu
    Copy the full SHA
    08c430e View commit details
    Browse the repository at this point in the history

Commits on Aug 6, 2023

  1. WIP hf Tokenizers support (2)

    @guillaume-be
    guillaume-be committed Aug 6, 2023
    Configuration menu
    Copy the full SHA
    8701a6a View commit details
    Browse the repository at this point in the history

  2. finalize interface methods for hf tokenizers

    @guillaume-be
    guillaume-be committed Aug 6, 2023
    Configuration menu
    Copy the full SHA
    09811d0 View commit details
    Browse the repository at this point in the history

  3. Addition of GPT2 example with hf tokenizers

    @guillaume-be
    guillaume-be committed Aug 6, 2023
    Configuration menu
    Copy the full SHA
    bd4ba1c View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2023

  1. Made hf-tokenizers optional, added doc for HFTokenizer

    @guillaume-be
    guillaume-be committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    4b49855 View commit details
    Browse the repository at this point in the history

  2. Addition of tests for hf tokenizers, addition to CI

    @guillaume-be
    guillaume-be committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    fb7d3f6 View commit details
    Browse the repository at this point in the history

  3. Updated changelog, extended documentation

    @guillaume-be
    guillaume-be committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    2e275cc View commit details
    Browse the repository at this point in the history

  4. Fix Clippy warnings

    @guillaume-be
    guillaume-be committed Aug 12, 2023
    Configuration menu
    Copy the full SHA
    6903628 View commit details
    Browse the repository at this point in the history