Commits

Branch selector

 master

User selector

n1t0

All time

Commit History

Commits on Feb 10, 2021

• Fix tokenizers training in notebook (huggingface#10110)

  n1t0

  committedFeb 10, 2021
  1fbaa3c

Commits on Feb 8, 2021

• Update tokenizers requirement (huggingface#10077)

  n1t0

  committedFeb 8, 2021
  f285e4c

Commits on Jan 18, 2021

• Remove duplicated extra["retrieval"] (huggingface#9621)

  n1t0

  committedJan 18, 2021
  72fc9ab

Commits on Oct 23, 2020

• Fix BatchEncoding.word_to_tokens for removed tokens (huggingface#7939)

  n1t0

  committedOct 23, 2020
  5e32301

Commits on Jul 6, 2020

• Various tokenizers fixes (huggingface#5558)

  n1t0

  committedJul 6, 2020
  5787e4c

Commits on Jun 25, 2020

• Fix convert_graph_to_onnx (huggingface#5230)

  n1t0

  committedJun 25, 2020
  0e1fce3

Commits on Jun 15, 2020

• [HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (huggingface#4510)

  

  3 peoplecommittedJun 15, 2020
  3643422

Commits on May 28, 2020

• Fix add_special_tokens on fast tokenizers (huggingface#4531)

  n1t0

  committedMay 28, 2020
  5e73701

Commits on May 22, 2020

• Fix convert_token_type_ids_from_sequences for fast tokenizers (huggingface#4503)

  n1t0

  committedMay 22, 2020
  35df911

Commits on Apr 22, 2020

• Bump tokenizers version to final 0.7.0 (huggingface#3898)

  n1t0

  committedApr 22, 2020
  13dd2ac

Commits on Apr 10, 2020

• Update tokenizers to 0.7.0-rc5 (huggingface#3705)

  n1t0

  committedApr 10, 2020
  b7cf9f4

Commits on Dec 27, 2019

• Hotfix tokenizers version for sdist installs

  n1t0

  committedDec 27, 2019
  bfe870b

Commits on Dec 26, 2019

• Better added_tokens handling

  n1t0

  committedDec 26, 2019
  e6ec24f

• Code style update

  n1t0

  committedDec 26, 2019
  599db13

• Handle unk_token

  n1t0

  committedDec 26, 2019
  835b76a

• FastPreTrainedTokenizer => PreTrainedTokenizerFast

  n1t0

  committedDec 26, 2019
  7ead04c

• Update for changes in tokenizers API

  n1t0

  committedDec 26, 2019
  1f82a5d

Commits on Dec 24, 2019

• tokenizers is now a real dependency

  n1t0

  committedDec 24, 2019
  734d29b

• Add tests for fast tokenizers

  n1t0

  committedDec 24, 2019
  2818e50

• Fix style

  n1t0

  committedDec 24, 2019
  31c56f2

• BertTokenizerFast

  n1t0

  committedDec 24, 2019
  951ae99

• GPT2TokenizerFast

  n1t0

  committedDec 24, 2019
  041eac2

• FastPreTrainedTokenizer

  n1t0

  committedDec 24, 2019
  3471ff0

Commits on Jul 31, 2019

• Fix small typos

  n1t0

  committedJul 31, 2019
  f2a3eb9