riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-06 14:20:04 +06:00
Code Issues Actions 16 Packages Projects Releases Wiki Activity
6,762 Commits 861 Branches 225 Tags 2.1 GiB
9f8619c6aa
Commit Graph

58 Commits

Author SHA1 Message Date
Thomas Wolf
9f5f646442
Merge pull request #2211 from huggingface/fast-tokenizers 
Fast tokenizers
 2019-12-27 10:24:29 +01:00
Anthony MOI
2818e50569
Add tests for fast tokenizers 2019-12-24 13:29:01 -05:00
Aymeric Augustin
e6c0019c80 Remove unused variables in tests. 2019-12-23 22:38:18 +01:00
Aymeric Augustin
1c62e87b34 Use built-in open(). 
On Python 3, `open is io.open`.
 2019-12-22 18:38:56 +01:00
Aymeric Augustin
798b3b3899 Remove sys.version_info[0] == 2 or 3. 2019-12-22 18:38:42 +01:00
Aymeric Augustin
c824d15aa1 Remove __future__ imports. 2019-12-22 17:47:54 +01:00
Aymeric Augustin
00204f2b4c Replace CommonTestCases for tokenizers with a mixin. 
This is the same change as for (TF)CommonTestCases for modeling.
 2019-12-22 15:35:25 +01:00
Aymeric Augustin
a3c5883f2c Rename file for consistency. 2019-12-22 15:35:25 +01:00