Anthony MOI
734d29b03d
tokenizers is now a real dependency
2019-12-24 13:32:41 -05:00
Anthony MOI
2818e50569
Add tests for fast tokenizers
2019-12-24 13:29:01 -05:00
Anthony MOI
31c56f2e0b
Fix style
2019-12-24 12:43:27 -05:00
Anthony MOI
951ae99bea
BertTokenizerFast
2019-12-24 12:24:24 -05:00
Anthony MOI
041eac2d6d
GPT2TokenizerFast
2019-12-24 12:24:14 -05:00
Anthony MOI
3471ff0d35
FastPreTrainedTokenizer
2019-12-24 12:23:30 -05:00
Thomas Wolf
81db12c3ba
Merge pull request #2271 from aaugustin/improve-setup-and-requirements
...
Improve setup and requirements
2019-12-24 11:21:20 +01:00
Aymeric Augustin
10724a8123
Run the slow tests every Monday morning.
2019-12-24 09:09:43 +01:00
Aymeric Augustin
a8d34e534e
Remove [--editable] in install instructions.
...
Use -e only in docs targeted at contributors.
If a user copy-pastes command line with [--editable], they will hit
an error. If they don't know the --editable option, we're giving them
a choice to make before they can move forwards, but this isn't a choice
they need to make right now.
2019-12-24 08:46:08 +01:00
Aymeric Augustin
072750f4dc
Merge pull request #2288 from aaugustin/better-handle-optional-imports
...
Improve handling of optional imports
2019-12-23 22:28:47 +01:00
Aymeric Augustin
4621ad6f9d
Use the same pattern as everywhere else.
...
This is really just for consistency.
2019-12-23 21:30:04 +01:00
Aymeric Augustin
a31d4a2971
Reraise ImportError when sentencepiece isn't installed.
...
Else, the next line fails with a confusion exception because the spm
variable isn't defined.
2019-12-23 21:27:42 +01:00
Aymeric Augustin
c8b0c1e551
Improve exception type.
...
ImportError isn't really appropriate when there's no import involved.
2019-12-23 21:27:38 +01:00
Aymeric Augustin
4c09a96096
Simplify re-raising exceptions.
...
Most module use the simpler `raise` version. Normalize those that don't.
2019-12-23 21:20:54 +01:00
Aymeric Augustin
5565dcdd35
Remove warning when scikit-learn isn't available.
...
Most users don't need it.
2019-12-23 21:16:26 +01:00
Aymeric Augustin
8a6881822a
Run some tests on Python 3.7.
...
This will improve version coverage.
2019-12-23 21:06:23 +01:00
Aymeric Augustin
7a865821d9
Remove stray egg-info directory automatically.
...
If a user or contributor ran `pip install -e .` on transformers < 3.0,
pip created a transformers.egg-info directory next to the transformers
directory at the root of the repository.
In transformers 3.0, the source is in a `src` subdirectory.
`pip install -e .` creates a transformers.egg-info directory there.
However, pip will still pick transformers.egg-info from the previous
location. This is a bug: https://github.com/pypa/pip/issues/5466
Users and contributors are likely to hit this problem because the
documentation for transformers 3.0 relies heavily on extra_requires
which didn't exist in earlier versions, so aren't defined in a stale
transformers.egg-info directory.
If such a directory exists, remove it. It's autogenerated, gitignored
and not supposed to contain anything of value.
2019-12-23 21:06:23 +01:00
Aymeric Augustin
70373a5f7c
Update contribution instructions.
...
Also provide shortcuts in a Makefile.
2019-12-23 21:05:30 +01:00
Aymeric Augustin
c3783399db
Remove redundant requirements with transformers.
2019-12-23 19:17:27 +01:00
Aymeric Augustin
d79e9c9a9a
Remove docs/requirements.txt.
...
It's superseded by the "docs" extras.
2019-12-23 19:17:07 +01:00
Aymeric Augustin
d73eb552e8
Remove requirements.txt.
...
It's redundant with setup.py and, also, incomplete (e.g. numpy).
2019-12-23 19:15:08 +01:00
Aymeric Augustin
9fcc532df6
Remove requirements-dev.txt.
...
It was generated once, likely in a non-reproducible way (pip freeze
in a contributor's local environment), and never updated.
2019-12-23 19:14:36 +01:00
Aymeric Augustin
76a1417f2a
Include all optional dependencies in extras.
...
Take advantage of this to simplify the Circle CI configuration.
Don't bother with tensorboardX: it's a fallback for PyTorch < 1.1.0.
2019-12-23 19:14:31 +01:00
Aymeric Augustin
9fc8dcb2a0
Standardize import.
...
Every other file uses this pattern.
2019-12-23 18:45:42 +01:00
Aymeric Augustin
f2522869ea
Review and update setup.py.
2019-12-23 18:45:42 +01:00
Aymeric Augustin
23dad8447c
Install deps from setup.py for building docs.
...
requirements.txt isn't up to date.
2019-12-23 17:06:32 +01:00
Aymeric Augustin
d8e33dbd67
Fix path to source code in docs config.
...
This should fix API docs, which went AWOL with yesterday's changes.
2019-12-23 16:49:35 +01:00
thomwolf
59b123bc50
fix tqdm logging level
2019-12-23 16:47:24 +01:00
Thomas Wolf
ba2378ced5
Merge pull request #2264 from upura/fix-doclink
...
Fix doc link in README
2019-12-23 12:31:00 +01:00
Thomas Wolf
e4e2a666c9
Merge pull request #2276 from ShnitzelKiller/scatterfix
...
fix error due to wrong argument name to Tensor.scatter()
2019-12-23 12:19:48 +01:00
James Noeckel
398bb03f98
fix out-of-place call to scatter, whose named argument name is source, not src
2019-12-22 23:30:52 -08:00
Aymeric Augustin
ce50305e5b
Merge pull request #2270 from aaugustin/remove-python-2
...
Remove support for Python 2
2019-12-22 23:04:37 +01:00
Aymeric Augustin
1a948d7020
Switch from comments to annotations for types.
2019-12-22 18:56:01 +01:00
Aymeric Augustin
1c62e87b34
Use built-in open().
...
On Python 3, `open is io.open`.
2019-12-22 18:38:56 +01:00
Aymeric Augustin
d6eaf4e6d2
Update comments mentioning Python 2.
2019-12-22 18:38:56 +01:00
Aymeric Augustin
45841eaf7b
Remove references to Python 2 in documentation.
2019-12-22 18:38:56 +01:00
Aymeric Augustin
0dddc1494d
Remove py3 marker.
2019-12-22 18:38:56 +01:00
Aymeric Augustin
75a23d24af
Remove import fallbacks.
2019-12-22 18:38:56 +01:00
Aymeric Augustin
798b3b3899
Remove sys.version_info[0] == 2 or 3.
2019-12-22 18:38:42 +01:00
Aymeric Augustin
8af25b1664
Remove six.
2019-12-22 17:56:09 +01:00
Aymeric Augustin
6b2200fc88
Remove u-prefixes.
2019-12-22 17:47:54 +01:00
Aymeric Augustin
c824d15aa1
Remove __future__ imports.
2019-12-22 17:47:54 +01:00
Aymeric Augustin
b6ea0f43ae
Remove duplicate -v flag.
2019-12-22 17:47:27 +01:00
Thomas Wolf
5daca95ddd
Merge pull request #2268 from aaugustin/improve-repository-structure
...
Improve repository structure
2019-12-22 16:41:53 +01:00
Thomas Wolf
54abc67aec
Merge pull request #2255 from aaugustin/implement-best-practices
...
Implement some Python best practices
2019-12-22 16:31:11 +01:00
Aymeric Augustin
00204f2b4c
Replace CommonTestCases for tokenizers with a mixin.
...
This is the same change as for (TF)CommonTestCases for modeling.
2019-12-22 15:35:25 +01:00
Aymeric Augustin
a3c5883f2c
Rename file for consistency.
2019-12-22 15:35:25 +01:00
Aymeric Augustin
daf8bebcdd
Remove unused GPTModelTester.
...
It isn't imported anywhere.
2019-12-22 15:35:25 +01:00
Aymeric Augustin
345c23a60f
Replace (TF)CommonTestCases for modeling with a mixin.
...
I suspect the wrapper classes were created in order to prevent the
abstract base class (TF)CommonModelTester from being included in test
discovery and running, because that would fail.
I solved this by replacing the abstract base class with a mixin.
Code changes are just de-indenting and automatic reformattings
performed by black to use the extra line space.
2019-12-22 15:35:18 +01:00
Aymeric Augustin
7e98e211f0
Remove unittest.main() in test modules.
...
This construct isn't used anymore these days.
Running python tests/test_foo.py puts the tests/ directory on
PYTHONPATH, which isn't representative of how we run tests.
Use python -m unittest tests/test_foo.py instead.
2019-12-22 14:42:03 +01:00