5 Commits

Author	SHA1	Message	Date
Patrick von Platen	cc36064960	up (#13988)	2021-10-14 10:54:20 +02:00
Nicolas Patry	5c7789d416	Fixing by correctly raising UnicodeDecodeError. (#13449)	2021-09-07 16:45:45 +02:00
Nicolas Patry	cf4eb8b3f9	Adding a test for multibytes unicode. (#13447) ... * Adding a test for multibytes unicode. * Adding some accents. * Making sure decoding works. * Make tests passing by being cheesy.	2021-09-06 16:11:23 +02:00
SaulLu	7223844df9	Change how "additional_special_tokens" argument in the ".from_pretrained" method of the tokenizer is taken into account (#13056) ... * add test * add change in PretrainedTokenizerBase * change Luke * deactivate * add the possibility to add additional special tokens for M2M100 * format * add special test for canine * proposed changes for mbart * proposed changes for mbart50 * proposed changes for byt5 * proposed changes for canine * proposed changes for t5 * test fast and slow * remove comment * remove comment * add fast version for all tests * replace break by continue * add more comments * add check to avoid duplicates * remove comment * format * proposed change for wave2vec2 * reverse changes mbart * uncomment * format	2021-08-23 14:35:18 +02:00
Patrick von Platen	47a98fc4cb	ByT5 model (#11971) ... * allow tf to use uneven num of layers * add tokenizer * finish docs * finish docs * Apply suggestions from code review * include in index * finish * Update docs/source/model_doc/byt5.rst Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com> * apply sylvais suggestions * make style Co-authored-by: NielsRogge <48327001+NielsRogge@users.noreply.github.com>	2021-06-01 19:07:37 +01:00