2 Commits

Author	SHA1	Message	Date
Funtowicz Morgan	b08259a120	run_ner.py / bert-base-multilingual-cased can output empty tokens (#2991) ... * Use tokenizer.num_added_tokens to count number of added special_tokens instead of hardcoded numbers. Signed-off-by: Morgan Funtowicz <morgan@huggingface.co> * run_ner.py - Do not add a label to the labels_ids if word_tokens is empty. This can happen when using bert-base-multilingual-cased with an input containing an unique space. In this case, the tokenizer will output just an empty word_tokens thus leading to an non-consistent behavior over the labels_ids tokens adding one more tokens than tokens vector. Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>	2020-03-27 10:59:55 -04:00
srush	b662f0e625	Support for torch-lightning in NER examples (#2890) ... * initial pytorch lightning commit * tested multigpu * Fix learning rate schedule * black formatting * fix flake8 * isort * isort * . Co-authored-by: Check your git settings! <chris@chris-laptop>	2020-02-20 11:50:05 -05:00