4 Commits

Author	SHA1	Message	Date
Sam Shleifer	efbc1c5a9d	[MarianTokenizer] implement save_vocabulary and other common methods (#4389)	2020-05-19 19:45:49 -04:00
Sam Shleifer	9a687ebb77	[Marian Fixes] prevent predicting pad_token_id before softmax, support language codes, name multilingual models (#4290)	2020-05-13 17:29:41 -04:00
Sam Shleifer	3487be75ef	[Marian] documentation and AutoModel support (#4152) ... - MarianSentencepieceTokenizer - > MarianTokenizer - Start using unk token. - add docs page - add better generation params to MarianConfig - more conversion utilities	2020-05-10 13:54:57 -04:00
Sam Shleifer	847e7f3379	MarianMTModel.from_pretrained('Helsinki-NLP/opus-marian-en-de') (#3908) ... Co-Authored-By: Stefan Schweter <stefan@schweter.it>	2020-04-28 18:22:37 -04:00