145 Commits

Author	SHA1	Message	Date
thomwolf	009273dbdd	big doc update [WIP]	2019-08-04 12:14:57 +02:00
Saket Khandelwal	836e513698	Fixed small typo	2019-08-04 16:05:10 +10:00
wangfei	a24f830604	Fix comment typo	2019-08-03 12:17:06 +08:00
David Pollack	c90119e543	spelling mistake	2019-07-29 16:56:02 +02:00
thomwolf	bfbe52ec39	cleaning up example docstrings	2019-07-27 20:25:39 +02:00
thomwolf	4cc1bf81ee	typos	2019-07-27 12:08:21 +02:00
thomwolf	ac27548b25	fix unk_token test	2019-07-27 11:50:47 +02:00
thomwolf	c717d38573	dictionnary => dictionary	2019-07-26 23:30:48 +02:00
thomwolf	7b6e474c9a	fix #901	2019-07-26 21:26:44 +02:00
thomwolf	632d711411	fix #908	2019-07-26 21:14:37 +02:00
thomwolf	27b0f86d36	clean up pretrained	2019-07-26 17:09:21 +02:00
thomwolf	57e54ec070	add unk_token to gpt2	2019-07-26 17:09:07 +02:00
thomwolf	ac42049c08	add auto models and auto tokenizer	2019-07-26 17:08:59 +02:00
David Pollack	09ecf225e9	fixed the fix. tf session madness.	2019-07-26 15:20:44 +02:00
David Pollack	edfd965ac8	fix convert_to_tf	2019-07-26 14:13:46 +02:00
Thomas Wolf	6219ad7216	Merge pull request #888 from rococode/patch-1 ... Update docs for parameter rename	2019-07-25 15:01:22 +02:00
Joel Grus	ae152cec09	make save_pretrained work with added tokens ... right now it's dumping the *decoder* when it should be dumping the *encoder*. this fixes that.	2019-07-24 16:54:48 -07:00
rococo // Ron	66b15f73f0	Update docs for parameter rename ... OpenAIGPTLMHeadModel now accepts `labels` instead of `lm_labels`	2019-07-24 11:27:08 -07:00
Thomas Wolf	067923d326	Merge pull request #873 from huggingface/identity_replacement ... Add nn.Identity replacement for old PyTorch	2019-07-23 18:16:35 +02:00
Thomas Wolf	368670ac31	Merge pull request #866 from xanlsh/master ... Rework how PreTrainedModel.from_pretrained handles its arguments	2019-07-23 18:05:30 +02:00
thomwolf	1383c7b87a	Fix #869	2019-07-23 17:52:20 +02:00
Anish Moorthy	4fb56c7729	Remove unused *args parameter from PreTrainedConfig.from_pretrained	2019-07-23 10:43:01 -04:00
Anish Moorthy	e179c55490	Add docs for from_pretrained functions, rename return_unused_args	2019-07-23 10:43:01 -04:00
thomwolf	0740e63e49	updating schedules for state_dict saving	2019-07-23 15:57:18 +02:00
Thomas Wolf	c4bc66886d	Merge pull request #860 from Yiqing-Zhou/patch-1 ... read().splitlines() -> readlines()	2019-07-23 15:24:25 +02:00
Yiqing-Zhou	b1019d2a8e	token[-1] -> token.rstrip('\n')	2019-07-23 20:41:26 +08:00
thomwolf	0227b4a940	fix #827	2019-07-23 14:06:43 +02:00
Anish Moorthy	490ebbdcf7	Fix PretrainedModel.from_pretrained not passing cache_dir forward	2019-07-22 18:03:08 -04:00
Anish Moorthy	b8009cb0da	Make PreTrainedModel.from_pretrained pass unused arguments to model	2019-07-22 18:03:08 -04:00
Yiqing-Zhou	bef0c629ca	fix ... Remove '\n' before adding token into vocab	2019-07-22 22:30:49 +08:00
Yiqing-Zhou	897d0841be	read().splitlines() -> readlines() ... splitlines() does not work as what we expect here for bert-base-chinese because there is a '\u2028' (unicode line seperator) token in vocab file. Value of '\u2028'.splitlines() is ['', '']. Perhaps we should use readlines() instead.	2019-07-22 20:49:09 +08:00
Minho Ryu	cd8980e1f4	import sys twice	2019-07-17 18:12:01 +09:00
thomwolf	5fe0b378d8	adding missing docstring fix #793	2019-07-16 21:35:53 +02:00
thomwolf	ed7549bb1a	release version 1.0	2019-07-16 16:10:58 +02:00
thomwolf	4acaa65068	model in evaluation mode by default after from_pretrained	2019-07-16 15:41:57 +02:00
thomwolf	f289e6cfe4	fix docstrings	2019-07-16 15:31:21 +02:00
thomwolf	9726b229cf	model name typo	2019-07-16 15:17:45 +02:00
thomwolf	1849aa7d39	update readme and pretrained model weight files	2019-07-16 15:11:29 +02:00
thomwolf	f31154cb9d	Merge branch 'xlnet'	2019-07-16 11:51:13 +02:00
thomwolf	1b35d05d4b	update conversion scripts and __main__	2019-07-16 09:41:55 +02:00
thomwolf	352e3ff998	added migration guide to readme	2019-07-16 09:03:49 +02:00
thomwolf	3b8b0e01bb	update readme	2019-07-16 00:12:55 +02:00
thomwolf	e691fc0963	update QA models tests + run_generation	2019-07-15 17:45:24 +02:00
thomwolf	15d8b1266c	update tokenizer - update squad example for xlnet	2019-07-15 17:30:42 +02:00
thomwolf	3b469cb422	updating squad for compatibility with XLNet	2019-07-15 15:28:37 +02:00
thomwolf	8ca767f13c	clean up optimization	2019-07-15 13:49:07 +02:00
thomwolf	74a24f0fe9	clean up file_utils	2019-07-15 13:49:01 +02:00
thomwolf	ab49fafc04	update tokenization docstrings for #328	2019-07-15 12:51:23 +02:00
thomwolf	a9ab15174c	fix #328	2019-07-15 12:42:12 +02:00
thomwolf	f7cd7392fd	fixed tests	2019-07-15 12:32:19 +02:00
thomwolf	e28d8bde0d	doc on base classes	2019-07-15 12:08:06 +02:00
thomwolf	44c985facd	update doc for XLM and XLNet	2019-07-15 11:36:50 +02:00
thomwolf	0201d86015	added doc for transformer-xl	2019-07-15 10:11:09 +02:00
thomwolf	4cb489457f	added doc for openai GPT	2019-07-15 09:58:01 +02:00
thomwolf	62b8eb43c1	fix add_start_docstrings on python 2 (removed)	2019-07-15 09:49:02 +02:00
thomwolf	5bc3d0cc5b	added gpt2 doc	2019-07-15 09:40:05 +02:00
thomwolf	183fedfed5	fix doc on python2	2019-07-15 09:00:09 +02:00
thomwolf	2397f958f9	updating examples and doc	2019-07-14 23:20:10 +02:00
thomwolf	7d4b200e40	good quality generation example for GPT, GPT-2, Transfo-XL, XLNet	2019-07-13 15:25:03 +02:00
thomwolf	699bc7e86e	fix gpt-2 unk token test	2019-07-12 11:46:57 +02:00
thomwolf	7442956361	save config file	2019-07-12 11:26:16 +02:00
Thomas Wolf	c57e9d946f	Merge branch 'xlnet' into embeddings	2019-07-12 11:10:14 +02:00
thomwolf	2918b7d2a0	updating tests	2019-07-12 10:57:58 +02:00
LysandreJik	3fbceed8d2	Fix layer reference loss + previous attempted fix	2019-07-11 22:29:55 -04:00
LysandreJik	6c2ee16c04	Test suite testing the tie_weights function as well as the resize_token_embeddings function. ... Patched an issue relating to the tied weights I had introduced with the TorchScript addition. Byte order mark management in TSV glue reading.	2019-07-11 22:09:16 -04:00
LysandreJik	e3fb4310d6	From pretrained correct initialization. Unknown token handling for gpt2.	2019-07-11 18:44:29 -04:00
thomwolf	bd404735a7	embeddings resizing + tie_weights	2019-07-12 00:02:49 +02:00
LysandreJik	50e62a4cb4	fix gpt/gpt-2 from pretrained	2019-07-11 16:50:21 -04:00
thomwolf	273617b86d	update config - fix gpt/gpt-2 from pretrained	2019-07-11 22:45:03 +02:00
thomwolf	c6bf1a400d	fix test examples et model pretrained	2019-07-11 22:29:08 +02:00
thomwolf	92a782b108	fix run_glue test	2019-07-11 22:20:10 +02:00
thomwolf	ccb6947dc1	optimization tests	2019-07-11 17:39:47 +02:00
Thomas Wolf	b87eb82b4f	Merge branch 'xlnet' into doc-sphinx	2019-07-11 15:46:27 +02:00
thomwolf	b21d84b027	update examples	2019-07-11 15:37:34 +02:00
thomwolf	ec07cf5a66	rewamp optimization	2019-07-11 14:48:22 +02:00
thomwolf	4fef5919a5	updating examples	2019-07-11 12:03:08 +02:00
LysandreJik	7fdbc47822	Added the two CLM XLM pretrained checkpoints. ... Fixed file extensions for config/vocab/merges of XLM models.	2019-07-10 19:37:24 -04:00
LysandreJik	dee3e45b93	Fixed XLM weights conversion script. Added 5 new checkpoints for XLM.	2019-07-10 19:04:21 -04:00
LysandreJik	5288913bdd	All TODOs to be checked by Thom have been added.	2019-07-10 15:16:40 -04:00
LysandreJik	f773faa258	Fixed all links. Removed TPU. Changed CLI to Converting TF models. Many minor formatting adjustments. Added "TODO Lysandre filled" where necessary.	2019-07-10 14:45:56 -04:00
thomwolf	50b7e52a7f	WIP examples	2019-07-10 15:33:34 +02:00
LysandreJik	83fb311ef7	Patched warnings + Refactored XLNet's Docstrings	2019-07-09 16:38:30 -04:00
LysandreJik	8fe2c9d98e	Refactored Docstrings of BERT, GPT2, GPT, TransfoXL, XLM and XLNet.	2019-07-09 15:55:31 -04:00
thomwolf	ed6c8d37f4	fix merge	2019-07-09 17:14:52 +02:00
thomwolf	e468192e2f	Merge branch 'pytorch-transformers' into xlnet	2019-07-09 17:05:37 +02:00
thomwolf	d743f2f34e	updating test	2019-07-09 15:58:58 +02:00
thomwolf	d0efbd3cd1	update sequencesummary module	2019-07-09 15:46:43 +02:00
thomwolf	d5481cbe1b	adding tests to examples - updating summary module - coverage update	2019-07-09 15:29:42 +02:00
thomwolf	c079d7ddff	fix python 2 tests	2019-07-09 10:40:59 +02:00
thomwolf	b19786985d	unified tokenizer api and serialization + tests	2019-07-09 10:25:18 +02:00
thomwolf	1113f97f33	clean up glue example	2019-07-05 16:31:13 +02:00
thomwolf	162ba383b0	fix model loading	2019-07-05 15:57:14 +02:00
thomwolf	6dacc79d39	fix python2 tests	2019-07-05 15:11:59 +02:00
thomwolf	36bca545ff	tokenization abstract class - tests for examples	2019-07-05 15:02:59 +02:00
thomwolf	0bab55d5d5	[BIG] name change	2019-07-05 11:55:36 +02:00