79 Commits

Author	SHA1	Message	Date
thomwolf	4356f791a2	XLM passing tests	2019-09-11 11:49:54 +02:00
thomwolf	465870c33f	Xlnet working - also added simple question answering model for XLNet	2019-09-10 16:44:41 +02:00
thomwolf	16b6361792	xlnet paassing first test	2019-09-10 12:39:27 +02:00
thomwolf	32aabe8c33	WIP XLNet	2019-09-10 12:17:18 +02:00
thomwolf	6b3438df21	fixing GPT2 double head model and updating the torch version tests	2019-09-09 12:48:36 +02:00
thomwolf	b7175a2701	fixed imports in tests and gpt2 config test	2019-09-09 11:04:03 +02:00
thomwolf	728637356c	WIP GPT2	2019-09-09 10:18:55 +03:00
thomwolf	01597e5b90	add tf auto models + tests	2019-09-08 15:02:06 +03:00
thomwolf	f5c698b21a	add weights tying, attention and hidden states output tests	2019-09-08 15:02:06 +03:00
thomwolf	6dc4b6f34c	skip transfo-xl tokenizer tests with tf for now	2019-09-08 15:02:06 +03:00
thomwolf	518307dfcd	test suite independent of framework	2019-09-08 15:02:06 +03:00
thomwolf	a4704b1263	skipping tf tests if tf is not installed	2019-09-08 15:02:06 +03:00
thomwolf	ad0ab9afe9	fix test when tf is not here	2019-09-08 15:02:06 +03:00
thomwolf	d68a8fe462	add tf bert files	2019-09-08 15:02:06 +03:00
thomwolf	69bff89935	clean ups	2019-09-08 15:02:06 +03:00
thomwolf	1efb1f1660	split configuration and modeling files	2019-09-08 15:02:06 +03:00
thomwolf	7fba47b7d9	WIP reordering	2019-09-04 22:39:23 +02:00
thomwolf	e25cba78cf	WIP reodering arguments for torchscript and TF	2019-09-04 22:39:23 +02:00
thomwolf	fede4ef45d	fixing #1133	2019-09-02 02:27:39 +02:00
LysandreJik	87747518e9	Blocks deletion from already deleted heads. Necessary integration test. ... Now raises a warning when a head to be deleted already has been deleted. An integration test verifying the total pipeline (-> from config -> save model -> load model -> additional head pruning) has been added.	2019-08-31 00:33:50 -04:00
LysandreJik	719cb3738d	Pruning for GPT and GPT-2	2019-08-31 00:33:50 -04:00
LysandreJik	fc1fbae45d	XLM can be pruned	2019-08-31 00:33:50 -04:00
Lysandre	42e00cf9e1	Pruning saved to configuration first try	2019-08-31 00:33:50 -04:00
LysandreJik	d7a4c3252e	Fixed filename	2019-08-31 00:08:56 -04:00
LysandreJik	7f006cdd87	Set seed for head_masking test	2019-08-30 23:58:49 -04:00
Thomas Wolf	d483cd8e46	Merge pull request #1074 from huggingface/improved_testing ... Shortcut to special tokens' ids - fix GPT2 & RoBERTa tokenizers - improved testing for GPT/GPT-2	2019-08-30 23:18:58 +02:00
Thomas Wolf	d2f21f08f5	Merge pull request #1092 from shijie-wu/xlm-tokenization ... Added cleaned configuration properties for tokenizer with serialization - improve tokenization of XLM	2019-08-30 23:15:40 +02:00
LysandreJik	25e8389439	Tests for added AutoModels	2019-08-30 12:48:55 -04:00
thomwolf	7044ed6b05	fix tokenizers serialization	2019-08-30 17:36:11 +02:00
Thomas Wolf	cd65c41a83	Merge branch 'master' into xlm-tokenization	2019-08-30 17:15:16 +02:00
thomwolf	69da972ace	added test and debug tokenizer configuration serialization	2019-08-30 17:09:36 +02:00
Thomas Wolf	50e615f43d	Merge branch 'master' into improved_testing	2019-08-30 13:40:35 +02:00
thomwolf	ce5ef4b35d	python2 doesn't spark joy	2019-08-30 13:22:43 +02:00
thomwolf	5dd7b677ad	clean up all byte-level bpe tests	2019-08-30 12:43:08 +02:00
thomwolf	ca1a00a302	fix for python2	2019-08-30 12:29:31 +02:00
thomwolf	abe734ca1f	fix GPT-2 and RoBERTa tests to be clean now	2019-08-30 12:20:18 +02:00
thomwolf	d51f72d5de	adding shortcut to the ids of all the special tokens	2019-08-30 11:41:11 +02:00
thomwolf	912a377e90	dilbert -> distilbert	2019-08-28 13:59:42 +02:00
thomwolf	c9bce1811c	fixing model to add torchscript, embedding resizing, head pruning and masking + tests	2019-08-28 13:22:45 +02:00
thomwolf	62df4ba59a	add dilbert tokenizer and tests	2019-08-28 12:22:56 +02:00
LysandreJik	c513415b19	Dilbert tests from CommonTests	2019-08-27 23:59:00 -04:00
Lysandre	55f69a11b6	OpenAI GPT tests now extend CommonTests	2019-08-21 18:09:25 -04:00
Lysandre	47267ba556	OpenAI GPT-2 now depends on CommonTests.	2019-08-21 17:50:16 -04:00
Thomas Wolf	e753f249e1	Merge pull request #806 from wschin/fix-a-path ... Fix a path so that a test can run on Windows	2019-08-21 01:14:40 +02:00
LysandreJik	3d87991f60	Fixed error with encoding	2019-08-13 12:00:24 -04:00
LysandreJik	634a3172d8	Added integration tests for sequence builders.	2019-08-12 15:14:15 -04:00
LysandreJik	75d5f98fd2	Roberta tokenization + fixed tests (py3 + py2).	2019-08-09 15:02:13 -04:00
LysandreJik	14e970c271	Tokenization encode/decode class-based sequence handling	2019-08-09 15:01:38 -04:00
LysandreJik	fbd746bd06	Updated test architecture	2019-08-08 18:21:34 -04:00
Julien Chaumond	9d0603148b	[RoBERTa] RobertaForSequenceClassification + conversion	2019-08-08 11:24:54 -04:00