riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-05 22:00:09 +06:00
Code Issues Actions 21 Packages Projects Releases Wiki Activity
19,383 Commits 861 Branches 225 Tags 2.1 GiB
remove-script-datasets-in-tests-test-datasets-main
Commit Graph

2613 Commits

Author SHA1 Message Date
dhanajitb
f872eb98c2
making unconditional generation work 
The unconditional generation works now but if the seed is fixed, the sample is the same every time.
n_samples > 1 will give different samples though.
I am giving the start token as '<|endoftext|>' for the unconditional generation.
 2019-03-28 22:46:15 +05:30
Thomas Wolf
694e2117f3
Merge pull request #388 from ananyahjha93/master 
Added remaining GLUE tasks to 'run_classifier.py'
 2019-03-28 09:06:53 +01:00
Thomas Wolf
cc8c2d2332
Merge pull request #396 from IndexFziQ/IndexFziQ 
add tqdm to the process of eval in examples/run_swag.py
 2019-03-27 12:03:26 +01:00
thomwolf
361aff6de5 typos 2019-03-27 11:54:59 +01:00
thomwolf
cea8ba1d59 adjusted formating and some wording in the readme 2019-03-27 11:53:44 +01:00
Matthew Carrigan
24e67fbf75 Minor README update 2019-03-25 12:33:30 +00:00
Matthew Carrigan
8d1d1ffde2 Corrected the displayed loss when gradient_accumulation_steps > 1 2019-03-25 12:15:19 +00:00
Matthew Carrigan
abb7d1ff6d Added proper context management to ensure cleanup happens in the right 
order.
 2019-03-21 17:50:03 +00:00
Matthew Carrigan
06a30cfdf3 Added a --reduce_memory option to the training script to keep training 
data on disc as a memmap rather than in memory
 2019-03-21 17:04:12 +00:00
Matthew Carrigan
7d1ae644ef Added a --reduce_memory option to the training script to keep training 
data on disc as a memmap rather than in memory
 2019-03-21 17:02:18 +00:00
Matthew Carrigan
2bba7f810e Added a --reduce_memory option to shelve docs to disc instead of keeping them in memory. 2019-03-21 16:50:16 +00:00
Matthew Carrigan
8733ffcb5e Removing a couple of other old unnecessary comments 2019-03-21 14:09:57 +00:00
Matthew Carrigan
8a861048dd Fixed up the notes on a possible future low-memory path 2019-03-21 14:08:39 +00:00
Matthew Carrigan
a8a577ba93 Reduced memory usage for pregenerating the data a lot by writing it 
out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.
 2019-03-21 14:05:52 +00:00
Matthew Carrigan
0ae59e662d Reduced memory usage for pregenerating the data a lot by writing it 
out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.
 2019-03-21 14:04:17 +00:00
Matthew Carrigan
6a9038ba53 Removed an old irrelevant comment 2019-03-21 13:36:41 +00:00
Yuqiang Xie
77944d1b31
add tqdm to the process of eval 
Maybe better.
 2019-03-21 20:59:33 +08:00
Matthew Carrigan
29a392fbcf Small README changes 2019-03-20 17:35:17 +00:00
Matthew Carrigan
832b2b0058 Adding README 2019-03-20 17:31:49 +00:00
Matthew Carrigan
934d3f4d2f Syncing up argument names between the scripts 2019-03-20 17:23:23 +00:00
Matthew Carrigan
f19ba35b2b Move old finetuning script into the new folder 2019-03-20 16:47:06 +00:00
Matthew Carrigan
7de5c6aa5e PEP8 and formatting cleanups 2019-03-20 16:44:04 +00:00
Matthew Carrigan
1798e98e5a Added final TODOs 2019-03-20 16:42:37 +00:00
Matthew Carrigan
c64c2fc4c2 Fixed embarrassing indentation problem 2019-03-20 15:42:57 +00:00
Matthew Carrigan
0540d360f2 Fixed logging 2019-03-20 15:36:51 +00:00
Matthew Carrigan
976554a472 First commit of the new LM finetuning 2019-03-20 14:23:51 +00:00
Ananya Harsh Jha
e5b63fb542 Merge branch 'master' of https://github.com/ananyahjha93/pytorch-pretrained-BERT 
pull current master to local
 2019-03-17 08:30:13 -04:00
Ananya Harsh Jha
8a4e90ff40 corrected folder creation error for MNLI-MM, verified GLUE results 2019-03-17 08:16:50 -04:00
Ananya Harsh Jha
e0bf01d9a9 added hack for mismatched MNLI 2019-03-16 14:10:48 -04:00
Ananya Harsh Jha
4c721c6b6a added eval time metrics for GLUE tasks 2019-03-15 23:21:24 -04:00
tseretelitornike
83857ffeaa
Added missing imports. 2019-03-15 12:45:48 +01:00
Yongbo Wang
d1e4fa98a9
typo in annotation 
modify `heruistic` to `heuristic` in line 660, `charcter` to `character` in line 661.
 2019-03-14 17:32:15 +08:00
Yongbo Wang
3d6452163d
typo 
modify `mull` to `null` in line 474 annotation.
 2019-03-14 17:03:38 +08:00
thomwolf
a98dfe4ced fixing #377 (empty nbest_predictions.json) 2019-03-14 09:57:06 +01:00
Ananya Harsh Jha
043c8781ef added code for all glue task processors 2019-03-14 04:24:04 -04:00
Yongbo Wang
22a465a91f
Simplify code, delete redundancy line 
delete redundancy line `if args.train`, simplify code.
 2019-03-13 09:42:06 +08:00
Elon Musk
66d8206809
Update run_gpt2.py 2019-03-08 11:59:08 -05:00
thomwolf
7cc35c3104 fix openai gpt example and updating readme 2019-03-06 11:43:21 +01:00
thomwolf
994d86609b fixing PYTORCH_PRETRAINED_BERT_CACHE use in examples 2019-03-06 10:21:24 +01:00
thomwolf
5c85fc3977 fix typo - logger info 2019-03-06 10:05:21 +01:00
Thomas Wolf
8e36da7acb
Merge pull request #347 from jplehmann/feature/sst2-processor 
Processor for SST-2 task
 2019-03-06 09:48:27 +01:00
Thomas Wolf
3c01dfb775
Merge pull request #338 from CatalinVoss/patch-3 
Fix top k generation for k != 0
 2019-03-06 09:47:33 +01:00
John Lehmann
0f96d4b1f7 Run classifier processor for SST-2. 2019-03-05 13:38:28 -06:00
Catalin Voss
4b4b079272
Fix top k generation for k != 0 2019-03-02 21:54:44 -08:00
Catalin Voss
c0cf0a04d5
Fix typo 2019-02-27 18:01:06 -08:00
Ben Johnson
8607233679
Update run_openai_gpt.py 2019-02-20 13:58:54 -05:00
thomwolf
0202da0271 remove unnecessary example 2019-02-18 13:51:42 +01:00
thomwolf
690a0dbf36 fix example - masking 2019-02-18 10:50:30 +01:00
thomwolf
fbb248a2e4 examples testing 2019-02-18 01:28:18 +01:00
thomwolf
b65f07d8c0 adding examples 2019-02-18 00:55:33 +01:00
wlhgtc
8efaf8f176
fix 'best_non_null_entry' is None error 2019-02-15 15:57:25 +08:00
Davide Fiocco
65df0d78ed
--do_lower_case is duplicated in parser args 
Deleting one repetition (please review!)
 2019-02-13 15:30:05 +01:00
Thomas Wolf
03cdb2a390
Merge pull request #254 from huggingface/python_2 
Adding OpenAI GPT and Transformer-XL models, compatibility with Python 2
 2019-02-11 14:19:26 +01:00
thomwolf
d38caba169 typo in run_squad 2019-02-11 14:10:27 +01:00
thomwolf
af62cc5f20 fix run_squad example 2019-02-11 14:06:32 +01:00
thomwolf
eebc8abbe2 clarify and unify model saving logic in examples 2019-02-11 14:04:19 +01:00
thomwolf
32fea876bb add distant debugging to run_transfo_xl 2019-02-11 12:53:32 +01:00
thomwolf
b31ba23913 cuda on in the examples by default 2019-02-11 12:15:43 +01:00
thomwolf
6cd769957e update transfo xl example 2019-02-09 16:59:17 +01:00
thomwolf
1320e4ec0c mc_token_mask => mc_token_ids 2019-02-09 16:58:53 +01:00
thomwolf
f4a07a392c mems not splitted 2019-02-09 16:14:31 +01:00
thomwolf
43b9af0cac mems initialized to None in run_transfo 2019-02-09 16:12:19 +01:00
thomwolf
b80684b23f fixing run openai gpt example 2019-02-08 22:31:32 +01:00
thomwolf
7b4b0cf966 logging 2019-02-08 11:16:29 +01:00
thomwolf
4bbb9f2d68 log loss - helpers 2019-02-08 11:14:29 +01:00
thomwolf
5d7e845712 fix model on cuda 2019-02-08 11:08:43 +01:00
thomwolf
eccb2f0163 hot fix 2019-02-08 11:05:20 +01:00
thomwolf
5adc20723b add distant debugging 2019-02-08 11:03:59 +01:00
thomwolf
777459b471 run openai example running 2019-02-08 10:33:14 +01:00
thomwolf
6bc082da0a updating examples 2019-02-08 00:02:26 +01:00
thomwolf
e77721e4fe renamed examples 2019-02-07 23:15:15 +01:00
thomwolf
d482e3d79d adding examples for openai and transformer-xl 2019-02-07 17:06:41 +01:00
tholor
9aebc711c9 adjust error message related to args.do_eval 2019-02-07 11:49:38 +01:00
tholor
4a450b25d5 removing unused argument eval_batch_size from LM finetuning #256 2019-02-07 10:06:38 +01:00
Baoyang Song
7ac3311e48
Fix the undefined variable in squad example 2019-02-06 19:36:08 +01:00
thomwolf
ed47cb6cba fixing transfo eval script 2019-02-06 16:22:17 +01:00
Thomas Wolf
848aae49e1
Merge branch 'master' into python_2 2019-02-06 00:13:20 +01:00
thomwolf
448937c00d python 2 compatibility 2019-02-06 00:07:46 +01:00
thomwolf
d609ba24cb resolving merge conflicts 2019-02-05 16:14:25 +01:00
Thomas Wolf
64ce900974
Merge pull request #248 from JoeDumoulin/squad1.1-fix 
fix prediction on run-squad.py example
 2019-02-05 16:00:51 +01:00
Thomas Wolf
e9e77cd3c4
Merge pull request #218 from matej-svejda/master 
Fix learning rate problems in run_classifier.py
 2019-02-05 15:40:44 +01:00
thomwolf
1579c53635 more explicit notation: num_train_step => num_train_optimization_steps 2019-02-05 15:36:33 +01:00
joe dumoulin
aa90e0c36a fix prediction on run-squad.py example 2019-02-01 10:15:44 -08:00
Thomas Wolf
8f8bbd4a4c
Merge pull request #244 from deepset-ai/prettify_lm_masking 
Avoid confusion of inplace LM masking
 2019-02-01 12:17:50 +01:00
tholor
ce75b169bd avoid confusion of inplace masking of tokens_a / tokens_b 2019-01-31 11:42:06 +01:00
Surya Kasturi
9bf528877e
Update run_squad.py 2019-01-30 15:09:31 -05:00
Surya Kasturi
af2b78601b
Update run_squad2.py 2019-01-30 15:08:56 -05:00
Matej Svejda
5169069997 make examples consistent, revert error in num_train_steps calculation 2019-01-30 11:47:25 +01:00
Matej Svejda
9c6a48c8c3 fix learning rate/fp16 and warmup problem for all examples 2019-01-27 14:07:24 +01:00
Matej Svejda
01ff4f82ba learning rate problems in run_classifier.py 2019-01-22 23:40:06 +01:00
liangtaiwan
be9fa192f0 don't save if do not train 2019-01-18 00:41:55 +08:00
thomwolf
a28dfc8659 fix eval for wt103 2019-01-16 11:18:19 +01:00
thomwolf
8831c68803 fixing various parts of model conversion, loading and weights sharing 2019-01-16 10:31:16 +01:00
thomwolf
bcd4aa8fe0 update evaluation example 2019-01-15 23:32:34 +01:00
thomwolf
a69ec2c722 improved corpus and tokenization conversion - added evaluation script 2019-01-15 23:17:46 +01:00
Thomas Wolf
4e0cba1053
Merge pull request #191 from nhatchan/20190113_py35_finetune 
lm_finetuning compatibility with Python 3.5
 2019-01-14 09:40:07 +01:00
nhatchan
6c65cb2492 lm_finetuning compatibility with Python 3.5 
dicts are not ordered in Python 3.5 or prior, which is a cause of #175.
This PR replaces one with a list, to keep its order.
 2019-01-13 21:09:13 +09:00
Li Dong
a2da2b4109
[bug fix] args.do_lower_case is always True 
The "default=True" makes args.do_lower_case always True.

```python
parser.add_argument("--do_lower_case",
                        default=True,
                        action='store_true')
```
 2019-01-13 19:51:11 +08:00
tholor
506e5bb0c8 add do_lower_case arg and adjust model saving for lm finetuning. 2019-01-11 08:32:46 +01:00
Thomas Wolf
e485829a41
Merge pull request #174 from abeljim/master 
Added Squad 2.0
 2019-01-10 23:40:45 +01:00
Sang-Kil Park
64326dccfb
Fix it to run properly even if without --do_train param. 
It was modified similar to `run_classifier.py`, and Fixed to run properly even if without `--do_train` param.
 2019-01-10 21:51:39 +09:00
thomwolf
e5c78c6684 update readme and few typos 2019-01-10 01:40:00 +01:00
thomwolf
fa5222c296 update readme 2019-01-10 01:25:28 +01:00
Unknown
b3628f117e Added Squad 2.0 2019-01-08 15:13:13 -08:00
thomwolf
ab90d4cddd adding docs and example for OpenAI GPT 2019-01-09 00:12:43 +01:00
thomwolf
2e4db64cab add do_lower_case tokenizer loading optino in run_squad and ine_tuning examples 2019-01-07 13:06:42 +01:00
thomwolf
c9fd350567 remove default when action is store_true in arguments 2019-01-07 13:01:54 +01:00
Thomas Wolf
d3d56f9a0b
Merge pull request #166 from likejazz/patch-1 
Fix error when `bert_model` param is path or url.
 2019-01-07 12:40:55 +01:00
Thomas Wolf
766c6b2ce3
Merge pull request #159 from jaderabbit/master 
Allow do_eval to be used without do_train and to use the pretrained model in the output folder
 2019-01-07 12:31:06 +01:00
Thomas Wolf
77966a43a4
Merge pull request #156 from rodgzilla/cl_args_doc 
Adding new pretrained model to the help of the `bert_model` argument.
 2019-01-07 12:27:16 +01:00
Thomas Wolf
2e8c5c00ec
Merge pull request #141 from SinghJasdeep/patch-1 
loading saved model when n_classes != 2
 2019-01-07 12:21:13 +01:00
Sang-Kil Park
ca4e7aaa72
Fix error when bert_model param is path or url. 
Error occurs when `bert_model` param is path or url. Therefore, if it is path, specify the last path to prevent error.
 2019-01-05 11:42:54 +09:00
Jade Abbott
193e2df8ba Remove rogue comment 2019-01-03 13:13:06 +02:00
Jade Abbott
c64de50ea4 nb_tr_steps is not initialized 2019-01-03 12:34:57 +02:00
Jade Abbott
b96149a19b Training loss is not initialized if only do_eval is specified 2019-01-03 10:32:10 +02:00
Jade Abbott
be3b9bcf4d Allow one to use the pretrained model in evaluation when do_train is not selected 2019-01-03 09:02:33 +02:00
Grégory Châtel
186f75342e Adding new pretrained model to the help of the bert_model argument. 2019-01-02 14:00:59 +01:00
Jasdeep Singh
99709ee61d
loading saved model when n_classes != 2 
Required to for: Assertion `t >= 0 && t < n_classes` failed,  if your default number of classes is not 2.
 2018-12-20 13:55:47 -08:00
tholor
e5fc98c542 add exemplary training data. update to nvidia apex. refactor 'item -> line in doc' mapping. add warning for unknown word. 2018-12-20 18:30:52 +01:00
deepset
a58361f197
Add example for fine tuning BERT language model (#1) 
Adds an example for loading a pre-trained BERT model and fine tune it as a language model (masked tokens & nextSentence) on your target corpus.
 2018-12-18 10:32:25 +01:00
thomwolf
ae88eb88a4 set encoding to 'utf-8' in calls to open 2018-12-14 13:48:58 +01:00
thomwolf
e1eab59aac no fp16 on evaluation 2018-12-13 14:54:02 +01:00
thomwolf
087798b7fa fix reloading model for evaluation in examples 2018-12-13 14:48:12 +01:00
thomwolf
0f544625f4 fix swag example for work with apex 2018-12-13 13:35:59 +01:00
thomwolf
0cf88ff084 make examples work without apex 2018-12-13 13:28:00 +01:00
thomwolf
d3fcec1a3e add saving and loading model in examples 2018-12-13 12:50:44 +01:00
thomwolf
b3caec5a56 adding save checkpoint and loading in examples 2018-12-13 12:48:13 +01:00
Thomas Wolf
91aab2a6d3
Merge pull request #116 from FDecaYed/deyuf/fp16_with_apex 
Change to use apex for better fp16 and multi-gpu support
 2018-12-13 12:32:37 +01:00
Thomas Wolf
ffe9075f48
Merge pull request #96 from rodgzilla/multiple-choice-code 
BertForMultipleChoice and Swag dataset example.
 2018-12-13 12:05:11 +01:00
Deyu Fu
c8ea286048 change to apex for better fp16 and multi-gpu support 2018-12-11 17:13:58 -08:00
Thomas Wolf
e622790a93
Merge pull request #91 from rodgzilla/convert-examples-code-improvement 
run_classifier.py improvements
 2018-12-11 05:12:04 -05:00
Grégory Châtel
df34f22854 Removing the dependency to pandas and using the csv module to load data. 2018-12-10 17:45:23 +01:00
Grégory Châtel
d429c15f25 Removing old code from copy-paste. 2018-12-06 19:19:21 +01:00
Grégory Châtel
63c45056aa Finishing the code for the Swag task. 2018-12-06 18:53:05 +01:00
Grégory Châtel
c45d8ac554 Storing the feature of each choice as a dict for readability. 2018-12-06 16:01:28 +01:00
Grégory Châtel
0812aee2c3 Fixing problems in convert_examples_to_features. 2018-12-06 15:53:07 +01:00
Grégory Châtel
f2b873e995 convert_examples_to_features code and small improvements. 2018-12-06 15:40:47 +01:00
Grégory Châtel
83fdbd6043 Adding read_swag_examples to load the dataset. 2018-12-06 14:02:46 +01:00
Grégory Châtel
7183cded4e SwagExample class. 2018-12-06 13:39:44 +01:00
Grégory Châtel
fa7daa247d Fixing the commentary of the SquadExample class. 2018-12-06 13:14:33 +01:00
Grégory Châtel
a994bf4076 Fixing related to issue #83. 2018-12-05 18:16:30 +01:00
Grégory Châtel
c6d9d5394e Simplifying code for easier understanding. 2018-12-05 17:53:09 +01:00
Grégory Châtel
793262e8ec Removing trailing whitespaces. 2018-12-05 17:52:39 +01:00
Davide Fiocco
e60e8a6068
Correct assignement for logits in classifier example 
I tried to address https://github.com/huggingface/pytorch-pretrained-BERT/issues/76
should be correct, but there's likely a more efficient way.
 2018-12-02 12:38:26 +01:00
Davide Fiocco
dc13e276ee
Point typo fix 2018-12-01 01:02:16 +01:00
thomwolf
89d47230d7 clean up classification model output 2018-11-30 22:54:53 +01:00
thomwolf
c588453a0f fix run_squad 2018-11-30 14:22:40 +01:00
thomwolf
0541442558 add do_lower_case in examples 2018-11-30 13:47:33 +01:00
Li Li
0aaedcc02f Bug fix in examples;correct t_total for distributed training;run prediction for full dataset 2018-11-27 01:08:37 -08:00
thomwolf
32167cdf4b remove convert_to_unicode and printable_text from examples 2018-11-26 23:33:22 +01:00
thomwolf
05053d163c update cache_dir in readme and examples 2018-11-26 10:45:13 +01:00
thomwolf
6b2136a8a9 fixing weights decay in run_squad example 2018-11-20 10:12:44 +01:00
Thomas Wolf
061eeca84a
Merge pull request #32 from xiaoda99/master 
Fix ineffective no_decay bug when using BERTAdam
 2018-11-20 10:11:46 +01:00
thomwolf
2f21497d3e fixing param.grad is None in fp16 examples 2018-11-20 10:01:21 +01:00
xiaoda99
6c4789e4e8
Fix ineffective no_decay bug 2018-11-18 16:16:21 +08:00
thomwolf
27ee0fff3c add no_cuda args in extract_features 2018-11-17 23:04:44 +01:00
thomwolf
aa50fd196f remove unused arguments in example scripts 2018-11-17 23:01:05 +01:00
thomwolf
47a7d4ec14 update examples from master 2018-11-17 12:21:35 +01:00
thomwolf
c8cba67742 clean up readme and examples 2018-11-17 12:19:16 +01:00
thomwolf
757750d6f6 fix tests 2018-11-17 11:58:14 +01:00
thomwolf
4e46affc34 updating examples 2018-11-17 10:30:54 +01:00
thomwolf
cba85a67b9 fix nan in optimizer_on_cpu 2018-11-15 21:47:41 +01:00
thomwolf
1de35b624b preparing for first release 2018-11-15 20:56:10 +01:00