yzy5630
a1fe4ba9c9
use new API for save and load
2019-07-18 15:45:23 +08:00
yzy5630
a7ba27b1b4
add parser for adam
2019-07-18 08:52:51 +08:00
yzy5630
d6522e2873
change loss and optimizer to new API
2019-07-17 21:22:34 +08:00
yzy5630
123da5a2fa
fix errors for lm_finetuning examples
2019-07-17 09:56:07 +08:00
yzy5630
60a1bdcdac
fix some errors for distributed lm_finetuning
2019-07-17 09:16:20 +08:00
thomwolf
f31154cb9d
Merge branch 'xlnet'
2019-07-16 11:51:13 +02:00
thomwolf
2397f958f9
updating examples and doc
2019-07-14 23:20:10 +02:00
thomwolf
0bab55d5d5
[BIG] name change
2019-07-05 11:55:36 +02:00
thomwolf
c41f2bad69
WIP XLM + refactoring
2019-07-03 22:54:39 +02:00
Mayhul Arora
08ff056c43
Added option to use multiple workers to create training data for lm fine tuning
2019-06-26 16:16:12 -07:00
Rocketknight1
c7b2808ed7
Update LM finetuning README to include a literature reference
2019-06-22 15:04:01 +01:00
Thomas Wolf
cad88e19de
Merge pull request #672 from oliverguhr/master
...
Add vocabulary and model config to the finetune output
2019-06-14 17:02:47 +02:00
Thomas Wolf
460d9afd45
Merge pull request #640 from Barqawiz/master
...
Support latest multi language bert fine tune
2019-06-14 16:57:02 +02:00
Oliver Guhr
5c08c8c273
adds the tokenizer + model config to the output
2019-06-11 13:46:33 +02:00
jeonsworld
a3a604cefb
Update pregenerate_training_data.py
...
apply Whole Word Masking technique.
referred to [create_pretraining_data.py](https://github.com/google-research/bert/blob/master/create_pretraining_data.py )
2019-06-10 12:17:23 +09:00
Ahmad Barqawi
c4fe56dcc0
support latest multi language bert fine tune
...
fix issue of bert-base-multilingual and add support for uncased multilingual
2019-05-27 11:27:41 +02:00
burcturkoglu
00c7fd2b79
Division to num_train_optimizer of global_step in lr_this_step is removed.
2019-05-09 10:57:03 +03:00
burcturkoglu
fa37b4da77
Merge branch 'master' of https://github.com/huggingface/pytorch-pretrained-BERT
2019-05-09 10:55:24 +03:00
burcturkoglu
5289b4b9e0
Division to num_train_optimizer of global_step in lr_this_step is removed.
2019-05-09 10:51:38 +03:00
MottoX
74dbba64bc
Prepare optimizer only when args.do_train is True
2019-05-02 19:09:29 +08:00
thomwolf
d94c6b0144
fix training schedules in examples to match new API
2019-04-23 11:17:06 +02:00
Matthew Carrigan
b8e2a9c584
Made --reduce_memory actually do something in finetune_on_pregenerated
2019-04-22 14:01:48 +01:00
Matthew Carrigan
dbbd6c7500
Replaced some randints with cleaner randranges, and added a helpful
...
error for users whose corpus is just one giant document.
2019-04-12 15:07:58 +01:00
Thomas Wolf
2cdfb8b254
Merge pull request #467 from yaroslavvb/patch-2
...
Update README.md
2019-04-11 21:53:23 +02:00
thomwolf
4bc4c69af9
finetuning any BERT model - fixes #455
2019-04-11 16:57:59 +02:00
Yaroslav Bulatov
8fffba5f47
Update README.md
...
Fix for
```> > > > 04/09/2019 21:39:38 - INFO - __main__ - device: cuda n_gpu: 1, distributed training: False, 16-bits training: False
Traceback (most recent call last):
File "/home/ubuntu/pytorch-pretrained-BERT/examples/lm_finetuning/simple_lm_finetuning.py", line 642, in <module>
main()
File "/home/ubuntu/pytorch-pretrained-BERT/examples/lm_finetuning/simple_lm_finetuning.py", line 502, in main
raise ValueError("Training is currently the only implemented execution option. Please set `do_train`.")
ValueError: Training is currently the only implemented execution option. Please set `do_train`.
```
2019-04-09 14:45:47 -07:00
jeonsworld
60005f464d
Update pregenerate_training_data.py
...
If the value of rand_end is returned from the randint function, the value of sampled_doc_index that matches current_idx is returned from searchsorted.
example:
cumsum_max = {int64} 30
doc_cumsum = {ndarray} [ 5 7 11 19 30]
doc_lengths = {list} <class 'list'>: [5, 2, 4, 8, 11]
if current_idx = 1,
rand_start = 7
rand_end = 35
sentence_index = randint(7, 35) % cumsum_max
if randint return 35, sentence_index becomes 5.
if sentence_index is 5, np.searchsorted returns 1 equal to current_index.
2019-03-30 14:50:17 +09:00
thomwolf
361aff6de5
typos
2019-03-27 11:54:59 +01:00
thomwolf
cea8ba1d59
adjusted formating and some wording in the readme
2019-03-27 11:53:44 +01:00
Matthew Carrigan
24e67fbf75
Minor README update
2019-03-25 12:33:30 +00:00
Matthew Carrigan
8d1d1ffde2
Corrected the displayed loss when gradient_accumulation_steps > 1
2019-03-25 12:15:19 +00:00
Matthew Carrigan
abb7d1ff6d
Added proper context management to ensure cleanup happens in the right
...
order.
2019-03-21 17:50:03 +00:00
Matthew Carrigan
06a30cfdf3
Added a --reduce_memory option to the training script to keep training
...
data on disc as a memmap rather than in memory
2019-03-21 17:04:12 +00:00
Matthew Carrigan
7d1ae644ef
Added a --reduce_memory option to the training script to keep training
...
data on disc as a memmap rather than in memory
2019-03-21 17:02:18 +00:00
Matthew Carrigan
2bba7f810e
Added a --reduce_memory option to shelve docs to disc instead of keeping them in memory.
2019-03-21 16:50:16 +00:00
Matthew Carrigan
8733ffcb5e
Removing a couple of other old unnecessary comments
2019-03-21 14:09:57 +00:00
Matthew Carrigan
8a861048dd
Fixed up the notes on a possible future low-memory path
2019-03-21 14:08:39 +00:00
Matthew Carrigan
a8a577ba93
Reduced memory usage for pregenerating the data a lot by writing it
...
out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.
2019-03-21 14:05:52 +00:00
Matthew Carrigan
0ae59e662d
Reduced memory usage for pregenerating the data a lot by writing it
...
out on the fly without shuffling - the Sampler in the finetuning script
will shuffle for us.
2019-03-21 14:04:17 +00:00
Matthew Carrigan
6a9038ba53
Removed an old irrelevant comment
2019-03-21 13:36:41 +00:00
Matthew Carrigan
29a392fbcf
Small README changes
2019-03-20 17:35:17 +00:00
Matthew Carrigan
832b2b0058
Adding README
2019-03-20 17:31:49 +00:00
Matthew Carrigan
934d3f4d2f
Syncing up argument names between the scripts
2019-03-20 17:23:23 +00:00
Matthew Carrigan
f19ba35b2b
Move old finetuning script into the new folder
2019-03-20 16:47:06 +00:00
Matthew Carrigan
7de5c6aa5e
PEP8 and formatting cleanups
2019-03-20 16:44:04 +00:00
Matthew Carrigan
1798e98e5a
Added final TODOs
2019-03-20 16:42:37 +00:00
Matthew Carrigan
c64c2fc4c2
Fixed embarrassing indentation problem
2019-03-20 15:42:57 +00:00
Matthew Carrigan
0540d360f2
Fixed logging
2019-03-20 15:36:51 +00:00
Matthew Carrigan
976554a472
First commit of the new LM finetuning
2019-03-20 14:23:51 +00:00
