From 1a9381c60df2e095b794d64c97d5bf278413564b Mon Sep 17 00:00:00 2001 From: Patrick von Platen Date: Mon, 25 Oct 2021 19:49:51 +0200 Subject: [PATCH] Update README.md --- examples/pytorch/speech-pretraining/README.md | 31 ++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/examples/pytorch/speech-pretraining/README.md b/examples/pytorch/speech-pretraining/README.md index 3efac8dd2bf..ec413f95fec 100644 --- a/examples/pytorch/speech-pretraining/README.md +++ b/examples/pytorch/speech-pretraining/README.md @@ -85,7 +85,36 @@ The results of this run can be seen [here](https://wandb.ai/patrickvonplaten/wav ### Base -TODO (currently running...) +To pre-train `"base-sized"` Wav2Vec2 model, *e.g.* [facebook/wav2vec2-base](https://huggingface.co/facebook/wav2vec2-base) +on [librispeech_asr](https://huggingface.co/datasets/librispeech_asr), the following command can be run: + +accelerate launch run_wav2vec2_pretraining_no_trainer.py \ + --dataset_name=librispeech_asr \ + --dataset_config_names clean clean other \ + --dataset_split_names train.100 train.360 train.500 \ + --model_name_or_path="patrickvonplaten/wav2vec2-base-v2" \ + --output_dir="./wav2vec2-pretrained-demo" \ + --max_train_steps="200000" \ + --num_warmup_steps="32000" \ + --gradient_accumulation_steps="4" \ + --learning_rate="0.001" \ + --weight_decay="0.01" \ + --max_duration_in_seconds="20.0" \ + --min_duration_in_seconds="2.0" \ + --logging_steps="1" \ + --saving_steps="10000" \ + --per_device_train_batch_size="8" \ + --per_device_eval_batch_size="8" \ + --adam_beta1="0.9" \ + --adam_beta2="0.98" \ + --adam_epsilon="1e-06" \ + --gradient_checkpointing \ + +The experiment was run on 8 GPU V100 (16 GB RAM each) for 4 days. +In case you have more than 8 GPUs available for a higher effective `batch_size`, +it is recommended to increase the `learning_rate` to `0.005` for faster convergence. + +The results of this run can be seen [here](https://wandb.ai/patrickvonplaten/test/reports/Wav2Vec2-Base--VmlldzoxMTUyODQ0?accessToken=rg6e8u9yizx964k8q47zctq1m4afpvtn1i3qi9exgdmzip6xwkfzvagfajpzj55n) and the checkpoint pretrained for 120,000 steps can be accessed [here](https://huggingface.co/patrickvonplaten/wav2vec2-base-repro-960h-libri-85k-steps) ### Large