From 0fe6e435b6890a5993047e13a3a38a2e5e6e4dde Mon Sep 17 00:00:00 2001 From: Stas Bekman Date: Thu, 17 Sep 2020 08:58:49 -0700 Subject: [PATCH] [model cards] ported allenai Deep Encoder, Shallow Decoder models (#7153) * [model cards] ported allenai Deep Encoder, Shallow Decoder models * typo * fix references * add allenai/wmt19-de-en-6-6 model cards * fill-in the missing info for the build script as provided by the searcher. --- .../allenai/wmt16-en-de-12-1/README.md | 95 +++++++++++++++++++ .../allenai/wmt16-en-de-dist-12-1/README.md | 95 +++++++++++++++++++ .../allenai/wmt16-en-de-dist-6-1/README.md | 95 +++++++++++++++++++ .../allenai/wmt19-de-en-6-6-base/README.md | 91 ++++++++++++++++++ .../allenai/wmt19-de-en-6-6-big/README.md | 91 ++++++++++++++++++ scripts/fsmt/gen-card-allenai-wmt19.py | 17 +++- 6 files changed, 482 insertions(+), 2 deletions(-) create mode 100644 model_cards/allenai/wmt16-en-de-12-1/README.md create mode 100644 model_cards/allenai/wmt16-en-de-dist-12-1/README.md create mode 100644 model_cards/allenai/wmt16-en-de-dist-6-1/README.md create mode 100644 model_cards/allenai/wmt19-de-en-6-6-base/README.md create mode 100644 model_cards/allenai/wmt19-de-en-6-6-big/README.md diff --git a/model_cards/allenai/wmt16-en-de-12-1/README.md b/model_cards/allenai/wmt16-en-de-12-1/README.md new file mode 100644 index 00000000000..a79089024d5 --- /dev/null +++ b/model_cards/allenai/wmt16-en-de-12-1/README.md @@ -0,0 +1,95 @@ + +--- + +language: en, de +thumbnail: +tags: +- translation +- wmt16 +- allenai +license: Apache 2.0 +datasets: +- http://www.statmt.org/wmt16/ ([test-set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)) + +metrics: +- http://www.statmt.org/wmt16/metrics-task.html +--- + +# FSMT + +## Model description + +This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de. + +For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). + +All 3 models are available: + +* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1) +* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1) +* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1) + +``` +@misc{kasai2020deep, + title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}, + author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}, + year={2020}, + eprint={2006.10369}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Intended uses & limitations + +#### How to use + +```python +from transformers.tokenization_fsmt import FSMTTokenizer +from transformers.modeling_fsmt import FSMTForConditionalGeneration +mname = "allenai/wmt16-en-de-12-1" +tokenizer = FSMTTokenizer.from_pretrained(mname) +model = FSMTForConditionalGeneration.from_pretrained(mname) + +input = "Machine learning is great, isn't it?" +input_ids = tokenizer.encode(input, return_tensors="pt") +outputs = model.generate(input_ids) +decoded = tokenizer.decode(outputs[0], skip_special_tokens=True) +print(decoded) # Maschinelles Lernen ist großartig, nicht wahr? + +``` + +#### Limitations and bias + + +## Training data + +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). + +## Eval results + +Here are the BLEU scores: + +model | fairseq | transformers +-------|---------|---------- +wmt16-en-de-12-1 | 26.9 | 25.75 + +The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs. + +The score was calculated using this code: + +```bash +git clone https://github.com/huggingface/transformers +cd transformers +export PAIR=en-de +export DATA_DIR=data/$PAIR +export SAVE_DIR=data/$PAIR +export BS=8 +export NUM_BEAMS=5 +mkdir -p $DATA_DIR +sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source +sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target +echo $PAIR +PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-12-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS +``` + diff --git a/model_cards/allenai/wmt16-en-de-dist-12-1/README.md b/model_cards/allenai/wmt16-en-de-dist-12-1/README.md new file mode 100644 index 00000000000..dd5f70da88c --- /dev/null +++ b/model_cards/allenai/wmt16-en-de-dist-12-1/README.md @@ -0,0 +1,95 @@ + +--- + +language: en, de +thumbnail: +tags: +- translation +- wmt16 +- allenai +license: Apache 2.0 +datasets: +- http://www.statmt.org/wmt16/ ([test-set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)) + +metrics: +- http://www.statmt.org/wmt16/metrics-task.html +--- + +# FSMT + +## Model description + +This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de. + +For more details, please see, [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). + +All 3 models are available: + +* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1) +* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1) +* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1) + +``` +@misc{kasai2020deep, + title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}, + author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}, + year={2020}, + eprint={2006.10369}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Intended uses & limitations + +#### How to use + +```python +from transformers.tokenization_fsmt import FSMTTokenizer +from transformers.modeling_fsmt import FSMTForConditionalGeneration +mname = "allenai/wmt16-en-de-dist-12-1" +tokenizer = FSMTTokenizer.from_pretrained(mname) +model = FSMTForConditionalGeneration.from_pretrained(mname) + +input = "Machine learning is great, isn't it?" +input_ids = tokenizer.encode(input, return_tensors="pt") +outputs = model.generate(input_ids) +decoded = tokenizer.decode(outputs[0], skip_special_tokens=True) +print(decoded) # Maschinelles Lernen ist großartig, nicht wahr? + +``` + +#### Limitations and bias + + +## Training data + +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). + +## Eval results + +Here are the BLEU scores: + +model | fairseq | transformers +-------|---------|---------- +wmt16-en-de-dist-12-1 | 28.3 | 27.52 + +The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs. + +The score was calculated using this code: + +```bash +git clone https://github.com/huggingface/transformers +cd transformers +export PAIR=en-de +export DATA_DIR=data/$PAIR +export SAVE_DIR=data/$PAIR +export BS=8 +export NUM_BEAMS=5 +mkdir -p $DATA_DIR +sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source +sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target +echo $PAIR +PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-dist-12-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS +``` + diff --git a/model_cards/allenai/wmt16-en-de-dist-6-1/README.md b/model_cards/allenai/wmt16-en-de-dist-6-1/README.md new file mode 100644 index 00000000000..29c94125f53 --- /dev/null +++ b/model_cards/allenai/wmt16-en-de-dist-6-1/README.md @@ -0,0 +1,95 @@ + +--- + +language: en, de +thumbnail: +tags: +- translation +- wmt16 +- allenai +license: Apache 2.0 +datasets: +- http://www.statmt.org/wmt16/ ([test-set](http://matrix.statmt.org/test_sets/newstest2016.tgz?1504722372)) + +metrics: +- http://www.statmt.org/wmt16/metrics-task.html +--- + +# FSMT + +## Model description + +This is a ported version of fairseq-based [wmt16 transformer](https://github.com/jungokasai/deep-shallow/) for en-de. + +For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). + +All 3 models are available: + +* [wmt16-en-de-dist-12-1](https://huggingface.co/allenai/wmt16-en-de-dist-12-1) +* [wmt16-en-de-dist-6-1](https://huggingface.co/allenai/wmt16-en-de-dist-6-1) +* [wmt16-en-de-12-1](https://huggingface.co/allenai/wmt16-en-de-12-1) + +``` +@misc{kasai2020deep, + title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}, + author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}, + year={2020}, + eprint={2006.10369}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Intended uses & limitations + +#### How to use + +```python +from transformers.tokenization_fsmt import FSMTTokenizer +from transformers.modeling_fsmt import FSMTForConditionalGeneration +mname = "allenai/wmt16-en-de-dist-6-1" +tokenizer = FSMTTokenizer.from_pretrained(mname) +model = FSMTForConditionalGeneration.from_pretrained(mname) + +input = "Machine learning is great, isn't it?" +input_ids = tokenizer.encode(input, return_tensors="pt") +outputs = model.generate(input_ids) +decoded = tokenizer.decode(outputs[0], skip_special_tokens=True) +print(decoded) # Maschinelles Lernen ist großartig, nicht wahr? + +``` + +#### Limitations and bias + + +## Training data + +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). + +## Eval results + +Here are the BLEU scores: + +model | fairseq | transformers +-------|---------|---------- +wmt16-en-de-dist-6-1 | 27.4 | 27.11 + +The score is slightly below the score reported in the paper, as the researchers don't use `sacrebleu` and measure the score on tokenized outputs. `transformers` score was measured using `sacrebleu` on detokenized outputs. + +The score was calculated using this code: + +```bash +git clone https://github.com/huggingface/transformers +cd transformers +export PAIR=en-de +export DATA_DIR=data/$PAIR +export SAVE_DIR=data/$PAIR +export BS=8 +export NUM_BEAMS=5 +mkdir -p $DATA_DIR +sacrebleu -t wmt16 -l $PAIR --echo src > $DATA_DIR/val.source +sacrebleu -t wmt16 -l $PAIR --echo ref > $DATA_DIR/val.target +echo $PAIR +PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt16-en-de-dist-6-1 $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS +``` + diff --git a/model_cards/allenai/wmt19-de-en-6-6-base/README.md b/model_cards/allenai/wmt19-de-en-6-6-base/README.md new file mode 100644 index 00000000000..6b841b74e87 --- /dev/null +++ b/model_cards/allenai/wmt19-de-en-6-6-base/README.md @@ -0,0 +1,91 @@ + +--- + +language: de, en +thumbnail: +tags: +- translation +- wmt19 +- allenai +license: Apache 2.0 +datasets: +- http://www.statmt.org/wmt19/ ([test-set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)) +metrics: +- http://www.statmt.org/wmt19/metrics-task.html +--- + +# FSMT + +## Model description + +This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for de-en. + +For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). + +2 models are available: + +* [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big) +* [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base) + +``` +@misc{kasai2020deep, + title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}, + author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}, + year={2020}, + eprint={2006.10369}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Intended uses & limitations + +#### How to use + +```python +from transformers.tokenization_fsmt import FSMTTokenizer +from transformers.modeling_fsmt import FSMTForConditionalGeneration +mname = "allenai/wmt19-de-en-6-6-base" +tokenizer = FSMTTokenizer.from_pretrained(mname) +model = FSMTForConditionalGeneration.from_pretrained(mname) + +input = "Maschinelles Lernen ist großartig, nicht wahr?" +input_ids = tokenizer.encode(input, return_tensors="pt") +outputs = model.generate(input_ids) +decoded = tokenizer.decode(outputs[0], skip_special_tokens=True) +print(decoded) # Machine learning is great, isn't it? + +``` + +#### Limitations and bias + + +## Training data + +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). + +## Eval results + +Here are the BLEU scores: + +model | transformers +-------|---------|---------- +wmt19-de-en-6-6-base | 38.37 + +The score was calculated using this code: + +```bash +git clone https://github.com/huggingface/transformers +cd transformers +export PAIR=de-en +export DATA_DIR=data/$PAIR +export SAVE_DIR=data/$PAIR +export BS=8 +export NUM_BEAMS=5 +mkdir -p $DATA_DIR +sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source +sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target +echo $PAIR +PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt19-de-en-6-6-base $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS +``` + diff --git a/model_cards/allenai/wmt19-de-en-6-6-big/README.md b/model_cards/allenai/wmt19-de-en-6-6-big/README.md new file mode 100644 index 00000000000..271a56b80e9 --- /dev/null +++ b/model_cards/allenai/wmt19-de-en-6-6-big/README.md @@ -0,0 +1,91 @@ + +--- + +language: de, en +thumbnail: +tags: +- translation +- wmt19 +- allenai +license: Apache 2.0 +datasets: +- http://www.statmt.org/wmt19/ ([test-set](http://matrix.statmt.org/test_sets/newstest2019.tgz?1556572561)) +metrics: +- http://www.statmt.org/wmt19/metrics-task.html +--- + +# FSMT + +## Model description + +This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for de-en. + +For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). + +2 models are available: + +* [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big) +* [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base) + +``` +@misc{kasai2020deep, + title={Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}, + author={Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}, + year={2020}, + eprint={2006.10369}, + archivePrefix={arXiv}, + primaryClass={cs.CL} +} +``` + +## Intended uses & limitations + +#### How to use + +```python +from transformers.tokenization_fsmt import FSMTTokenizer +from transformers.modeling_fsmt import FSMTForConditionalGeneration +mname = "allenai/wmt19-de-en-6-6-big" +tokenizer = FSMTTokenizer.from_pretrained(mname) +model = FSMTForConditionalGeneration.from_pretrained(mname) + +input = "Maschinelles Lernen ist großartig, nicht wahr?" +input_ids = tokenizer.encode(input, return_tensors="pt") +outputs = model.generate(input_ids) +decoded = tokenizer.decode(outputs[0], skip_special_tokens=True) +print(decoded) # Machine learning is great, isn't it? + +``` + +#### Limitations and bias + + +## Training data + +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). + +## Eval results + +Here are the BLEU scores: + +model | transformers +-------|---------|---------- +wmt19-de-en-6-6-big | 39.9 + +The score was calculated using this code: + +```bash +git clone https://github.com/huggingface/transformers +cd transformers +export PAIR=de-en +export DATA_DIR=data/$PAIR +export SAVE_DIR=data/$PAIR +export BS=8 +export NUM_BEAMS=5 +mkdir -p $DATA_DIR +sacrebleu -t wmt19 -l $PAIR --echo src > $DATA_DIR/val.source +sacrebleu -t wmt19 -l $PAIR --echo ref > $DATA_DIR/val.target +echo $PAIR +PYTHONPATH="src:examples/seq2seq" python examples/seq2seq/run_eval.py allenai/wmt19-de-en-6-6-big $DATA_DIR/val.source $SAVE_DIR/test_translations.txt --reference_path $DATA_DIR/val.target --score_path $SAVE_DIR/test_bleu.json --bs $BS --task translation --num_beams $NUM_BEAMS +``` + diff --git a/scripts/fsmt/gen-card-allenai-wmt19.py b/scripts/fsmt/gen-card-allenai-wmt19.py index a4be917db13..0214e5190d3 100755 --- a/scripts/fsmt/gen-card-allenai-wmt19.py +++ b/scripts/fsmt/gen-card-allenai-wmt19.py @@ -42,13 +42,26 @@ metrics: ## Model description -This is a ported version of fairseq-based wmt19 transformer created by [jungokasai]](https://github.com/jungokasai/) @ allenai for {src_lang}-{tgt_lang}. +This is a ported version of fairseq-based [wmt19 transformer](https://github.com/jungokasai/deep-shallow/) for {src_lang}-{tgt_lang}. + +For more details, please, see [Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation](https://arxiv.org/abs/2006.10369). 2 models are available: * [wmt19-de-en-6-6-big](https://huggingface.co/allenai/wmt19-de-en-6-6-big) * [wmt19-de-en-6-6-base](https://huggingface.co/allenai/wmt19-de-en-6-6-base) +``` +@misc{{kasai2020deep, + title={{Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation}}, + author={{Jungo Kasai and Nikolaos Pappas and Hao Peng and James Cross and Noah A. Smith}}, + year={{2020}}, + eprint={{2006.10369}}, + archivePrefix={{arXiv}}, + primaryClass={{cs.CL}} +}} +``` + ## Intended uses & limitations #### How to use @@ -73,7 +86,7 @@ print(decoded) # {texts[tgt_lang]} ## Training data -Pretrained weights were left identical to the original model released by the researcher. +Pretrained weights were left identical to the original model released by allenai. For more details, please, see the [paper](https://arxiv.org/abs/2006.10369). ## Eval results