transformers/docs/source
Stas Bekman 1eeb206bef
[ported model] FSMT (FairSeq MachineTranslation) (#6940)
* ready for PR

* cleanup

* correct FSMT_PRETRAINED_MODEL_ARCHIVE_LIST

* fix

* perfectionism

* revert change from another PR

* odd, already committed this one

* non-interactive upload workaround

* backup the failed experiment

* store langs in config

* workaround for localizing model path

* doc clean up as in https://github.com/huggingface/transformers/pull/6956

* style

* back out debug mode

* document: run_eval.py --num_beams 10

* remove unneeded constant

* typo

* re-use bart's Attention

* re-use EncoderLayer, DecoderLayer from bart

* refactor

* send to cuda and fp16

* cleanup

* revert (moved to another PR)

* better error message

* document run_eval --num_beams

* solve the problem of tokenizer finding the right files when model is local

* polish, remove hardcoded config

* add a note that the file is autogenerated to avoid losing changes

* prep for org change, remove unneeded code

* switch to model4.pt, update scores

* s/python/bash/

* missing init (but doesn't impact the finetuned model)

* cleanup

* major refactor (reuse-bart)

* new model, new expected weights

* cleanup

* cleanup

* full link

* fix model type

* merge porting notes

* style

* cleanup

* have to create a DecoderConfig object to handle vocab_size properly

* doc fix

* add note (not a public class)

* parametrize

* - add bleu scores integration tests

* skip test if sacrebleu is not installed

* cache heavy models/tokenizers

* some tweaks

* remove tokens that aren't used

* more purging

* simplify code

* switch to using decoder_start_token_id

* add doc

* Revert "major refactor (reuse-bart)"

This reverts commit 226dad15ca.

* decouple from bart

* remove unused code #1

* remove unused code #2

* remove unused code #3

* update instructions

* clean up

* move bleu eval to examples

* check import only once

* move data+gen script into files

* reuse via import

* take less space

* add prepare_seq2seq_batch (auto-tested)

* cleanup

* recode test to use json instead of yaml

* ignore keys not needed

* use the new -y in transformers-cli upload -y

* [xlm tok] config dict: fix str into int to match definition (#7034)

* [s2s] --eval_max_generate_length (#7018)

* Fix CI with change of name of nlp (#7054)

* nlp -> datasets

* More nlp -> datasets

* Woopsie

* More nlp -> datasets

* One last

* extending to support allen_nlp wmt models

- allow a specific checkpoint file to be passed
- more arg settings
- scripts for allen_nlp models

* sync with changes

* s/fsmt-wmt/wmt/ in model names

* s/fsmt-wmt/wmt/ in model names (p2)

* s/fsmt-wmt/wmt/ in model names (p3)

* switch to a better checkpoint

* typo

* make non-optional args such - adjust tests where possible or skip when there is no other choice

* consistency

* style

* adjust header

* cards moved (model rename)

* use best custom hparams

* update info

* remove old cards

* cleanup

* s/stas/facebook/

* update scores

* s/allen_nlp/allenai/

* url maps aren't needed

* typo

* move all the doc / build /eval generators to their own scripts

* cleanup

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* Apply suggestions from code review

Co-authored-by: Lysandre Debut <lysandre@huggingface.co>

* fix indent

* duplicated line

* style

* use the correct add_start_docstrings

* oops

* resizing can't be done with the core approach, due to 2 dicts

* check that the arg is a list

* style

* style

Co-authored-by: Sam Shleifer <sshleifer@gmail.com>
Co-authored-by: Sylvain Gugger <35901082+sgugger@users.noreply.github.com>
Co-authored-by: Lysandre Debut <lysandre@huggingface.co>
2020-09-17 11:31:29 -04:00
..
_static Update docs stable version 2020-09-01 11:02:24 -04:00
imgs Guide to fixed-length model perplexity evaluation (#5449) 2020-07-07 16:04:15 -06:00
internal Doc pipelines (#6175) 2020-08-03 11:44:46 -04:00
main_classes Compute loss method (#7074) 2020-09-11 12:06:31 -04:00
model_doc [ported model] FSMT (FairSeq MachineTranslation) (#6940) 2020-09-17 11:31:29 -04:00
benchmarks.rst Small docfile fixes (#6328) 2020-08-10 05:37:12 -04:00
bertology.rst [doc] Fix broken links + remove crazy big notebook 2020-05-07 18:44:18 -04:00
conf.py Release: v3.1.0 2020-09-01 14:27:52 +02:00
contributing.md Update installation page and add contributing to the doc (#5084) 2020-06-17 14:01:10 -04:00
converting_tensorflow_models.rst Add ALBERT to the Tensorflow to Pytorch model conversion cli (#3933) 2020-05-11 13:10:00 -04:00
custom_datasets.rst add intro to nlp lib & dataset links to custom datasets tutorial (#6583) 2020-08-20 10:32:51 -04:00
examples.md per_device instead of per_gpu/error thrown when argument unknown (#4618) 2020-05-27 11:36:55 -04:00
favicon.ico Adding usage examples for common tasks (#2850) 2020-02-25 13:48:24 -05:00
glossary.rst minor docs grammar fixes (#6889) 2020-09-02 06:45:19 -04:00
index.rst [ported model] FSMT (FairSeq MachineTranslation) (#6940) 2020-09-17 11:31:29 -04:00
installation.md [doc] fix invalid env vars (#6504) 2020-08-17 11:11:40 +08:00
migration.md Add hugs (#5225) 2020-06-24 07:56:14 -04:00
model_sharing.rst add -y to bypass prompt for transformers-cli upload (#7035) 2020-09-10 04:58:29 -04:00
model_summary.rst Funnel transformer (#6908) 2020-09-08 08:08:08 -04:00
multilingual.rst Refactor Code samples; Test code samples (#5036) 2020-06-25 16:46:00 -04:00
notebooks.md Update notebooks (#3620) 2020-04-06 14:32:39 -04:00
perplexity.rst tiny ppl doc typo fix (#5751) 2020-07-14 10:39:44 -06:00
philosophy.rst typos (#6505) 2020-08-17 10:57:36 +08:00
preprocessing.rst Fix #6575 (#6596) 2020-08-19 13:04:33 -04:00
pretrained_models.rst Funnel transformer (#6908) 2020-09-08 08:08:08 -04:00
quicktour.rst minor docs grammar fixes (#6889) 2020-09-02 06:45:19 -04:00
serialization.rst fix torchscript docs (#6740) 2020-08-26 04:51:56 -04:00
task_summary.rst replace _ with __ rst links (#6541) 2020-08-17 12:27:02 -04:00
testing.rst [doc] improve/expand the Parametrization section (#7156) 2020-09-16 08:45:50 -04:00
tokenizer_summary.rst [docs] Fix number of 'ug' occurrences in tokenizer_summary (#6574) 2020-08-18 10:23:25 -04:00
training.rst Import accuracy_score (#6480) 2020-08-14 08:16:16 -04:00