c0f8d055ce
* toctree * not-doctested.txt * collapse sections * feedback * update * rewrite get started sections * fixes * fix * loading models * fix * customize models * share * fix link * contribute part 1 * contribute pt 2 * fix toctree * tokenization pt 1 * Add new model (#32615) * v1 - working version * fix * fix * fix * fix * rename to correct name * fix title * fixup * rename files * fix * add copied from on tests * rename to `FalconMamba` everywhere and fix bugs * fix quantization + accelerate * fix copies * add `torch.compile` support * fix tests * fix tests and add slow tests * copies on config * merge the latest changes * fix tests * add few lines about instruct * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix * fix tests --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * "to be not" -> "not to be" (#32636) * "to be not" -> "not to be" * Update sam.md * Update trainer.py * Update modeling_utils.py * Update test_modeling_utils.py * Update test_modeling_utils.py * fix hfoption tag * tokenization pt. 2 * image processor * fix toctree * backbones * feature extractor * fix file name * processor * update not-doctested * update * make style * fix toctree * revision * make fixup * fix toctree * fix * make style * fix hfoption tag * pipeline * pipeline gradio * pipeline web server * add pipeline * fix toctree * not-doctested * prompting * llm optims * fix toctree * fixes * cache * text generation * fix * chat pipeline * chat stuff * xla * torch.compile * cpu inference * toctree * gpu inference * agents and tools * gguf/tiktoken * finetune * toctree * trainer * trainer pt 2 * optims * optimizers * accelerate * parallelism * fsdp * update * distributed cpu * hardware training * gpu training * gpu training 2 * peft * distrib debug * deepspeed 1 * deepspeed 2 * chat toctree * quant pt 1 * quant pt 2 * fix toctree * fix * fix * quant pt 3 * quant pt 4 * serialization * torchscript * scripts * tpu * review * model addition timeline * modular * more reviews * reviews * fix toctree * reviews reviews * continue reviews * more reviews * modular transformers * more review * zamba2 * fix * all frameworks * pytorch * supported model frameworks * flashattention * rm check_table * not-doctested.txt * rm check_support_list.py * feedback * updates/feedback * review * feedback * fix * update * feedback * updates * update --------- Co-authored-by: Younes Belkada <49240599+younesbelkada@users.noreply.github.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> Co-authored-by: Quentin Gallouédec <45557362+qgallouedec@users.noreply.github.com>
8.1 KiB
RoBERTa-PreLayerNorm
Overview
The RoBERTa-PreLayerNorm model was proposed in fairseq: A Fast, Extensible Toolkit for Sequence Modeling by Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, Michael Auli.
It is identical to using the --encoder-normalize-before flag in fairseq.
The abstract from the paper is the following:
fairseq is an open-source sequence modeling toolkit that allows researchers and developers to train custom models for translation, summarization, language modeling, and other text generation tasks. The toolkit is based on PyTorch and supports distributed training across multiple GPUs and machines. We also support fast mixed-precision training and inference on modern GPUs.
This model was contributed by andreasmaden. The original code can be found here.
Usage tips
- The implementation is the same as Roberta except instead of using Add and Norm it does Norm and Add. Add and Norm refers to the Addition and LayerNormalization as described in Attention Is All You Need.
- This is identical to using the
--encoder-normalize-beforeflag in fairseq.
Resources
- Text classification task guide
- Token classification task guide
- Question answering task guide
- Causal language modeling task guide
- Masked language modeling task guide
- Multiple choice task guide
RobertaPreLayerNormConfig
autodoc RobertaPreLayerNormConfig
RobertaPreLayerNormModel
autodoc RobertaPreLayerNormModel - forward
RobertaPreLayerNormForCausalLM
autodoc RobertaPreLayerNormForCausalLM - forward
RobertaPreLayerNormForMaskedLM
autodoc RobertaPreLayerNormForMaskedLM - forward
RobertaPreLayerNormForSequenceClassification
autodoc RobertaPreLayerNormForSequenceClassification - forward
RobertaPreLayerNormForMultipleChoice
autodoc RobertaPreLayerNormForMultipleChoice - forward
RobertaPreLayerNormForTokenClassification
autodoc RobertaPreLayerNormForTokenClassification - forward
RobertaPreLayerNormForQuestionAnswering
autodoc RobertaPreLayerNormForQuestionAnswering - forward
TFRobertaPreLayerNormModel
autodoc TFRobertaPreLayerNormModel - call
TFRobertaPreLayerNormForCausalLM
autodoc TFRobertaPreLayerNormForCausalLM - call
TFRobertaPreLayerNormForMaskedLM
autodoc TFRobertaPreLayerNormForMaskedLM - call
TFRobertaPreLayerNormForSequenceClassification
autodoc TFRobertaPreLayerNormForSequenceClassification - call
TFRobertaPreLayerNormForMultipleChoice
autodoc TFRobertaPreLayerNormForMultipleChoice - call
TFRobertaPreLayerNormForTokenClassification
autodoc TFRobertaPreLayerNormForTokenClassification - call
TFRobertaPreLayerNormForQuestionAnswering
autodoc TFRobertaPreLayerNormForQuestionAnswering - call
FlaxRobertaPreLayerNormModel
autodoc FlaxRobertaPreLayerNormModel - call
FlaxRobertaPreLayerNormForCausalLM
autodoc FlaxRobertaPreLayerNormForCausalLM - call
FlaxRobertaPreLayerNormForMaskedLM
autodoc FlaxRobertaPreLayerNormForMaskedLM - call
FlaxRobertaPreLayerNormForSequenceClassification
autodoc FlaxRobertaPreLayerNormForSequenceClassification - call
FlaxRobertaPreLayerNormForMultipleChoice
autodoc FlaxRobertaPreLayerNormForMultipleChoice - call
FlaxRobertaPreLayerNormForTokenClassification
autodoc FlaxRobertaPreLayerNormForTokenClassification - call
FlaxRobertaPreLayerNormForQuestionAnswering
autodoc FlaxRobertaPreLayerNormForQuestionAnswering - call