mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-16 11:08:23 +06:00

* added rag WIP * path fix * Formatting / renaming prior to actual work * added rag WIP * path fix * Formatting / renaming prior to actual work * added rag WIP * path fix * Formatting / renaming prior to actual work * added rag WIP * Formatting / renaming prior to actual work * First commit * improve comments * Retrieval evaluation scripts * refactor to include modeling outputs + MPI retriever * Fix rag-token model + refactor * Various fixes + finetuning logic * use_bos fix * Retrieval refactor * Finetuning refactoring and cleanup * Add documentation and cleanup * Remove set_up_rag_env.sh file * Fix retrieval wit HF index * Fix import errors * Fix quality errors * Refactor as per suggestions in https://github.com/huggingface/transformers/pull/6813#issuecomment-687208867 * fix quality * Fix RAG Sequence generation * minor cleanup plus initial tests * fix test * fix tests 2 * Comments fix * post-merge fixes * Improve readme + post-rebase refactor * Extra dependencied for tests * Fix tests * Fix tests 2 * Refactor test requirements * Fix tests 3 * Post-rebase refactor * rename nlp->datasets * RAG integration tests * add tokenizer to slow integration test and allow retriever to run on cpu * add tests; fix position ids warning * change structure * change structure * add from encoder generator * save working solution * make all integration tests pass * add RagTokenizer.save/from_pretrained and RagRetriever.save/from_pretrained * don't save paths * delete unnecessary imports * pass config to AutoTokenizer.from_pretrained for Rag tokenizers * init wiki_dpr only once * hardcode legacy index and passages paths (todo: add the right urls) * finalize config * finalize retriver api and config api * LegacyIndex index download refactor * add dpr to autotokenizer * make from pretrained more flexible * fix ragfortokengeneration * small name changes in tokenizer * add labels to models * change default index name * add retrieval tests * finish token generate * align test with previous version and make all tests pass * add tests * finalize tests * implement thoms suggestions * add first version of test * make first tests work * make retriever platform agnostic * naming * style * add legacy index URL * docstrings + simple retrieval test for distributed * clean model api * add doc_ids to retriever's outputs * fix retrieval tests * finish model outputs * finalize model api * fix generate problem for rag * fix generate for other modles * fix some tests * save intermediate * set generate to default * big refactor generate * delete rag_api * correct pip faiss install * fix auto tokenization test * fix faiss install * fix test * move the distributed logic to examples * model page * docs * finish tests * fix dependencies * fix import in __init__ * Refactor eval_rag and finetune scripts * start docstring * add psutil to test * fix tf test * move require torch to top * fix retrieval test * align naming * finish automodel * fix repo consistency * test ragtokenizer save/load * add rag model output docs * fix ragtokenizer save/load from pretrained * fix tokenizer dir * remove torch in retrieval * fix docs * fixe finetune scripts * finish model docs * finish docs * remove auto model for now * add require torch * remove solved todos * integrate sylvains suggestions * sams comments * correct mistake on purpose * improve README * Add generation test cases * fix rag token * clean token generate * fix test * add note to test * fix attention mask * add t5 test for rag * Fix handling prefix in finetune.py * don't overwrite index_name Co-authored-by: Patrick Lewis <plewis@fb.com> Co-authored-by: Aleksandra Piktus <piktus@devfair0141.h2.fair> Co-authored-by: Aleksandra Piktus <piktus@learnfair5102.h2.fair> Co-authored-by: Aleksandra Piktus <piktus@learnfair5067.h2.fair> Co-authored-by: Your Name <you@example.com> Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com> Co-authored-by: Quentin Lhoest <lhoest.q@gmail.com>
89 lines
3.3 KiB
ReStructuredText
89 lines
3.3 KiB
ReStructuredText
RAG
|
|
----------------------------------------------------
|
|
|
|
Overview
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models.
|
|
RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs.
|
|
The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt to downstream tasks.
|
|
|
|
It is based on the paper `Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks <https://arxiv.org/abs/2005.11401>`__ by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Large pre-trained language models have been shown to store factual knowledge
|
|
in their parameters, and achieve state-of-the-art results when fine-tuned on
|
|
downstream NLP tasks. However, their ability to access and precisely manipulate
|
|
knowledge is still limited, and hence on knowledge-intensive tasks, their
|
|
performance lags behind task-specific architectures. Additionally, providing
|
|
provenance for their decisions and updating their world knowledge remain open
|
|
research problems. Pre-trained models with a differentiable access mechanism to
|
|
explicit nonparametric memory can overcome this issue, but have so far been only
|
|
investigated for extractive downstream tasks. We explore a general-purpose
|
|
fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine
|
|
pre-trained parametric and non-parametric memory for language generation. We
|
|
introduce RAG models where the parametric memory is a pre-trained seq2seq model and
|
|
the non-parametric memory is a dense vector index of Wikipedia, accessed with
|
|
a pre-trained neural retriever. We compare two RAG formulations, one which
|
|
conditions on the same retrieved passages across the whole generated sequence, the
|
|
other can use different passages per token. We fine-tune and evaluate our models
|
|
on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art
|
|
on three open domain QA tasks, outperforming parametric seq2seq models and
|
|
task-specific retrieve-and-extract architectures. For language generation tasks, we
|
|
find that RAG models generate more specific, diverse and factual language than a
|
|
state-of-the-art parametric-only seq2seq baseline.*
|
|
|
|
|
|
|
|
RagConfig
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagConfig
|
|
:members:
|
|
|
|
|
|
RagTokenizer
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagTokenizer
|
|
:members:
|
|
|
|
|
|
Rag specific outputs
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.modeling_rag.RetrievAugLMMarginOutput
|
|
:members:
|
|
|
|
.. autoclass:: transformers.modeling_rag.RetrievAugLMOutput
|
|
:members:
|
|
|
|
|
|
RAGRetriever
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagRetriever
|
|
:members:
|
|
|
|
|
|
RagModel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagModel
|
|
:members: forward
|
|
|
|
|
|
RagSequenceForGeneration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagSequenceForGeneration
|
|
:members: forward, generate
|
|
|
|
|
|
RagTokenForGeneration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagTokenForGeneration
|
|
:members: forward, generate
|