mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-06 14:20:04 +06:00
033f29c625
* add CustomHFIndex * typo in config * update tests * add custom dataset example * clean script * update test data * minor in test * docs * docs * style * fix imports * allow to pass the indexed dataset directly * update tests * use multiset DPR * address thom and patrick's comments * style * update dpr tokenizer * add output_dir flag in use_own_knowledge_dataset.py * allow custom datasets in examples/rag/finetune.py * add test for custom dataset in distributed rag retriever
91 lines
3.3 KiB
ReStructuredText
91 lines
3.3 KiB
ReStructuredText
RAG
|
|
----------------------------------------------------
|
|
|
|
Overview
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and
|
|
sequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate
|
|
outputs. The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing
|
|
both retrieval and generation to adapt to downstream tasks.
|
|
|
|
It is based on the paper `Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
|
|
<https://arxiv.org/abs/2005.11401>`__ by Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir
|
|
Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Large pre-trained language models have been shown to store factual knowledge
|
|
in their parameters, and achieve state-of-the-art results when fine-tuned on
|
|
downstream NLP tasks. However, their ability to access and precisely manipulate
|
|
knowledge is still limited, and hence on knowledge-intensive tasks, their
|
|
performance lags behind task-specific architectures. Additionally, providing
|
|
provenance for their decisions and updating their world knowledge remain open
|
|
research problems. Pre-trained models with a differentiable access mechanism to
|
|
explicit nonparametric memory can overcome this issue, but have so far been only
|
|
investigated for extractive downstream tasks. We explore a general-purpose
|
|
fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine
|
|
pre-trained parametric and non-parametric memory for language generation. We
|
|
introduce RAG models where the parametric memory is a pre-trained seq2seq model and
|
|
the non-parametric memory is a dense vector index of Wikipedia, accessed with
|
|
a pre-trained neural retriever. We compare two RAG formulations, one which
|
|
conditions on the same retrieved passages across the whole generated sequence, the
|
|
other can use different passages per token. We fine-tune and evaluate our models
|
|
on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art
|
|
on three open domain QA tasks, outperforming parametric seq2seq models and
|
|
task-specific retrieve-and-extract architectures. For language generation tasks, we
|
|
find that RAG models generate more specific, diverse and factual language than a
|
|
state-of-the-art parametric-only seq2seq baseline.*
|
|
|
|
|
|
|
|
RagConfig
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagConfig
|
|
:members:
|
|
|
|
|
|
RagTokenizer
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagTokenizer
|
|
:members: prepare_seq2seq_batch
|
|
|
|
|
|
Rag specific outputs
|
|
~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.modeling_rag.RetrievAugLMMarginOutput
|
|
:members:
|
|
|
|
.. autoclass:: transformers.modeling_rag.RetrievAugLMOutput
|
|
:members:
|
|
|
|
RagRetriever
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagRetriever
|
|
:members:
|
|
|
|
|
|
RagModel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagModel
|
|
:members: forward
|
|
|
|
|
|
RagSequenceForGeneration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagSequenceForGeneration
|
|
:members: forward, generate
|
|
|
|
|
|
RagTokenForGeneration
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.RagTokenForGeneration
|
|
:members: forward, generate
|