riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-08-02 19:21:31 +06:00
Code Issues Actions 2 Packages Projects Releases Wiki Activity

Docs: fix indentation in HammingDiversityLogitsProcessor (#25756)

Browse Source
This commit is contained in:
Joao Gante 2023-08-25 14:56:39 +01:00 committed by GitHub
parent 35c570c80e
commit 8b0a7bfcdc
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 80 additions and 129 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

139
src/transformers/generation/logits_process.py
View File

@ -1074,9 +1074,9 @@ class HammingDiversityLogitsProcessor(LogitsProcessor):
<Tip>
Diverse beam search can be particularly useful in scenarios where a variety of different outputs is desired,
rather than multiple similar sequences. It allows the model to explore different generation paths and provides
a broader coverage of possible outputs.
Diverse beam search can be particularly useful in scenarios where a variety of different outputs is desired, rather
than multiple similar sequences. It allows the model to explore different generation paths and provides a broader
coverage of possible outputs.
</Tip>
@ -1087,9 +1087,8 @@ class HammingDiversityLogitsProcessor(LogitsProcessor):
</Tip>
Traditional beam search often generates very similar sequences across different beams.
The `HammingDiversityLogitsProcessor` addresses this by penalizing beams that generate tokens already chosen by
other beams in the same time step.
`HammingDiversityLogitsProcessor` addresses this by penalizing beams that generate tokens already chosen by other
beams in the same time step.
How It Works:
- **Grouping Beams**: Beams are divided into groups. Each group selects tokens independently of the others.
@ -1104,48 +1103,30 @@ class HammingDiversityLogitsProcessor(LogitsProcessor):
Args:
diversity_penalty (`float`):
This value is subtracted from a beam's score if it generates a token same as any beam from other group
at a particular time. Note that `diversity_penalty` is only effective if `group beam search` is
enabled.
- The penalty applied to a beam's score when it generates a token that has already been chosen
by another beam within the same group during the same time step.
- A higher `diversity_penalty` will enforce greater diversity among the beams,
making it less likely for multiple beams to choose the same token.
- Conversely, a lower penalty will allow beams to more freely choose similar tokens. --
Adjusting
this value can help strike a balance between diversity and natural likelihood.
This value is subtracted from a beam's score if it generates a token same as any beam from other group at a
particular time. Note that `diversity_penalty` is only effective if group beam search is enabled. The
penalty applied to a beam's score when it generates a token that has already been chosen by another beam
within the same group during the same time step. A higher `diversity_penalty` will enforce greater
diversity among the beams, making it less likely for multiple beams to choose the same token. Conversely, a
lower penalty will allow beams to more freely choose similar tokens. Adjusting this value can help strike a
balance between diversity and natural likelihood.
num_beams (`int`):
Number of beams used for group beam search. See [this paper](https://arxiv.org/pdf/1610.02424.pdf) for
more details.
- Beam search is a method used that maintains beams (or "multiple hypotheses") at each step,
expanding each one and keeping the top-scoring sequences.
- A higher `num_beams` will explore more potential sequences
This can increase chances of finding a high-quality output but also increases computational
cost.
Number of beams used for group beam search. Beam search is a method used that maintains beams (or "multiple
hypotheses") at each step, expanding each one and keeping the top-scoring sequences. A higher `num_beams`
will explore more potential sequences. This can increase chances of finding a high-quality output but also
increases computational cost.
num_beam_groups (`int`):
Number of groups to divide `num_beams` into in order to ensure diversity among different groups of
beams. See [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details.
- Each group of beams will operate independently, selecting tokens without considering the
choices of other groups.
- This division promotes diversity by ensuring that beams within different groups explore
different paths.
- For instance, if `num_beams` is 6 and `num_beam_groups` is 2, there will be 2 groups each
containing 3 beams.
- The choice of `num_beam_groups` should be made considering the desired level of output
diversity and the total number of beams.
Number of groups to divide `num_beams` into in order to ensure diversity among different groups of beams.
Each group of beams will operate independently, selecting tokens without considering the choices of other
groups. This division promotes diversity by ensuring that beams within different groups explore different
paths. For instance, if `num_beams` is 6 and `num_beam_groups` is 2, there will be 2 groups each containing
3 beams. The choice of `num_beam_groups` should be made considering the desired level of output diversity
and the total number of beams. See [this paper](https://arxiv.org/pdf/1610.02424.pdf) for more details.
Example: the below example shows a comparison before and after applying Hamming Diversity.
Examples:
```python
>>> from transformers import (
... AutoTokenizer,
... AutoModelForSeq2SeqLM,
... LogitsProcessorList,
... MinLengthLogitsProcessor,
... HammingDiversityLogitsProcessor,
... BeamSearchScorer,
... )
>>> from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
>>> import torch
>>> # Initialize the model and tokenizer
@ -1154,67 +1135,37 @@ class HammingDiversityLogitsProcessor(LogitsProcessor):
>>> # A long text about the solar system
>>> text = "The Solar System is a gravitationally bound system comprising the Sun and the objects that orbit it, either directly or indirectly. Of the objects that orbit the Sun directly, the largest are the eight planets, with the remainder being smaller objects, such as the five dwarf planets and small Solar System bodies. The Solar System formed 4.6 billion years ago from the gravitational collapse of a giant interstellar molecular cloud."
>>> inputs = tokenizer("summarize: " + text, return_tensors="pt")
>>> encoder_input_str = "summarize: " + text
>>> encoder_input_ids = tokenizer(encoder_input_str, return_tensors="pt").input_ids
>>> # Set up for diverse beam search
>>> num_beams = 6
>>> num_beam_groups = 2
>>> model_kwargs = {
... "encoder_outputs": model.get_encoder()(
... encoder_input_ids.repeat_interleave(num_beams, dim=0), return_dict=True
... )
... }
>>> beam_scorer = BeamSearchScorer(
... batch_size=1,
... max_length=model.config.max_length,
... num_beams=num_beams,
... device=model.device,
... num_beam_groups=num_beam_groups,
... )
>>> # Initialize the diversity logits processor
>>> # set the logits processor list, note that `HammingDiversityLogitsProcessor` is effective only if `group beam search` is enabled
>>> logits_processor_diverse = LogitsProcessorList(
... [
... HammingDiversityLogitsProcessor(5.5, num_beams=num_beams, num_beam_groups=num_beam_groups),
... MinLengthLogitsProcessor(10, eos_token_id=model.config.eos_token_id),
... ]
... )
>>> # generate the diverse summary using group_beam_search
>>> outputs_diverse = model.group_beam_search(
... encoder_input_ids.repeat_interleave(num_beams, dim=0),
... beam_scorer,
... logits_processor=logits_processor_diverse,
... **model_kwargs,
>>> # Generate diverse summary
>>> outputs_diverse = model.generate(
... **inputs,
... num_beam_groups=2,
... diversity_penalty=10.0,
... max_length=100,
... num_beams=4,
... num_return_sequences=2,
... )
>>> summaries_diverse = tokenizer.batch_decode(outputs_diverse, skip_special_tokens=True)
>>> # Generate non-diverse summary
>>> outputs_non_diverse = model.generate(
... encoder_input_ids,
... **inputs,
... max_length=100,
... num_beams=num_beams,
... no_repeat_ngram_size=2,
... early_stopping=True,
... num_beams=4,
... num_return_sequences=2,
... )
>>> summary_non_diverse = tokenizer.batch_decode(outputs_non_diverse, skip_special_tokens=True)
>>> # Decode and print the summaries
>>> summaries_diverse = tokenizer.batch_decode(outputs_diverse, skip_special_tokens=True)
>>> summary_non_diverse = tokenizer.decode(outputs_non_diverse[0], skip_special_tokens=True)
>>> print("Diverse Summary:")
>>> print(summaries_diverse[0])
>>> # The Solar System is a gravitationally bound system comprising the Sun and the objects that orbit it, either directly or indirectly. Of the objects that orbit the Sun directly, the largest are the eight planets, with the remainder being smaller objects, such as the five dwarf planets and small Solar System bodies. The Solar System formed 4.6 billion years ago from the gravitational collapse of a giant interstellar molecular cloud.
>>> print("\nNon-Diverse Summary:")
>>> # With `diversity_penalty`, the resulting beams are much more diverse
>>> print(summary_non_diverse)
>>> # The Sun and the objects that orbit it directly are the eight planets, with the remainder being smaller objects, such as the five dwarf worlds and small Solar System bodies. It formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. ```
['the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.',
'the Solar System formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.']
>>> print(summaries_diverse)
['the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets.',
'the solar system formed 4.6 billion years ago from the collapse of a giant interstellar molecular cloud. of the objects that orbit the Sun directly, the largest are the eight planets. the rest of the objects are smaller objects, such as the five dwarf planets and small solar system bodies.']
```
For more details, see [Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence
Models](https://arxiv.org/pdf/1610.02424.pdf).
"""
def __init__(self, diversity_penalty: float, num_beams: int, num_beam_groups: int):