mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-04 13:20:12 +06:00
115 lines
5.4 KiB
ReStructuredText
115 lines
5.4 KiB
ReStructuredText
..
|
|
Copyright 2020 The HuggingFace Team. All rights reserved.
|
|
|
|
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with
|
|
the License. You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on
|
|
an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the
|
|
specific language governing permissions and limitations under the License.
|
|
|
|
SqueezeBERT
|
|
-----------------------------------------------------------------------------------------------------------------------
|
|
|
|
Overview
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
The SqueezeBERT model was proposed in `SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
|
|
<https://arxiv.org/abs/2006.11316>`__ by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer. It's a
|
|
bidirectional transformer similar to the BERT model. The key difference between the BERT architecture and the
|
|
SqueezeBERT architecture is that SqueezeBERT uses `grouped convolutions <https://blog.yani.io/filter-group-tutorial>`__
|
|
instead of fully-connected layers for the Q, K, V and FFN layers.
|
|
|
|
The abstract from the paper is the following:
|
|
|
|
*Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets,
|
|
large computing systems, and better neural network models, natural language processing (NLP) technology has made
|
|
significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant
|
|
opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we
|
|
consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's
|
|
highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with
|
|
BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods
|
|
such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these
|
|
techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in
|
|
self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called
|
|
SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test
|
|
set. The SqueezeBERT code will be released.*
|
|
|
|
Tips:
|
|
|
|
- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
|
|
rather than the left.
|
|
- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
|
|
efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
|
|
with a causal language modeling (CLM) objective are better in that regard.
|
|
- For best results when finetuning on sequence classification tasks, it is recommended to start with the
|
|
`squeezebert/squeezebert-mnli-headless` checkpoint.
|
|
|
|
This model was contributed by `forresti <https://huggingface.co/forresti>`__.
|
|
|
|
|
|
SqueezeBertConfig
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertConfig
|
|
:members:
|
|
|
|
|
|
SqueezeBertTokenizer
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertTokenizer
|
|
:members: build_inputs_with_special_tokens, get_special_tokens_mask,
|
|
create_token_type_ids_from_sequences, save_vocabulary
|
|
|
|
|
|
SqueezeBertTokenizerFast
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertTokenizerFast
|
|
:members:
|
|
|
|
|
|
SqueezeBertModel
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertModel
|
|
:members:
|
|
|
|
|
|
SqueezeBertForMaskedLM
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertForMaskedLM
|
|
:members:
|
|
|
|
|
|
SqueezeBertForSequenceClassification
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertForSequenceClassification
|
|
:members:
|
|
|
|
|
|
SqueezeBertForMultipleChoice
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertForMultipleChoice
|
|
:members:
|
|
|
|
|
|
SqueezeBertForTokenClassification
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertForTokenClassification
|
|
:members:
|
|
|
|
|
|
SqueezeBertForQuestionAnswering
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
|
|
.. autoclass:: transformers.SqueezeBertForQuestionAnswering
|
|
:members:
|