29ca043856
* Created model card for XLM model * Revised model card structure and content of XLM model * Update XLM model documentation with improved examples and code snippets for predicting <mask> tokens using Pipeline and AutoModel.
4.5 KiB
XLM
XLM demonstrates cross-lingual pretraining with two approaches, unsupervised training on a single language and supervised training on more than one language with a cross-lingual language model objective. The XLM model supports the causal language modeling objective, masked language modeling, and translation language modeling (an extension of the BERT) masked language modeling objective to multiple language inputs).
You can find all the original XLM checkpoints under the Facebook AI community organization.
Tip
Click on the XLM models in the right sidebar for more examples of how to apply XLM to different cross-lingual tasks like classification, translation, and question answering.
The example below demonstrates how to predict the <mask> token with [Pipeline], [AutoModel] and from the command line.
import torch
from transformers import pipeline
pipeline = pipeline(
task="fill-mask",
model="facebook/xlm-roberta-xl",
torch_dtype=torch.float16,
device=0
)
pipeline("Bonjour, je suis un modèle <mask>.")
import torch
from transformers import AutoModelForMaskedLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(
"FacebookAI/xlm-mlm-en-2048",
)
model = AutoModelForMaskedLM.from_pretrained(
"FacebookAI/xlm-mlm-en-2048",
torch_dtype=torch.float16,
device_map="auto",
)
inputs = tokenizer("Hello, I'm a <mask> model.", return_tensors="pt").to("cuda")
with torch.no_grad():
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)
predicted_token = tokenizer.decode(predictions[0][inputs["input_ids"][0] == tokenizer.mask_token_id])
print(f"Predicted token: {predicted_token}")
echo -e "Plants create <mask> through a process known as photosynthesis." | transformers-cli run --task fill-mask --model FacebookAI/xlm-mlm-en-2048 --device 0
XLMConfig
autodoc XLMConfig
XLMTokenizer
autodoc XLMTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary
XLM specific outputs
autodoc models.xlm.modeling_xlm.XLMForQuestionAnsweringOutput
XLMModel
autodoc XLMModel - forward
XLMWithLMHeadModel
autodoc XLMWithLMHeadModel - forward
XLMForSequenceClassification
autodoc XLMForSequenceClassification - forward
XLMForMultipleChoice
autodoc XLMForMultipleChoice - forward
XLMForTokenClassification
autodoc XLMForTokenClassification - forward
XLMForQuestionAnsweringSimple
autodoc XLMForQuestionAnsweringSimple - forward
XLMForQuestionAnswering
autodoc XLMForQuestionAnswering - forward
TFXLMModel
autodoc TFXLMModel - call
TFXLMWithLMHeadModel
autodoc TFXLMWithLMHeadModel - call
TFXLMForSequenceClassification
autodoc TFXLMForSequenceClassification - call
TFXLMForMultipleChoice
autodoc TFXLMForMultipleChoice - call
TFXLMForTokenClassification
autodoc TFXLMForTokenClassification - call
TFXLMForQuestionAnsweringSimple
autodoc TFXLMForQuestionAnsweringSimple - call