transformers/docs/source/en/model_doc/roc_bert.md

RoCBert

Overview

The RoCBert model was proposed in RoCBert: Robust Chinese Bert with Multimodal Contrastive Pretraining by HuiSu, WeiweiShi, XiaoyuShen, XiaoZhou, TuoJi, JiaruiFang, JieZhou. It's a pretrained Chinese language model that is robust under various forms of adversarial attacks.

The abstract from the paper is the following:

Large-scale pretrained language models have achieved SOTA results on NLP tasks. However, they have been shown vulnerable to adversarial attacks especially for logographic languages like Chinese. In this work, we propose ROCBERT: a pretrained Chinese Bert that is robust to various forms of adversarial attacks like word perturbation, synonyms, typos, etc. It is pretrained with the contrastive learning objective which maximizes the label consistency under different synthesized adversarial examples. The model takes as input multimodal information including the semantic, phonetic and visual features. We show all these features are important to the model robustness since the attack can be performed in all the three forms. Across 5 Chinese NLU tasks, ROCBERT outperforms strong baselines under three blackbox adversarial algorithms without sacrificing the performance on clean testset. It also performs the best in the toxic content detection task under human-made attacks.

This model was contributed by weiweishi.

Resources

• Text classification task guide
• Token classification task guide
• Question answering task guide
• Causal language modeling task guide
• Masked language modeling task guide
• Multiple choice task guide

RoCBertConfig

autodoc RoCBertConfig - all

RoCBertTokenizer

autodoc RoCBertTokenizer - build_inputs_with_special_tokens - get_special_tokens_mask - create_token_type_ids_from_sequences - save_vocabulary

RoCBertModel

autodoc RoCBertModel - forward

RoCBertForPreTraining

autodoc RoCBertForPreTraining - forward

RoCBertForCausalLM

autodoc RoCBertForCausalLM - forward

RoCBertForMaskedLM

autodoc RoCBertForMaskedLM - forward

RoCBertForSequenceClassification

autodoc transformers.RoCBertForSequenceClassification - forward

RoCBertForMultipleChoice

autodoc transformers.RoCBertForMultipleChoice - forward

RoCBertForTokenClassification

autodoc transformers.RoCBertForTokenClassification - forward

RoCBertForQuestionAnswering

autodoc RoCBertForQuestionAnswering - forward