Add tokenizer docs (#13373)

This commit is contained in:
NielsRogge 2021-09-02 09:46:05 +02:00 committed by GitHub
parent 872e6be03d
commit 4114c9a75b
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23

View File

@ -40,6 +40,15 @@ One can directly plug in the weights of LayoutXLM into a LayoutLMv2 model, like
model = LayoutLMv2Model.from_pretrained('microsoft/layoutxlm-base')
Note that LayoutXLM requires a different tokenizer, based on :class:`~transformers.XLMRobertaTokenizer`. You can
initialize it as follows:
.. code-block::
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained('microsoft/layoutxlm-base')
As LayoutXLM's architecture is equivalent to that of LayoutLMv2, one can refer to :doc:`LayoutLMv2's documentation page
<layoutlmv2>` for all tips, code examples and notebooks.