mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-07 14:50:07 +06:00
187 lines
6.8 KiB
Markdown
187 lines
6.8 KiB
Markdown
<!---
|
||
Copyright 2020 The HuggingFace Team. All rights reserved.
|
||
|
||
Licensed under the Apache License, Version 2.0 (the "License");
|
||
you may not use this file except in compliance with the License.
|
||
You may obtain a copy of the License at
|
||
|
||
http://www.apache.org/licenses/LICENSE-2.0
|
||
|
||
Unless required by applicable law or agreed to in writing, software
|
||
distributed under the License is distributed on an "AS IS" BASIS,
|
||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||
See the License for the specific language governing permissions and
|
||
limitations under the License.
|
||
-->
|
||
|
||
## Token classification
|
||
|
||
Fine-tuning the library models for token classification task such as Named Entity Recognition (NER) or Parts-of-speech
|
||
tagging (POS). The main scrip `run_ner.py` leverages the 🤗 Datasets library and the Trainer API. You can easily
|
||
customize it to your needs if you need extra processing on your datasets.
|
||
|
||
It will either run on a datasets hosted on our [hub](https://huggingface.co/datasets) or with your own text files for
|
||
training and validation.
|
||
|
||
The following example fine-tunes BERT on CoNLL-2003:
|
||
|
||
```bash
|
||
python run_ner.py \
|
||
--model_name_or_path bert-base-uncased \
|
||
--dataset_name conll2003 \
|
||
--output_dir /tmp/test-ner \
|
||
--do_train \
|
||
--do_eval
|
||
```
|
||
|
||
or just can just run the bash script `run.sh`.
|
||
|
||
To run on your own training and validation files, use the following command:
|
||
|
||
```bash
|
||
python run_ner.py \
|
||
--model_name_or_path bert-base-uncased \
|
||
--train_file path_to_train_file \
|
||
--validation_file path_to_validation_file \
|
||
--output_dir /tmp/test-ner \
|
||
--do_train \
|
||
--do_eval
|
||
```
|
||
|
||
**Note:** This script only works with models that have a fast tokenizer (backed by the 🤗 Tokenizers library) as it
|
||
uses special features of those tokenizers. You can check if your favorite model has a fast tokenizer in
|
||
[this table](https://huggingface.co/transformers/index.html#bigtable), if it doesn't you can still use the old version
|
||
of the script.
|
||
|
||
## Old version of the script
|
||
|
||
You can find the old version of the PyTorch script [here](https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py).
|
||
|
||
### TensorFlow version
|
||
|
||
The following examples are covered in this section:
|
||
|
||
* NER on the GermEval 2014 (German NER) dataset
|
||
* Emerging and Rare Entities task: WNUT’17 (English NER) dataset
|
||
|
||
Details and results for the fine-tuning provided by @stefan-it.
|
||
|
||
### GermEval 2014 (German NER) dataset
|
||
|
||
#### Data (Download and pre-processing steps)
|
||
|
||
Data can be obtained from the [GermEval 2014](https://sites.google.com/site/germeval2014ner/data) shared task page.
|
||
|
||
Here are the commands for downloading and pre-processing train, dev and test datasets. The original data format has four (tab-separated) columns, in a pre-processing step only the two relevant columns (token and outer span NER annotation) are extracted:
|
||
|
||
```bash
|
||
curl -L 'https://drive.google.com/uc?export=download&id=1Jjhbal535VVz2ap4v4r_rN1UEHTdLK5P' \
|
||
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp
|
||
curl -L 'https://drive.google.com/uc?export=download&id=1ZfRcQThdtAR5PPRjIDtrVP7BtXSCUBbm' \
|
||
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp
|
||
curl -L 'https://drive.google.com/uc?export=download&id=1u9mb7kNJHWQCWyweMDRMuTFoOHOfeBTH' \
|
||
| grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp
|
||
```
|
||
|
||
The GermEval 2014 dataset contains some strange "control character" tokens like `'\x96', '\u200e', '\x95', '\xad' or '\x80'`.
|
||
One problem with these tokens is, that `BertTokenizer` returns an empty token for them, resulting in misaligned `InputExample`s.
|
||
The `preprocess.py` script located in the `scripts` folder a) filters these tokens and b) splits longer sentences into smaller ones (once the max. subtoken length is reached).
|
||
|
||
Let's define some variables that we need for further pre-processing steps and training the model:
|
||
|
||
```bash
|
||
export MAX_LENGTH=128
|
||
export BERT_MODEL=bert-base-multilingual-cased
|
||
```
|
||
|
||
Run the pre-processing script on training, dev and test datasets:
|
||
|
||
```bash
|
||
python3 scripts/preprocess.py train.txt.tmp $BERT_MODEL $MAX_LENGTH > train.txt
|
||
python3 scripts/preprocess.py dev.txt.tmp $BERT_MODEL $MAX_LENGTH > dev.txt
|
||
python3 scripts/preprocess.py test.txt.tmp $BERT_MODEL $MAX_LENGTH > test.txt
|
||
```
|
||
|
||
The GermEval 2014 dataset has much more labels than CoNLL-2002/2003 datasets, so an own set of labels must be used:
|
||
|
||
```bash
|
||
cat train.txt dev.txt test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > labels.txt
|
||
```
|
||
|
||
#### Prepare the run
|
||
|
||
Additional environment variables must be set:
|
||
|
||
```bash
|
||
export OUTPUT_DIR=germeval-model
|
||
export BATCH_SIZE=32
|
||
export NUM_EPOCHS=3
|
||
export SAVE_STEPS=750
|
||
export SEED=1
|
||
```
|
||
|
||
#### Run the Tensorflow 2 version
|
||
|
||
To start training, just run:
|
||
|
||
```bash
|
||
python3 run_tf_ner.py --data_dir ./ \
|
||
--labels ./labels.txt \
|
||
--model_name_or_path $BERT_MODEL \
|
||
--output_dir $OUTPUT_DIR \
|
||
--max_seq_length $MAX_LENGTH \
|
||
--num_train_epochs $NUM_EPOCHS \
|
||
--per_device_train_batch_size $BATCH_SIZE \
|
||
--save_steps $SAVE_STEPS \
|
||
--seed $SEED \
|
||
--do_train \
|
||
--do_eval \
|
||
--do_predict
|
||
```
|
||
|
||
Such as the Pytorch version, if your GPU supports half-precision training, just add the `--fp16` flag. After training, the model will be both evaluated on development and test datasets.
|
||
|
||
#### Evaluation
|
||
|
||
Evaluation on development dataset outputs the following for our example:
|
||
```bash
|
||
precision recall f1-score support
|
||
|
||
LOCderiv 0.7619 0.6154 0.6809 52
|
||
PERpart 0.8724 0.8997 0.8858 4057
|
||
OTHpart 0.9360 0.9466 0.9413 711
|
||
ORGpart 0.7015 0.6989 0.7002 269
|
||
LOCpart 0.7668 0.8488 0.8057 496
|
||
LOC 0.8745 0.9191 0.8963 235
|
||
ORGderiv 0.7723 0.8571 0.8125 91
|
||
OTHderiv 0.4800 0.6667 0.5581 18
|
||
OTH 0.5789 0.6875 0.6286 16
|
||
PERderiv 0.5385 0.3889 0.4516 18
|
||
PER 0.5000 0.5000 0.5000 2
|
||
ORG 0.0000 0.0000 0.0000 3
|
||
|
||
micro avg 0.8574 0.8862 0.8715 5968
|
||
macro avg 0.8575 0.8862 0.8713 5968
|
||
```
|
||
|
||
On the test dataset the following results could be achieved:
|
||
```bash
|
||
precision recall f1-score support
|
||
|
||
PERpart 0.8847 0.8944 0.8896 9397
|
||
OTHpart 0.9376 0.9353 0.9365 1639
|
||
ORGpart 0.7307 0.7044 0.7173 697
|
||
LOC 0.9133 0.9394 0.9262 561
|
||
LOCpart 0.8058 0.8157 0.8107 1150
|
||
ORG 0.0000 0.0000 0.0000 8
|
||
OTHderiv 0.5882 0.4762 0.5263 42
|
||
PERderiv 0.6571 0.5227 0.5823 44
|
||
OTH 0.4906 0.6667 0.5652 39
|
||
ORGderiv 0.7016 0.7791 0.7383 172
|
||
LOCderiv 0.8256 0.6514 0.7282 109
|
||
PER 0.0000 0.0000 0.0000 11
|
||
|
||
micro avg 0.8722 0.8774 0.8748 13869
|
||
macro avg 0.8712 0.8774 0.8740 13869
|
||
```
|