## Token classification Fine-tuning the library models for token classification task such as Named Entity Recognition (NER) or Parts-of-speech tagging (POS). The main scrip `run_ner.py` leverages the 🤗 Datasets library and the Trainer API. You can easily customize it to your needs if you need extra processing on your datasets. It will either run on a datasets hosted on our [hub](https://huggingface.co/datasets) or with your own text files for training and validation. The following example fine-tunes BERT on CoNLL-2003: ```bash python run_ner.py \ --model_name_or_path bert-base-uncased \ --dataset_name conll2003 \ --output_dir /tmp/test-ner \ --do_train \ --do_eval ``` or just can just run the bash script `run.sh`. To run on your own training and validation files, use the following command: ```bash python run_ner.py \ --model_name_or_path bert-base-uncased \ --train_file path_to_train_file \ --validation_file path_to_validation_file \ --output_dir /tmp/test-ner \ --do_train \ --do_eval ``` **Note:** This script only works with models that have a fast tokenizer (backed by the 🤗 Tokenizers library) as it uses special features of those tokenizers. You can check if your favorite model has a fast tokenizer in [this table](https://huggingface.co/transformers/index.html#bigtable), if it doesn't you can still use the old version of the script. ## Old version of the script You can find the old version of the PyTorch script [here](https://github.com/huggingface/transformers/blob/master/examples/legacy/token-classification/run_ner.py). ### TensorFlow version The following examples are covered in this section: * NER on the GermEval 2014 (German NER) dataset * Emerging and Rare Entities task: WNUT’17 (English NER) dataset Details and results for the fine-tuning provided by @stefan-it. ### GermEval 2014 (German NER) dataset #### Data (Download and pre-processing steps) Data can be obtained from the [GermEval 2014](https://sites.google.com/site/germeval2014ner/data) shared task page. Here are the commands for downloading and pre-processing train, dev and test datasets. The original data format has four (tab-separated) columns, in a pre-processing step only the two relevant columns (token and outer span NER annotation) are extracted: ```bash curl -L 'https://drive.google.com/uc?export=download&id=1Jjhbal535VVz2ap4v4r_rN1UEHTdLK5P' \ | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > train.txt.tmp curl -L 'https://drive.google.com/uc?export=download&id=1ZfRcQThdtAR5PPRjIDtrVP7BtXSCUBbm' \ | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > dev.txt.tmp curl -L 'https://drive.google.com/uc?export=download&id=1u9mb7kNJHWQCWyweMDRMuTFoOHOfeBTH' \ | grep -v "^#" | cut -f 2,3 | tr '\t' ' ' > test.txt.tmp ``` The GermEval 2014 dataset contains some strange "control character" tokens like `'\x96', '\u200e', '\x95', '\xad' or '\x80'`. One problem with these tokens is, that `BertTokenizer` returns an empty token for them, resulting in misaligned `InputExample`s. The `preprocess.py` script located in the `scripts` folder a) filters these tokens and b) splits longer sentences into smaller ones (once the max. subtoken length is reached). Let's define some variables that we need for further pre-processing steps and training the model: ```bash export MAX_LENGTH=128 export BERT_MODEL=bert-base-multilingual-cased ``` Run the pre-processing script on training, dev and test datasets: ```bash python3 scripts/preprocess.py train.txt.tmp $BERT_MODEL $MAX_LENGTH > train.txt python3 scripts/preprocess.py dev.txt.tmp $BERT_MODEL $MAX_LENGTH > dev.txt python3 scripts/preprocess.py test.txt.tmp $BERT_MODEL $MAX_LENGTH > test.txt ``` The GermEval 2014 dataset has much more labels than CoNLL-2002/2003 datasets, so an own set of labels must be used: ```bash cat train.txt dev.txt test.txt | cut -d " " -f 2 | grep -v "^$"| sort | uniq > labels.txt ``` #### Prepare the run Additional environment variables must be set: ```bash export OUTPUT_DIR=germeval-model export BATCH_SIZE=32 export NUM_EPOCHS=3 export SAVE_STEPS=750 export SEED=1 ``` #### Run the Tensorflow 2 version To start training, just run: ```bash python3 run_tf_ner.py --data_dir ./ \ --labels ./labels.txt \ --model_name_or_path $BERT_MODEL \ --output_dir $OUTPUT_DIR \ --max_seq_length $MAX_LENGTH \ --num_train_epochs $NUM_EPOCHS \ --per_device_train_batch_size $BATCH_SIZE \ --save_steps $SAVE_STEPS \ --seed $SEED \ --do_train \ --do_eval \ --do_predict ``` Such as the Pytorch version, if your GPU supports half-precision training, just add the `--fp16` flag. After training, the model will be both evaluated on development and test datasets. #### Evaluation Evaluation on development dataset outputs the following for our example: ```bash precision recall f1-score support LOCderiv 0.7619 0.6154 0.6809 52 PERpart 0.8724 0.8997 0.8858 4057 OTHpart 0.9360 0.9466 0.9413 711 ORGpart 0.7015 0.6989 0.7002 269 LOCpart 0.7668 0.8488 0.8057 496 LOC 0.8745 0.9191 0.8963 235 ORGderiv 0.7723 0.8571 0.8125 91 OTHderiv 0.4800 0.6667 0.5581 18 OTH 0.5789 0.6875 0.6286 16 PERderiv 0.5385 0.3889 0.4516 18 PER 0.5000 0.5000 0.5000 2 ORG 0.0000 0.0000 0.0000 3 micro avg 0.8574 0.8862 0.8715 5968 macro avg 0.8575 0.8862 0.8713 5968 ``` On the test dataset the following results could be achieved: ```bash precision recall f1-score support PERpart 0.8847 0.8944 0.8896 9397 OTHpart 0.9376 0.9353 0.9365 1639 ORGpart 0.7307 0.7044 0.7173 697 LOC 0.9133 0.9394 0.9262 561 LOCpart 0.8058 0.8157 0.8107 1150 ORG 0.0000 0.0000 0.0000 8 OTHderiv 0.5882 0.4762 0.5263 42 PERderiv 0.6571 0.5227 0.5823 44 OTH 0.4906 0.6667 0.5652 39 ORGderiv 0.7016 0.7791 0.7383 172 LOCderiv 0.8256 0.6514 0.7282 109 PER 0.0000 0.0000 0.0000 11 micro avg 0.8722 0.8774 0.8748 13869 macro avg 0.8712 0.8774 0.8740 13869 ```