From 43b7ad5df551faa091c6b52a9a8bf2c429741e3d Mon Sep 17 00:00:00 2001 From: Vitalii Radchenko Date: Tue, 7 Jul 2020 13:40:23 +0300 Subject: [PATCH] ukr-roberta-base model card (#5514) --- .../youscan/ukr-roberta-base/README.md | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 model_cards/youscan/ukr-roberta-base/README.md diff --git a/model_cards/youscan/ukr-roberta-base/README.md b/model_cards/youscan/ukr-roberta-base/README.md new file mode 100644 index 00000000000..6cc6a5c2dac --- /dev/null +++ b/model_cards/youscan/ukr-roberta-base/README.md @@ -0,0 +1,26 @@ +--- +language: +- ukrainian +--- + +# ukr-roberta-base + +## Pre-training corpora +Below is the list of corpora used along with the output of wc command (counting lines, words and characters). These corpora were concatenated and tokenized with HuggingFace Roberta Tokenizer. + +| Tables | Lines | Words | Characters | +| ------------- |--------------:| -----:| -----:| +| [Ukrainian Wikipedia - May 2020](https://dumps.wikimedia.org/ukwiki/latest/ukwiki-latest-pages-articles.xml.bz2) | 18 001 466| 201 207 739 | 2 647 891 947 | +| [Ukrainian OSCAR deduplicated dataset](https://oscar-public.huma-num.fr/shuffled/uk_dedup.txt.gz) | 56 560 011 | 2 250 210 650 | 29 705 050 592 | +| Sampled mentions from social networks | 11 245 710 | 128 461 796 | 1 632 567 763 | +| Total | 85 807 187 | 2 579 880 185 | 33 985 510 302 | + +## Pre-training details + +* Ukrainian Roberta was trained with code provided in [HuggingFace tutorial](https://huggingface.co/blog/how-to-train) +* Currently released model follows roberta-base-cased model architecture (12-layer, 768-hidden, 12-heads, 125M parameters) +* The model was trained on 4xV100 (85 hours) +* Training configuration you can find in the [original repository](https://github.com/youscan/language-models) + +## Author +Vitalii Radchenko - contact me on Twitter [@vitaliradchenko](https://twitter.com/vitaliradchenko)