riaz.somc/transformers
1
0
Fork 0
mirror of https://github.com/huggingface/transformers.git synced 2025-07-31 02:02:21 +06:00
Code Issues Actions 33 Packages Projects Releases Wiki Activity

ukr-roberta-base model card (#5514)

Browse Source
This commit is contained in:
Vitalii Radchenko 2020-07-07 13:40:23 +03:00 committed by GitHub
parent 87aa857d7e
commit 43b7ad5df5
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 26 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
model_cards/youscan/ukr-roberta-base/README.md Normal file
View File

@ -0,0 +1,26 @@
---
language:
- ukrainian
---
# ukr-roberta-base
## Pre-training corpora
Below is the list of corpora used along with the output of wc command (counting lines, words and characters). These corpora were concatenated and tokenized with HuggingFace Roberta Tokenizer.
| Tables | Lines | Words | Characters |
| ------------- |--------------:| -----:| -----:|
| [Ukrainian Wikipedia - May 2020](https://dumps.wikimedia.org/ukwiki/latest/ukwiki-latest-pages-articles.xml.bz2) | 18 001 466| 201 207 739 | 2 647 891 947 |
| [Ukrainian OSCAR deduplicated dataset](https://oscar-public.huma-num.fr/shuffled/uk_dedup.txt.gz) | 56 560 011 | 2 250 210 650 | 29 705 050 592 |
| Sampled mentions from social networks | 11 245 710 | 128 461 796 | 1 632 567 763 |
| Total | 85 807 187 | 2 579 880 185 | 33 985 510 302 |
## Pre-training details
* Ukrainian Roberta was trained with code provided in [HuggingFace tutorial](https://huggingface.co/blog/how-to-train)
* Currently released model follows roberta-base-cased model architecture (12-layer, 768-hidden, 12-heads, 125M parameters)
* The model was trained on 4xV100 (85 hours)
* Training configuration you can find in the [original repository](https://github.com/youscan/language-models)
## Author
Vitalii Radchenko - contact me on Twitter [@vitaliradchenko](https://twitter.com/vitaliradchenko)