mirror of
https://github.com/huggingface/transformers.git
synced 2025-07-03 21:00:08 +06:00
762cba3bda
* add dataset for albert pretrain * datacollator for albert pretrain * naming, comprehension, file reading change * data cleaning is no needed after this modification * delete prints * fix a bug * file structure change * add tests for albert datacollator * remove random seed * add back len and get item function * sample file for testing and test code added * format change for black * more format change * Style * var assignment issue resolve * add back wrongly deleted DataCollatorWithPadding in init file * Style Co-authored-by: Lysandre Debut <lysandre@huggingface.co> Co-authored-by: Lysandre <lysandre.debut@reseau.eseo.fr> |
||
|---|---|---|
| .. | ||
| tests_samples | ||
| dummy-config.json | ||
| empty.txt | ||
| input.txt | ||
| sample_text.txt | ||
| spiece.model | ||
| test_sentencepiece.model | ||