11 Commits

Author	SHA1	Message	Date
Teven	2a7e8e1608	[Examples] Add automatic dataset splitting in language-modeling examples (#9133) ... * replaced jnp.split + removing textual model inputs + ensuring warmup_steps > 0 * Add automatic dataset splitting in language-modeling examples	2020-12-15 16:02:43 -05:00
Sylvain Gugger	62d30e0583	Small fix to the run clm script (#8973)	2020-12-07 17:32:09 -05:00
Sylvain Gugger	a0c62d2493	Fix training from scratch in new scripts (#8623)	2020-11-18 12:15:26 -05:00
Julien Chaumond	042a6aa777	Tokenizers: ability to load from model subfolder (#8586) ... * <small>tiny typo</small> * Tokenizers: ability to load from model subfolder * use subfolder for local files as well * Uniformize model shortcut name => model id * from s3 => from huggingface.co Co-authored-by: Quentin Lhoest <lhoest.q@gmail.com>	2020-11-17 08:58:45 -05:00
Julien Plu	27b3ff316a	Try to understand and apply Sylvain's comments (#8458)	2020-11-12 13:43:00 -05:00
zeyuyun1	924c624a46	quick fix on concatenating text to support more datasets (#8474)	2020-11-12 09:47:08 -05:00
Sylvain Gugger	cf89724696	Fix validation file loading in scripts (#8298)	2020-11-04 10:42:18 -05:00
Sylvain Gugger	9eb3a410cd	Remove deprecated arguments from new run_clm (#8197)	2020-10-30 15:27:20 -04:00
Sylvain Gugger	cdc48ce92d	Finalize lm examples (#8188) ... * Finish the cleanup of the language-modeling examples * Update main README * Apply suggestions from code review Co-authored-by: Lysandre Debut <lysandre@huggingface.co> * Apply suggestions from code review Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com> * Propagate changes Co-authored-by: Lysandre Debut <lysandre@huggingface.co> Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>	2020-10-30 14:20:18 -04:00
Sylvain Gugger	691176283d	Add a template for examples and apply it for mlm and plm examples (#8153) ... * Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Add a template for example scripts and apply it to mlm * Formatting * Fix test * Add plm script * Styling	2020-10-29 13:38:11 -04:00
Sylvain Gugger	47dfa65b0c	New run_clm script (#8105) ... * New run_clm script * Formatting * More comments * Remove unused imports * Apply suggestions from code review Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com> * Address review comments * Change link to the hub Co-authored-by: Thomas Wolf <thomwolf@users.noreply.github.com>	2020-10-28 10:38:58 -04:00