9 Commits

Author	SHA1	Message	Date
Matthew Carrigan	2bba7f810e	Added a --reduce_memory option to shelve docs to disc instead of keeping them in memory.	2019-03-21 16:50:16 +00:00
Matthew Carrigan	8733ffcb5e	Removing a couple of other old unnecessary comments	2019-03-21 14:09:57 +00:00
Matthew Carrigan	8a861048dd	Fixed up the notes on a possible future low-memory path	2019-03-21 14:08:39 +00:00
Matthew Carrigan	0ae59e662d	Reduced memory usage for pregenerating the data a lot by writing it ... out on the fly without shuffling - the Sampler in the finetuning script will shuffle for us.	2019-03-21 14:04:17 +00:00
Matthew Carrigan	6a9038ba53	Removed an old irrelevant comment	2019-03-21 13:36:41 +00:00
Matthew Carrigan	934d3f4d2f	Syncing up argument names between the scripts	2019-03-20 17:23:23 +00:00
Matthew Carrigan	7de5c6aa5e	PEP8 and formatting cleanups	2019-03-20 16:44:04 +00:00
Matthew Carrigan	1798e98e5a	Added final TODOs	2019-03-20 16:42:37 +00:00
Matthew Carrigan	976554a472	First commit of the new LM finetuning	2019-03-20 14:23:51 +00:00