01c37dcdb5
* remove output_past from pt * make style * add optional input length for gpt2 * add use cache to prepare input * save memory in gpt2 * correct gpt2 test inputs * make past input optional for gpt2 * finish use_cache for all models * make style * delete modeling_gpt2 change in test file * correct docstring * correct is true statements for gpt2 |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| evaluate_cnn.py | ||
| README.md | ||
| run_bart_sum.py | ||
| run_train.sh | ||
| test_bart_examples.py | ||
| utils.py | ||
Get Preprocessed CNN Data
To be able to reproduce the authors' results on the CNN/Daily Mail dataset you first need to download both CNN and Daily Mail datasets from Kyunghyun Cho's website (the links next to "Stories") in the same folder. Then uncompress the archives by running:
wget https://s3.amazonaws.com/datasets.huggingface.co/summarization/cnn_dm.tgz
tar -xzvf cnn_dm.tgz
this should make a directory called cnn_dm/ with files like test.source.
To use your own data, copy that files format. Each article to be summarized is on its own line.
Evaluation
To create summaries for each article in dataset, run:
python evaluate_cnn.py <path_to_test.source> cnn_test_summaries.txt
the default batch size, 8, fits in 16GB GPU memory, but may need to be adjusted to fit your system.
Training
Run/modify run_train.sh
Where is the code?
The core model is in src/transformers/modeling_bart.py. This directory only contains examples.
(WIP) Rouge Scores
Stanford CoreNLP Setup
ptb_tokenize () {
cat $1 | java edu.stanford.nlp.process.PTBTokenizer -ioFileList -preserveLines > $2
}
sudo apt install openjdk-8-jre-headless
sudo apt-get install ant
wget http://nlp.stanford.edu/software/stanford-corenlp-full-2018-10-05.zip
unzip stanford-corenlp-full-2018-10-05.zip
cd stanford-corenlp-full-2018-10-05
export CLASSPATH=stanford-corenlp-3.9.2.jar:stanford-corenlp-3.9.2-models.jar
Then run ptb_tokenize on test.target and your generated hypotheses.
Rouge Setup
Install files2rouge following the instructions at here.
I also needed to run sudo apt-get install libxml-parser-perl
from files2rouge import files2rouge
from files2rouge import settings
files2rouge.run(<path_to_tokenized_hypo>,
<path_to_tokenized_target>,
saveto='rouge_output.txt')