Details on Train Data Formatting

[monikernemo]

Hi, very nice repo here, just want to confirm with you the formatting of the data before it is being fed into preprocess.py.

1. Are the inputs delimited by new lines?
2. or can we assume that each input here before it sees preprocess.txt is itself is an article.

I have been trying to replicate the results of this work in other languages but to no avail. I understand that because of the span random sampling, theoretically the input should not make a huge difference unless of course each paragraph is significantly shorter than that.

Even though the Wikipedia - En corpus is considered to be a "canonical" dataset, I would appreciate if you could share the size of the dataset used to replicate your previous result just to observe what could be tweaked. Thanks!