Press Kit

Yet another attempt at creating a full-stack parsing/analysis framework. We do it for fun anyway.

Datasets

• Fetched pages - 4.7 Gb (updated 29-06-2014)
• Parsed pages - 100 Mb (mongo dump, updated 29-06-2014)

Running

There are several steps:

• Obtain the URLs
• Save the articles in a folder
• Parse the articles
• Squash the results into one big JSON
• Analyze the full data set
• Create a visualization

Run make run, wait around 6 hours and you'll have everything. You'll also need around 7GB of disk space.