Allow experimentation script to take time parameter indicating how long to run (Experimentation.py) #114
Description
A user should be able to specify a given time budget for experimentation/tuning and the script should automatically adjust its work to honor that time budget. This requires some careful though/measurement/design. An initial plan could be:
- Measure the time it takes for a single VW pass on the data (this might change depending on the parameters, however).
- Divide up the time budget across the different experimentation areas (learning rate, regularization, feature interaction, estimators). Should the division be equal? Which areas matter more?
- Report the total time taken and best set of parameters found so far.