Add regularization to synthesis methods via penalty_function

[dherrera1911]

Describe the change in this PR at a high-level

This PR adds basic functionality to include regularization in the synthesis objects Metamer and MADCompetition

Link any related issues, discussions, PRs

This closes Issue #363

Outstanding questions / particular feedback

• Should we add a test for the case when an invalid penalty_function is passed? E.g. requiring more than one input
• Any missing tests for the functionality? Currently only save/load tests were added, I think.
• There is a need for an example of the new functionality in the documentation. Should this be done in this PR, or in a separate one? Can use feedback of where to add this, and what new penalty_function to use as an example.
• I need to check whether other places of current documentation need adjusting. LMK if you already know some.

Describe changes

• Removed old parameters range_penalty_lambda and allowed_range as inputs to Metamer and MADCompetition
• Added parameters penalty_function and penalty_lambda
• penalty_function defaults to the old behavior
• Added tests for save/load of new parameters
• Hard-coded allowed_range in validate_input
• Small changes to the tests, to pass without allowed_range in some cases
• Created new module po.tools.regularization, moved penalize_range there from po.tools.optim

Checklist

Affirm that you have done the following:

• [X ] I have described the changes in this PR, following the template above.
• [ X] I have added any necessary tests.
• I have added any necessary documentation. This includes docstrings, updates to existing files found in docs/, or (for large changes) adding new files to the docs/ folder.

[dherrera1911]

I see some failings from build / Execute notebooks that look like there is a problem with saved/cached objects that can't load because of the changes to the Metamer and MADCompetition classes. I'm not sure how to fix that, but let me know if my diagnostic sounds right.

[dherrera1911]

Hey @billbrod I think the PR is ready for a code review, everything on my TODO list is done with for this PR.

There seem to be some errors in the automatic checks, having to do with the FutureWarning from the load/save of PortillaSimoncelli objects or something like that. We had commented on this before, and you mentioned:

Run the failing notebooks (and Jenkins tests) locally to create new versions of the saved files, which need to be uploaded to OSF.
For that last step, it's probably easier if I do it, so I can do that once we've done the other work (the save files also change in #381, so depending on the relative speed of these PRs, we might have to re-generate them anyway).

[billbrod]

Thanks for your work Daniel!

Looking through this, I'm realizing I got confused as to where we ended up with partials. From what you have here, my understanding is, in order to change the allowed_range, users will have to either use partials or define their own custom penalty like

def custom_penalty(x):
    return po.tools.regularization.penalize_range(x, allowed_range=(.2, .8))

If they use partials, they will be unable to save/load, right (because of the issue with _get_name)? In that case, I do not think we should tell people to use partials, since that's a weird sharp edge.

Instead, I think the docs / examples should show people how to write a custom penalty like above. Advantage here, is it should be easy to see that you can easily combine two penalties in that new function (and the docs should say so, explicitly).

The reason to use partials is if you're lazy (like me) and want to do it in the same line where you pass it to the init method:

po.synth.Metamer(
    img,
    model,
    penalty_function=functools.partial(po.tools.regularization.penalize_range, allowed_range=(0.0, 255.0)
)

And I don't think that's worth telling people about -- advanced users will realize they can do that based on the custom def example, and otherwise we should explain (or at least link) to what partials mean for non-advanced users

Other notes:

• I'm not sure what to do for the description of penalty_function in the docstrings. I don't think "It constrains certain properties of the metamer" is that informative, but without that sentence, it reads fairly abstract. What about something like "For example, this can penalize all values that lie outside of a specified range"?
• Need to add a paragraph to the reproducibility.md docs page about the change to penalty_function. should be pretty straightforward, more or less a copy of the existing paragraph under FutureWarning in load in plenoptic 1.4
• Metamer and MADCompetition's init docstrings should have examples setting penalty_function

For the failing PS synthesis tests -- yes, I'll go ahead and update these once we've otherwise finished with this PR.

[billbrod]

split this into two UserWarning. that's the proper way for numpydoc, I believe.

[dherrera1911]

You mean like:

    Warns
    -----
    UserWarning
        If ``input_tensor`` is not 4d.
        If  ``input_tensor`` has values outside (0, 1).

Or

    Warns
    -----
    UserWarning
        If ``input_tensor`` is not 4d.
    UserWarning
        If  ``input_tensor`` has values
        outside (0, 1).

Also, I think I wrote it like this following validate_model that has

UserWarning
   If ``model`` is in training mode or returns an output with other than 3 or 4
   dimensions.

[billbrod]

The second. (see the ssim docstring for an example)

And oh you're totally right. Can you change validate_model as well?

[billbrod]

instead, have this intentionally fail because of the dtype

[dherrera1911]

You mean, change "Raise warning:" for "Intentionally fail because of the dtype:"?

[billbrod]

no, I mean have this block still intentionally fail, but for a different reason: because the dtype is an int.

[dherrera1911]

@billbrod I pushed the last edits, addressing everything... but I made a huge mess with git, as you might see. If you can help me fix this on a call, that'd be great.

[billbrod]

@billbrod I pushed the last edits, addressing everything... but I made a huge mess with git, as you might see. If you can help me fix this on a call, that'd be great.

Okay, let's find a time to meet next week. I sent you an email to find a time.