Separating construction of weights matrix and thresholding in reconstructors

[sdmccabe]

As in #174, we should explicitly separate thresholding as a separate behavior in graph reconstruction. The reconstructors should return either complete graphs or weight matrices, that are then thresholded or backboned with a separate .threshold() (or something similar) call.

The advantages of this, from my perspective, are:

1. the current thresholding behavior is filled with odd defaults and unexpected behavior around kwargs. We did this to make things as simple as possible, but adding the slight additional complexity of requiring two method calls to get a useful graph makes the process much more explicit and predictable.
2. By being explicit about thresholding as a separate behavior, it sets us up to write more modular (and useful) thresholding and backboning code, perhaps even as another submodule. (netrdb)

[sdmccabe]

A sketch of what this could look like:

class BaseReconstructor:
    def __init__(self):
        self.matrix = np.array([])
        self.results = {}

    def fit(self, TS, **kwargs):
        self.matrix = np.zeros(TS.shape[0])
        return self

    def threshold_on_degree(self, avg_k=None, binary=True):
        if not avg_k:
            avg_k = 1
            warnings.warn(
                "Setting 'avg_k' argument is strongly encouraged. Using "
                "average degree of 1 for thresholding.",
                RuntimeWarning,
            )

        n = len(self.matrix)
        A = np.ones((n, n))

        if np.mean(np.sum(A, 1)) <= avg_k:
            # degenerate case: threshold the whole matrix
            thresholded_mat = self.matrix
        else:
            for m in sorted(self.matrix.flatten()):
                A[self.matrix == m] = 0
                if np.mean(np.sum(A, 1)) <= avg_k:
                    break
            thresholded_mat = self.matrix * (self.matrix > m)

        if binary:
            thresholded_mat = np.abs(np.sign(thresholded_mat))

        return self

    def to_matrix(self):
        return self.matrix

    def to_graph(self, remove_self_loops=True, create_using=None):
        A = self.matrix
        if remove_self_loops:
            np.fill_diagonal(A, 0)

        if create_using is None:
            if np.allclose(A, A.T):
                G = nx.from_numpy_array(A, create_using=nx.Graph())
            else:
                G = nx.from_numpy_array(A, create_using=nx.DiGraph())
        else:
            G = nx.from_numpy_array(A, create_using=create_using)

        return G

So the usage would be something like

TS = some_data

R = netrd.reconstruction.BaseReconstructor()
G = R.fit(TS).threshold_by_degree(k_avg = 16).to_graph()
A = R.to_matrix()

X = R.results # i.e. we keep self.results for intermediate things

Most of this is just shuffling things around, like moving the thresholding and graph creation functions out of utilities and turning them into class methods.

[sdmccabe]

Thoughts on this, @tlarock @leotrs @jkbren?

[leotrs]

I like this, though I'm unfamiliar with the design that returns self. Are we sure we are covering all the possibilities here?

Also, nitpicking, threshold_on_degree should assign to self.matrix, right?

[sdmccabe]

1. return self sets up the method chaining used in the example; there might be a better way to achieve that.
2. yeah, I flubbed a few lines in the threshold example. I think threshold functions should build up the new thresholded_matrix and then assign that to self.matrix, or modify it directly; I prefer the former in case there's an exception during the function logic.

[leotrs]

I think I like it though I would like to see a few more examples. This return self seems error prone? Is there a way of chaining calls that results in deleting data?

It's fine if a certain chain of calls does not work (not all chains should be supported). But I'm worried there's a chain somewhere that looks reasonable but results in a catastrophe.

[sdmccabe]

FWIW, if you look at Pandas's internals you see some return selfs but also some return self.copy()s.