Allow conditional permutation tests?

[linstonwin]

In the section on permutation tests, we've written, "We recommend attempting the permutation test with mock outcome data and actual covariate data before analyzing the actual outcome data. The mock permutation test may reveal that on some randomizations, the t-statistic cannot be computed because the regressors are collinear or because the HC2 or BM SE is undefined (see the section above on 'Avoiding regression models that do not allow the BM adjustment'). In such cases, covariates should be dropped from the model until the mock permutation test runs without errors."

I'm thinking to change this so that if the t-statistic is uncomputable on only a small % of randomizations (e.g., less than 5%), we do a conditional permutation test (i.e., randomizations where the t-stat is undefined are excluded from both the numerator and the denominator of the p-value).

One situation where this might happen is if the PAP specifies poststratification and there are some randomizations where all units in some poststratum are assigned to one treatment condition.

[donaldpgreen]

That sounds sensible -- have you ever encountered a case where more than 5%
of the permutations were upended by this kind of collinearity?

On Tue, Jul 19, 2016 at 1:52 PM, Winston Lin notifications@github.com
wrote:

In the section on permutation tests, we've written, "We recommend
attempting the permutation test with mock outcome data and actual covariate
data before analyzing the actual outcome data. The mock permutation test
may reveal that on some randomizations, the t-statistic cannot be computed
because the regressors are collinear or because the HC2 or BM SE is
undefined (see the section above on 'Avoiding regression models that do not
allow the BM adjustment'). In such cases, covariates should be dropped from
the model until the mock permutation test runs without errors."

I'm thinking to change this so that if the t-statistic is uncomputable on
only a small % of randomizations (e.g., less than 5%), we do a conditional
permutation test (i.e., randomizations where the t-stat is undefined are
excluded from both the numerator and the denominator of the p-value).

One situation where this might happen is if the PAP specifies
poststratification and there are some randomizations where all units in
some poststratum are assigned to one treatment condition.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#5, or mute the thread
https://github.com/notifications/unsubscribe-auth/AQnUS97UULrmuYXkJZhoamAOC7Y1U_A8ks5qXQ7kgaJpZM4JP_XK
.

[linstonwin]

No, it's just a theoretical possibility that occurred to me. :)

winston

Sent from my phone

On Jul 19, 2016, at 3:57 PM, donaldpgreen notifications@github.com wrote:

That sounds sensible -- have you ever encountered a case where more than 5%
of the permutations were upended by this kind of collinearity?

On Tue, Jul 19, 2016 at 1:52 PM, Winston Lin notifications@github.com
wrote:

In the section on permutation tests, we've written, "We recommend
attempting the permutation test with mock outcome data and actual covariate
data before analyzing the actual outcome data. The mock permutation test
may reveal that on some randomizations, the t-statistic cannot be computed
because the regressors are collinear or because the HC2 or BM SE is
undefined (see the section above on 'Avoiding regression models that do not
allow the BM adjustment'). In such cases, covariates should be dropped from
the model until the mock permutation test runs without errors."

I'm thinking to change this so that if the t-statistic is uncomputable on
only a small % of randomizations (e.g., less than 5%), we do a conditional
permutation test (i.e., randomizations where the t-stat is undefined are
excluded from both the numerator and the denominator of the p-value).

One situation where this might happen is if the PAP specifies
poststratification and there are some randomizations where all units in
some poststratum are assigned to one treatment condition.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#5, or mute the thread
https://github.com/notifications/unsubscribe-auth/AQnUS97UULrmuYXkJZhoamAOC7Y1U_A8ks5qXQ7kgaJpZM4JP_XK
.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.