AUPRC as a loss function

[pedro-ribeirosantiago]

Hi all,

Thanks for the excellent package! Is there any way to use the Area Under the Precision-Recall Curve (AUPRC) as the measure to evaluate cross-validation? This would be useful in cases of an imbalanced dataset.

Saito, T., & Rehmsmeier, M. (2015). The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets. PloS one, 10(3), e0118432.

Cheers,
Pedro

[jaredhuling]

I would be happy to look at a pull request if you want to code it up

[pedro-ribeirosantiago]

Hello Jared,

I believe I figured out the code to use the AUPRC as the measure to evaluate cross-validation. However, it will not be possible to account for observation weights (as in the case of the AUC using the function glmnet:::auc.mat) since there is no R function to compute the Weighted AUPRC. Should I go ahead and open a pull request?

Cheers,
Pedro

[glycerine]

I would be interested in AUPRC too. @pedro-ribeirosantiago could you describe how far you got, or post a pull request here? Thanks!

[glycerine]

Ah! I see you did submit a PR.

#24

@jaredhuling Could you comment? Would this be okay to merge in? Thanks guys.

[pedro-ribeirosantiago]

Hey @glycerine and @jaredhuling,

I did the pull request last year. The build had a failure only in the 'R_VERSION=devel, R_ARCH=x64'. I'm experienced with R programming but, unfortunately, I was unable to identify the problem since the modifications were built correctly in the "stable" and "patched" versions. I'm happy to provide further support and go ahead. Please let me know how I can help!

Cheers,
Pedro

[glycerine]

Thanks @pedro-ribeirosantiago ! I was able to merge and load the patch just fine on my machine. I'm using linux, amd64, and R 4.1.0.

I tried to look at why the resulting oem package would install fine on stable and patched but not on devel. However the artifact log gets deleted after a month, and so AppVeyor does not have it any more.

@pedro-ribeirosantiago Could you make a small tiny change (to a comment perhaps) in the pull request and push it that so that AppVeyor will re-run the check and we can inspect the C:/projects/oem/oem.Rcheck/00install.out file which should contain the error.

[pedro-ribeirosantiago]

Dear @glycerine and @jaredhuling,

I just made a new pull request and now it passed all tests, so it would be ready for merging. Please let me know if you need anything from my side and I'm happy to keep making changes if needed!

Cheers,
Pedro