Instability in audit pass/fail

[kf6kjg]

I've nearly minimized a reproduction of the problem at https://github.com/kf6kjg/audit-ci-tests, you can review the actions runs for the CI side.

There are two cases to this problem, not sure if they are related or not.

1. Locally I can run the audit, but I get stuck in a loop if I just keep following the instructions:

   $ npm run test
   
   > audit-ci-tests@1.0.0 test
   > npx audit-ci@7.1.0 --moderate --report-type summary --config ./.audit-ci.jsonc
   
   NPM audit report summary:
   {
     "vulnerabilities": {
       "info": 0,
       "low": 0,
       "moderate": 88,
       "high": 10,
       "critical": 1,
       "total": 99
     },
     "dependencies": {
       "prod": 1,
       "dev": 820,
       "optional": 56,
       "peer": 372,
       "peerOptional": 0,
       "total": 820
     }
   }
   Consider not allowlisting paths: GHSA-566m-qj78-rww5|css-blank-pseudo>postcss>, GHSA-566m-qj78-rww5|css-has-pseudo>postcss, GHSA-7fh5-64p2-3v2j|css-blank-pseudo>postcss>, GHSA-7fh5-64p2-3v2j|css-has-pseudo>postcss, GHSA-hwj9-h5mp-3pm3|css-blank-pseudo>postcss>, GHSA-hwj9-h5mp-3pm3|css-has-pseudo>postcss.
   Passed npm security audit.

   I then removed all the paths indicated.

   diff --git a/.audit-ci.jsonc b/.audit-ci.jsonc
   index d8d4b47..8c1c11f 100644
   --- a/.audit-ci.jsonc
   +++ b/.audit-ci.jsonc
   @@ -4,9 +4,7 @@
      // Empty list fails.
      "GHSA-3rfm-jhwj-7488|resolve-url-loader>loader-utils",
      "GHSA-566m-qj78-rww5|@tailwindcss/postcss7-compat>postcss",
   -    "GHSA-566m-qj78-rww5|css-blank-pseudo>postcss>",
      "GHSA-566m-qj78-rww5|css-declaration-sorter>postcss",
   -    "GHSA-566m-qj78-rww5|css-has-pseudo>postcss",
      "GHSA-566m-qj78-rww5|css-loader>postcss",
      "GHSA-566m-qj78-rww5|css-prefers-color-scheme>postcss>",
      "GHSA-566m-qj78-rww5|cssnano-preset-default>postcss>",
   @@ -89,9 +87,7 @@
      "GHSA-566m-qj78-rww5|stylehacks>postcss>",
      "GHSA-76p3-8jx3-jpfq|resolve-url-loader>loader-utils",
      "GHSA-7fh5-64p2-3v2j|@tailwindcss/postcss7-compat>postcss",
   -    "GHSA-7fh5-64p2-3v2j|css-blank-pseudo>postcss>",
      "GHSA-7fh5-64p2-3v2j|css-declaration-sorter>postcss",
   -    "GHSA-7fh5-64p2-3v2j|css-has-pseudo>postcss",
      "GHSA-7fh5-64p2-3v2j|css-loader>postcss",
      "GHSA-7fh5-64p2-3v2j|css-prefers-color-scheme>postcss>",
      "GHSA-7fh5-64p2-3v2j|cssnano-preset-default>postcss>",
   @@ -183,9 +179,7 @@
      "GHSA-grv7-fg5c-xmjg|watchpack>watchpack-chokidar2>chokidar>braces",
      "GHSA-hhq3-ff78-jv3g|resolve-url-loader>loader-utils",
      "GHSA-hwj9-h5mp-3pm3|@tailwindcss/postcss7-compat>postcss",
   -    "GHSA-hwj9-h5mp-3pm3|css-blank-pseudo>postcss>",
      "GHSA-hwj9-h5mp-3pm3|css-declaration-sorter>postcss",
   -    "GHSA-hwj9-h5mp-3pm3|css-has-pseudo>postcss",
      "GHSA-hwj9-h5mp-3pm3|css-loader>postcss",
      "GHSA-hwj9-h5mp-3pm3|css-prefers-color-scheme>postcss>",
      "GHSA-hwj9-h5mp-3pm3|cssnano-preset-default>postcss>",

   And then re-ran the audit to find it asking me to add them back:

   $ npm run test
   
   > audit-ci-tests@1.0.0 test
   > npx audit-ci@7.1.0 --moderate --report-type summary --config ./.audit-ci.jsonc
   
   NPM audit report summary:
   {
     "vulnerabilities": {
       "info": 0,
       "low": 0,
       "moderate": 88,
       "high": 10,
       "critical": 1,
       "total": 99
     },
     "dependencies": {
       "prod": 1,
       "dev": 820,
       "optional": 56,
       "peer": 372,
       "peerOptional": 0,
       "total": 820
     }
   }
   Consider not allowlisting paths: GHSA-566m-qj78-rww5|css-blank-pseudo>postcss, GHSA-566m-qj78-rww5|css-has-pseudo>postcss>, GHSA-7fh5-64p2-3v2j|css-blank-pseudo>postcss, GHSA-7fh5-64p2-3v2j|css-has-pseudo>postcss>, GHSA-hwj9-h5mp-3pm3|css-blank-pseudo>postcss, GHSA-hwj9-h5mp-3pm3|css-has-pseudo>postcss>.
   Found vulnerable advisory paths:
   GHSA-566m-qj78-rww5|css-blank-pseudo>postcss>
   GHSA-566m-qj78-rww5|css-has-pseudo>postcss
   GHSA-7fh5-64p2-3v2j|css-blank-pseudo>postcss>
   GHSA-7fh5-64p2-3v2j|css-has-pseudo>postcss
   GHSA-hwj9-h5mp-3pm3|css-blank-pseudo>postcss>
   GHSA-hwj9-h5mp-3pm3|css-has-pseudo>postcss
   Failed security audit due to moderate vulnerabilities.
   Vulnerable advisories are:
   https://github.com/advisories/GHSA-566m-qj78-rww5
   https://github.com/advisories/GHSA-7fh5-64p2-3v2j
   https://github.com/advisories/GHSA-hwj9-h5mp-3pm3
   Exiting...

2. The second problem is that even if I have it passing locally it can still fail on CI as is demonstrated in my reproduction repo.

Additionally in the past, same version of audit-ci, I thought I'd observed that running either npm ci or rm -rf node_modules before the audit call could have behavior, but I'm not observing that in my test repo.