OSCAL validation checks failing

[funnelfiasco]

Our OSCAL validation has apparently been failing for some time. See, e.g., https://github.com/ossf/security-baseline/actions/runs/20123940750/job/57749652527

However, we only notice it now because #450 fixes a bug in the action that caused the errors to not be reflected in the exit code.

[xee5ch]

Apologies, that's on yours truly! I can take a look and help later this week. Feel free to assign to me if it helps. That, or I can work with anyone who wants the help.

[puerco]

Thanks @xee5ch ! I'm looking at the output, I'll ping if we get stuck

[iMichaela]

@funnelfiasco - It looks like your validation pipeline is not executing the NIST oscal-cli command line properly. NIST oscal-cli requires to specify teh model , catalog in this case, then the action: validate The metaschema-framework oscal-cli (which does not belong to NIST) is determining the model and then you will not need to specify the model (aka catalog).

This is my conclusion based on the 30 sec look at the log. I can dedicate more time later today (tonight)

oscal-club's oscal-cli is most likely not using the oscal-cli released by NIST. If you are using the OSCAL v1.1.3, I can make the corresponding oscal-cli available. If you are using an older version - then you can find it in maven. I would recommend using at minimum OSCAL 1.1.3. The latest OSCAL release is OSCAL 1.2.0 which adds teh Control Mapping Model to OSCAL.

[iMichaela]

@funnelfiasco - can you point me please to the oscal-sample.json file that fails the validation? Thank you.

[iMichaela]

The OSCAL validation action is checking out from maven the oscal-cli-enhanced which is not the NIST version of the oscal-cli. See https://mvnrepository.com/artifact/dev.metaschema.oscal/oscal-cli-enhanced page. If you want to use this validator, you need to ensure it created using official OSCAL version you expect/want.

NIST oscal-cli released to Maven can be found here: https://mvnrepository.com/artifact/gov.nist.secauto.oscal.tools.oscal-cli/cli-core. Please note, the latest version was not released to maven due to a policy change at NIST, but we can assist if needed.

If you run the

oscal-cli --version

you will be Abel to identify the OSCAL version you are using.
Example output:

oscal-cli 1.0.4 built at 2025-03-12 00:23 from branch main (5cfe61c) at https://github.com/usnistgov/oscal-cli.git
liboscal-java v3.0.3 built at 2025-03-12 00:10 from branch main (f2ca876) at https://github.com/usnistgov/liboscal-java.git
oscal v1.1.3 built at 2025-03-12 00:10 from branch b123c11bd12c8b8f1bcc8bf85763e5775c0423e9 (b123c11) at https://github.com/usnistgov/OSCAL.git
metaschema-java 0.12.2 built at 2023-09-19T20:12:43+0000 from branch f50c0658054175fd2d72fb8755404e8a998e0fb3 (f50c065) at https://github.com/usnistgov/metaschema-java
metaschema v0.9.0 built at 2023-09-19T20:12:43+0000 from branch a36f579e1e30abb2263895242cdbd2cf4bd29513 (a36f579) at https://github.com/usnistgov/metaschema

[puerco]

Thanks @iMichaela! I'm working on an update of the generator, I can upload a sample later today, we generate it on the fly when testing.

[david-waltermire]

The NIST OSCAL CLI hasn't seen any significant work in over the last 2 years. It's also 2 versions of OSCAL behind (1.1.3 and 1.2.0).

A few of us gave up waiting for PRs against the NIST versions to get merged and new releases to be made. We started a derived work at metaschema-framework/oscal-cli. This is not the NIST version, but it is actively maintained, with regular updates for NIST OSCAL releases. The code base and features are way more advanced as compared to the NIST version it is orginally based on. You are welcome to use it and we have made every effort to ensure that compatibility has been preserved with the NIST version and OSCAL releases, while fixing bugs and adding new features requested by users. This tool is extensively used by many in the OSCAL community. We also welcome any bug reports or feature requests you might have.

Please try the latest release, as it will likely fix your issue. If not, please make a bug report.

[david-waltermire]

Looking at this in #450, there is an OSCAL constraints issue. I believe this relates to usnistgov/OSCAL#2118 which has not yet been resolved.

[iMichaela]

I am glad you were able to fix the oscal-cli error reported in usnistgov/OSCAL#2118 in the version you are maintaining and that the pipeline is using a fixed version of the validation tool. This issue indicates that the `oscal-cli wrongly reports a validation error when the syntax is correct per OSCAL schema.
In this context, the usnistgov/OSCAL#2118 you pointed out to, is not to be blamed, because this pipeline is not using the NIST tool.

I have built and I am using the oscal-cli for OSCAL v1.1.3 - as you can see above - and I would like to have the oscal-sample.json file to provide more accurate guidance.

From the log, to me, the top, FATAL CRITICAL errors are not constraint errors. There are plenty of syntax errors and possibly tool invocation warning/error - based on teh schema matching statements below..

Run /home/runner/work/_actions/oscal-club/oscal-cli-action/b5b0c80a1a158797bea4475d13d12c494b56019b/entrypoint.sh
This command path is deprecated. Please use 'validate'.
Validating 'file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json' as JSON.
Validation identified the following issues:
FATAL: [CRITICAL] [#/catalog/groups] #/catalog/groups: 56 schema violations found [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0] #/catalog/groups/0: #: only 1 subschema matches out of 2 [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0] #/catalog/groups/0: #: no subschema matched out of the total 2 subschemas [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0] #/catalog/groups/0: 5 schema violations found [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0/controls] #/catalog/groups/0/controls: 4 schema violations found [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0/controls/0/title] #/catalog/groups/0/controls/0/title: string [The project's version control system MUST require multi-factor
authentication for users modifying the project repository
settings or accessing sensitive data.] does not match pattern ^[^
]+$ [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0/controls/1/title] #/catalog/groups/0/controls/1/title: string [The project's version control system MUST restrict collaborator
permissions to the lowest available privileges by default.] does not match pattern ^[^
]+$ [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]
FATAL: [CRITICAL] [#/catalog/groups/0/controls/2/title] #/catalog/groups/0/controls/2/title: string [The project's version control system MUST prevent unintentional
modification of the primary branch.] does not match pattern ^[^
]+$ [file:/home/runner/work/security-baseline/security-baseline/oscal-sample.json]

The bottom errors are triggered by OSCAL structure and constraints. It is obvious that the values inserted are not OSCAL expected values:

[ERROR] [/catalog/group[8]/control[1]/part[1]/@name] Value 'OSPS-VM-01.01' doesn't match one of 'assessment, assessment-method, assessment-objective, example, guidance, objective, overview, or statement' at path '/catalog/group[8]/control[1]/part[1]/@name'

The error above says that in the first control of the 8th group, the first part of the control has the name="OSPS-VM-01.01". I believe"OSPS-VM-01.01" is the proper value for the id="OSPS-VM-01.01" and the name should be one of the permitted ones, probably name="overview"; name="statement" or name="guidance". My guess is based on the top-level placement (first part) of the information.