yaml-reference

Using ruamel.yaml, support cross-file references in YAML files using tags !reference and !reference-all.

Install the package from PyPI with:

# pip
pip install yaml-reference
# poetry
poetry add yaml-reference
# uv
uv add yaml-reference

Spec

This Python library implements the YAML specification for cross-file references in YAML files using tags !reference and !reference-all as defined in the yaml-reference-specs project.

Example

# root.yaml
version: "3.1"
services:
  - !reference
    path: "services/website.yaml"

  - !reference
    path: "services/database.yaml"

networkConfigs:
  !reference-all
  glob: "networks/*.yaml"

Supposing there are services/website.yaml and services/database.yaml files in the same directory as root.yaml, and a networks directory with YAML files, the above will be expanded to account for the referenced files with the following Python code:

from yaml_reference import load_yaml_with_references

data = load_yaml_with_references("root.yaml")
print(data)
# {"networkConfigs": [{"network": "vpn","version": "1.1"},{"network": "nfs","version": "1.0"}],"services": ["website","database"],"version": "3.1"}

Note that the load_yaml_with_references function instantiates a ruamel.yaml.YAML loader class (typ='safe') to perform the deserialization of the YAML files, and returns a Python dictionary with the recursively-expanded YAML data.

If you wish to resolve one "layer" of references without recursively exhausting the entire reference graph, the parse_yaml_with_references function can be used to obtain the original YAML document's contents with !reference/!reference-all tags as dedicated objects called Reference and ReferenceAll.

from yaml_reference import parse_yaml_with_references

data = parse_yaml_with_references("root.yaml")
print(data["networkConfigs"])
# ReferenceAll(glob="networks/*.yaml", location="/path/to/root.yaml")

VSCode squigglies

To get red of red squigglies in VSCode when using the !reference and !reference-all tags, you can add the following to your settings.json file:

    "yaml.customTags": [
        "!reference mapping",
        "!reference-all mapping"
    ]

CLI interface

There is a CLI interface for this package which can be used to read a YAML file which contains !reference tags and dump its contents as pretty-printed JSON with references expanded. This is useful for generating a single file for deployment or other purposes. Note that the keys of mappings will be sorted alphabetically. This CLI interface is used to test the contract of this package against the yaml-reference-specs project.

$ yaml-reference-cli -h
  usage: yaml-reference-cli [-h] input_file

  Compile a YAML file containing !reference tags into a new YAML file with resolved references. Expects a YAML file to be provided via the "input_file" argument. Outputs JSON content to stdout.

  positional arguments:
    input_file  Path to the input YAML file with references to resolve and print as JSON.

  options:
    -h, --help  show this help message and exit
$ yaml-reference-cli root.yaml
  {
    "networkConfigs": [
      {
        "network": "vpn",
        "version": "1.1"
      },
      {
        "network": "nfs",
        "version": "1.0"
      }
    ],
    "services": [
      "website",
      "database"
    ],
    "version": "3.1"
  }

It's still possible to yield the results as a YAML file using the yq CLI tool (mikefarah/yq).

$ yaml-reference-cli root.yaml | yq -P
networkConfigs:
  - network: vpn
    version: 1.1
  - network: nfs
    version: 1.0
services:
  - website
  - database
version: 3.1
# Pipe it to a result file
$ yaml-reference-cli root.yaml | yq -P > .compiled/root.yaml

Safety note

As of now, the specification does not require any explicit protection against circular references. This package does not check for circular references and will result in an infinite loop (max recursion depth exceeded) if a circular reference is encountered. Onus is on the users of this package to ensure that circular references are not present in their referential YAML files.

Acknowledgements

Author(s):

• David Sillman dsillman2000@gmail.com
  • Personal website: https://www.dsillman.com