Feature request: support YAML front matter

[lvh]

Summary

YAML front matter (delimited by ---) is widely used in static site generators (Hugo, Jekyll, etc.) and documentation tools. Currently, nextjournal/markdown parses front matter as regular markdown content, producing incorrect results.

Current behavior

(require '[nextjournal.markdown :as md])

(md/parse "---
title: My Post
date: 2024-01-01
---

## Actual Heading")

Produces:

• A :ruler node (from the opening ---)
• An h2 heading with text "title: My Post" (incorrectly parsed as markdown heading)
• A paragraph with "date: 2024-01-01"
• Another :ruler node (from the closing ---)
• The actual h2 heading

Expected behavior

Front matter should be:

1. Recognized and excluded from markdown parsing, OR
2. Parsed and included as a separate :front-matter node with the YAML content

Example of option 2:

{:type :doc
 :front-matter {:title "My Post" :date "2024-01-01"}
 :content [{:type :heading :heading-level 2 :content [...]}]}

Workaround

Users currently need to strip front matter before calling parse, which is error-prone and loses the metadata.

Implementation note

The underlying commonmark-java library already has a YamlFrontMatterExtension in the commonmark-ext-yaml-front-matter artifact with YamlFrontMatterVisitor for extracting metadata. Using this extension at the commonmark-java level (rather than preprocessing the string) would have the advantage that other features like source line numbers (#64) would work correctly for free, since the parser would know where the actual content starts.

References:

• commonmark-java YAML front matter extension
• Hugo front matter
• Jekyll front matter

[jackrusher]

I'm very lukewarm on this one, as I generally prefer a stronger separation of concerns. OTOH there are likely many people for whom this would be a great convenience 🤷🏻‍♂️

[borkdude]

I've looked at this before and then decided to not include a hard dependency on a YAML parser or any other front-matter specific data format and keep this library purely about markdown (note that there are other front-matter formats like JSON too).

Instead I copied processing logic from markdown-clj and use that in combination with nextjournal.markdown: https://github.com/borkdude/quickblog/blob/main/src/quickblog/internal/frontmatter.clj

[borkdude]

Note that the commonmark-java YAML front-end matter logic only supports a subset of YAML and isn't very compliant to the YAML spec (1.2 or 1.3?). Avoiding dealing with YAML has my preference :).

[borkdude]

Maybe we could just document a snippet in the README how you can accomplish reading YAML front-matters yourself as a solution to this topic.

[jackrusher]

I agree that it would definitely be too much to bring in a whole YAML parser. @lvh has a point about all the line numbers being off-by-N relative to the file in the case where one manually trims the front matter, but computers are good at arithmetic so adding the length of the front matter to every line number shouldn't be difficult :)

[lvh]

Yeah, to be clear; even in the thing I wrote I'm not necessarily parsing the YAML. I think a baby helper function that just takes out the front matter (so you can parse it if and wherever you like) and fixes up the lines is a small but nice convenience that's just tricky enough to get right it's worth a function. I am not suggesting you have a hard dependency on a YAML parser.