chore: TCO for tailstrict. by He-Pin · Pull Request #619 · databricks/sjsonnet

He-Pin · 2026-03-01T12:42:38Z

Motivation:
Add partial TCO.

local sum(n, accum=0) =
  if n <= 0 then accum
  else sum(n - 1, accum + n) tailstrict;
 
local sz = 10000;
std.assertEqual(sum(sz), sz * (sz + 1) / 2)

can work now and evaluate to true but was:

on main

125] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
125] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.434          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.076          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       41.836          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       13.310          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       32.440          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.453          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        2.910          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.060          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.067          ms/op
125] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        0.998          ms/op
125] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        2.000          ms/op
125] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        2.384          ms/op
125] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.287          ms/op
125] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       75.854          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.846          ms/op
125] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.650          ms/op
125] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt        9.567          ms/op
125] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.495          ms/op
125] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt       23.110          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       80.098          ms/op
125] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.051          ms/op
125] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt        9.389          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.652          ms/op
125] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.073          ms/op
125] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.090          ms/op
125] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.076          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.744          ms/op
125] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.053          ms/op
125] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       11.664          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.655          ms/op
125] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.635          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.164          ms/op
125/125, SUCCESS] ./mill bench.runRegressions 399s

current:

125] Benchmark                                                                  (path)  Mode  Cnt   Score   Error  Units
125] RegressionBenchmark.main             bench/resources/bug_suite/assertions.jsonnet  avgt        0.324          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.01.jsonnet  avgt        0.072          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.02.jsonnet  avgt       41.850          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.03.jsonnet  avgt       12.604          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.04.jsonnet  avgt       32.272          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.06.jsonnet  avgt        0.449          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.07.jsonnet  avgt        3.592          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.08.jsonnet  avgt        0.059          ms/op
125] RegressionBenchmark.main               bench/resources/cpp_suite/bench.09.jsonnet  avgt        0.067          ms/op
125] RegressionBenchmark.main         bench/resources/cpp_suite/gen_big_object.jsonnet  avgt        1.012          ms/op
125] RegressionBenchmark.main      bench/resources/cpp_suite/large_string_join.jsonnet  avgt        2.212          ms/op
125] RegressionBenchmark.main  bench/resources/cpp_suite/large_string_template.jsonnet  avgt        2.375          ms/op
125] RegressionBenchmark.main             bench/resources/cpp_suite/realistic1.jsonnet  avgt        3.277          ms/op
125] RegressionBenchmark.main             bench/resources/cpp_suite/realistic2.jsonnet  avgt       73.374          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/base64.jsonnet  avgt        0.834          ms/op
125] RegressionBenchmark.main            bench/resources/go_suite/base64Decode.jsonnet  avgt        0.639          ms/op
125] RegressionBenchmark.main       bench/resources/go_suite/base64DecodeBytes.jsonnet  avgt        9.436          ms/op
125] RegressionBenchmark.main       bench/resources/go_suite/base64_byte_array.jsonnet  avgt        1.502          ms/op
125] RegressionBenchmark.main              bench/resources/go_suite/comparison.jsonnet  avgt       22.254          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/comparison2.jsonnet  avgt       77.183          ms/op
125] RegressionBenchmark.main        bench/resources/go_suite/escapeStringJson.jsonnet  avgt        0.051          ms/op
125] RegressionBenchmark.main                   bench/resources/go_suite/foldl.jsonnet  avgt        9.352          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/lstripChars.jsonnet  avgt        0.647          ms/op
125] RegressionBenchmark.main          bench/resources/go_suite/manifestJsonEx.jsonnet  avgt        0.073          ms/op
125] RegressionBenchmark.main          bench/resources/go_suite/manifestTomlEx.jsonnet  avgt        0.088          ms/op
125] RegressionBenchmark.main         bench/resources/go_suite/manifestYamlDoc.jsonnet  avgt        0.076          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/member.jsonnet  avgt        0.752          ms/op
125] RegressionBenchmark.main                bench/resources/go_suite/parseInt.jsonnet  avgt        0.053          ms/op
125] RegressionBenchmark.main                 bench/resources/go_suite/reverse.jsonnet  avgt       11.094          ms/op
125] RegressionBenchmark.main             bench/resources/go_suite/rstripChars.jsonnet  avgt        0.655          ms/op
125] RegressionBenchmark.main              bench/resources/go_suite/stripChars.jsonnet  avgt        0.637          ms/op
125] RegressionBenchmark.main                  bench/resources/go_suite/substr.jsonnet  avgt        0.167          ms/op
125/125, SUCCESS] ./mill bench.runRegressions 399s

Another case is :

local sum = function(n, acc=0)
    if n == 0 then
      acc
    else
      sum(n - 1, acc + n) tailstrict; 

sum(100000)

Copilot

Pull request overview

This PR introduces tail-call optimization support for tailstrict function calls and makes materialization iterative to avoid JVM stack overflows, with accompanying tests and updated error stack expectations.

Changes:

Add a TailCall sentinel + TailstrictMode protocol and integrate a trampoline (TailCall.resolve) into function application paths.
Rewrite Materializer.apply0 to use an explicit stack (iterative traversal) and add Settings.maxMaterializeDepth as a safety bound.
Add comprehensive tailstrict/Materializer integration tests and update the expected stack trace golden output.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
sjsonnet/test/src/sjsonnet/TailCallOptimizationTests.scala	New test suite covering `tailstrict` semantics, deep recursion, and Materializer integration (no TailCall leakage).
sjsonnet/test/resources/test_suite/error.tailstrict_stack.jsonnet.golden	Updates expected stack trace frame labeling after TCO changes.
sjsonnet/src/sjsonnet/Val.scala	Implements `TailCall`, `TailCall.resolve`, `TailstrictMode`, and resolves TailCalls at function-application boundaries.
sjsonnet/src/sjsonnet/Settings.scala	Adds `maxMaterializeDepth` setting used by the new iterative materializer.
sjsonnet/src/sjsonnet/Materializer.scala	Replaces recursive materialization with an iterative, stack-based algorithm + defensive TailCall leakage check.
sjsonnet/src/sjsonnet/Expr.scala	Adds `TailstrictableExpr` marker trait used by the evaluator’s tail-call support.
sjsonnet/src/sjsonnet/Evaluator.scala	Integrates TailCall trampoline wrapping for `tailstrict` calls and adds tail-position evaluation support.
sjsonnet/src/sjsonnet/Error.scala	Refactors frame addition to allow adding frames by `exprErrorString` (`addFrameString`).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sjsonnet/src/sjsonnet/Expr.scala

sjsonnet/src/sjsonnet/Val.scala

sjsonnet/src/sjsonnet/Materializer.scala

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sjsonnet/src/sjsonnet/Settings.scala

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

sjsonnet/src/sjsonnet/Evaluator.scala

sjsonnet/test/src/sjsonnet/TailCallOptimizationTests.scala

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot AI review requested due to automatic review settings March 1, 2026 12:42

Copilot started reviewing on behalf of He-Pin March 1, 2026 12:43 View session

He-Pin changed the title ~~chore: TOC for tailstrict and Materializer.~~ chore: TCO for tailstrict and Materializer. Mar 1, 2026

Copilot AI reviewed Mar 1, 2026

View reviewed changes

sjsonnet/src/sjsonnet/Expr.scala Outdated Show resolved Hide resolved

sjsonnet/src/sjsonnet/Val.scala Show resolved Hide resolved

sjsonnet/src/sjsonnet/Materializer.scala Outdated Show resolved Hide resolved

He-Pin marked this pull request as draft March 1, 2026 12:57

He-Pin force-pushed the toc branch 2 times, most recently from d57e5e0 to 5dd9b27 Compare March 1, 2026 14:57

He-Pin mentioned this pull request Mar 1, 2026

chore: Make materializer stackless for obj and arr #620

Draft

He-Pin changed the title ~~chore: TCO for tailstrict and Materializer.~~ chore: TCO for tailstrict. Mar 1, 2026

He-Pin requested a review from Copilot March 1, 2026 15:11

Copilot started reviewing on behalf of He-Pin March 1, 2026 15:11 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

sjsonnet/src/sjsonnet/Settings.scala Outdated Show resolved Hide resolved

He-Pin force-pushed the toc branch from 5dd9b27 to cc44c7b Compare March 1, 2026 15:39

He-Pin marked this pull request as ready for review March 1, 2026 15:40

He-Pin requested a review from Copilot March 1, 2026 16:01

Copilot started reviewing on behalf of He-Pin March 1, 2026 16:01 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

sjsonnet/src/sjsonnet/Evaluator.scala Show resolved Hide resolved

sjsonnet/test/src/sjsonnet/TailCallOptimizationTests.scala Show resolved Hide resolved

He-Pin force-pushed the toc branch from d973244 to bac21d9 Compare March 1, 2026 17:49

He-Pin requested a review from Copilot March 1, 2026 17:49

Copilot started reviewing on behalf of He-Pin March 1, 2026 17:49 View session

Copilot AI reviewed Mar 1, 2026

View reviewed changes

chore: TCO for tailstrict

5f62371

He-Pin force-pushed the toc branch from bac21d9 to 5f62371 Compare March 1, 2026 18:47

He-Pin mentioned this pull request Mar 2, 2026

TCO in StaticOptimizer #623

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: TCO for tailstrict.#619

chore: TCO for tailstrict.#619
He-Pin wants to merge 1 commit intodatabricks:masterfrom
He-Pin:toc

He-Pin commented Mar 1, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

He-Pin commented Mar 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

He-Pin commented Mar 1, 2026 •

edited

Loading