Benchmark Ideas

[salman1993]

Really cool project! I am curious about the benchmarks, especially around reliability across runs.

Say we have workflows like these:

  # Slack Channel Summary DM
  # Fetches messages from #general and #random, summarizes each, and DMs you

  import "slack" from "mcp:remote-slack-server"

  agent fetcher:
    model: haiku
    skills: ["slack"]
    prompt: "You fetch Slack messages"

  agent summarizer:
    model: sonnet
    skills: ["slack"]
    prompt: "You create concise one-line summaries"

  # Fetch both channels in parallel
  parallel:
    general_msgs = session: fetcher
      prompt: "Get messages from #general channel for the last 7 days"

    random_msgs = session: fetcher
      prompt: "Get messages from #random channel for the last 7 days"

  # Summarize each channel in parallel
  parallel:
    general_summary = session: summarizer
      prompt: "Write a single one-line summary of the key activity/themes"
      context: general_msgs

    random_summary = session: summarizer
      prompt: "Write a single one-line summary of the key activity/themes"
      context: random_msgs

  # DM the summaries
  session: summarizer
    prompt: """
      Send me a Slack DM with this format:

      📊 *7-Day Channel Summary*

      *#general:* {general_summary}
      *#random:* {random_summary}
    """
    context: { general_summary, random_summary }

I am curious how prose program perform compared to detailed English instructions since its still inferred by LLMs:

  1. Call slack__get_msgs with channel='general', num_days=7
  2. Call slack__get_msgs with channel='random', num_days=7
  3. Write a one-line summary of #general activity
  4. Write a one-line summary of #random activity
  5. Call slack__send_dm to me with both summaries formatted as:
     "📊 7-Day Channel Summary"
     "#general: <summary>"
     "#random: <summary>"

The one above is obviously really simple but if we were to do this over 100 slack channels, then it gets hard & there's lot of variance across runs (completing without traversing all channels for example). Have you looked into this at all or can you share anything?

[irl-dan]

Hi @salman1993

It's a great question. Benchmarking for consistent execution flow across runs of the same .prose is a great class of benchmarks.

I have lots of benchmarking ideas and very little benchmarking money. Here are the other categories of benchmark that I've been dreaming of:

https://x.com/irl_danB/status/2008549961508540626

At some point I would like to write this up into a formal spec, and prioritize them from there.

RIght now I'm going off my own "vibes bench" for all improvements. If you have benchmarking experience or further ideas, they are welcome in this thread. I'll keep this thread open as a grounds to collect these ideas.

Dan

[salman1993]

my "vibe bench" tells me to keep going and keep pulling this thread

it'd be great if you can share what are some of your vibe runs or tests, esp in the context of

prompt => .prose => agent execution performance
vs.
prompt => agent execution performance

i am curious cause both prompt and prose are inferred & executed by LLMs. prose definitely has more structure which might help. do you have examples you can share where translating to .prose helps. also, totally get it if you haven't run benchmarks yet & those are definitely expensive to run!