[?Wed 19 Mar] Testing Your Prompts Like You Test Your Code

[alistairjcbrown]

Testing Your Prompts Like You Test Your Code

The idea of treating prompts as something that should have regression tests is still pretty novel for most developers. The angle of "I changed a prompt and it broke 15 matches I didn't expect" is very relatable to anyone working with LLMs in production.

Once you've figured out the approach, there's a lot of interesting stuff to cover: what does a test case even look like for a prompt, how do you define pass/fail when LLM output is non-deterministic, comparing runs over time, and whether you're testing the prompt or the model (especially relevant when models get updated under you).