Add proxy validation tool #60
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
knjiang
force-pushed
the
add_validate_cli_tool_to_test_payloads_against_gateways
branch
from
52b1449 to
e51e53f
Compare
knjiang
force-pushed
the
add_validate_cli_tool_to_test_payloads_against_gateways
branch
from
e51e53f to
7482ae1
Compare
remh
approved these changes
Jan 16, 2026
remh
left a comment
remh
left a comment
There was a problem hiding this comment.
Nice stuff. I have a few tiny nitpicks, feel free to address the ones worth addressing and merge.
|
|
||
| ```bash | ||
| # Test with both OpenAI and Anthropic models through chat-completions format | ||
| pnpm validate --proxy-url http://localhost:8080 --models openai,anthropic |
There was a problem hiding this comment.
tiny nitpick but we should call the option provider instead of models
| | `--proxy-url <url>` | Proxy URL (required) | | ||
| | `--api-key <key>` | API key for gateway | | ||
| | `--format <formats>` | Formats to test: `chat-completions`, `responses`, `anthropic` | | ||
| | `--models <models>` | Model providers: `openai`, `anthropic` | |
There was a problem hiding this comment.
it supports bedrock and google as well no? let's list them
| }) | ||
| .join("\\."); | ||
|
|
||
| const regex = new RegExp(`^${regexPattern}$`); |
There was a problem hiding this comment.
likely not needed now but we could cache the regex if things start getting too slow, i doubt that's the case now though
| @@ -0,0 +1,202 @@ | |||
| // JSON comparison utilities with field ignore support | |||
There was a problem hiding this comment.
let's generate a couple of unit tests for those functions?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TL;DR
Added a validation tool to compare LLM proxy responses against captured snapshots, ensuring API compatibility.
Testing - ran manually:
Why make this change?
This validation tool enables systematic testing of LLM proxies and gateways against known-good responses. It helps ensure that proxies maintain API compatibility with original providers.