fix: map interactive UI query params to request headers for GET-based toggles

[VarunKumar-05]

🐛 Problem Statement

The interactive code builder in the Jina Reader API has a critical bug: UI toggles don't affect the actual API request behavior because they're sent as query parameters but the server only recognizes header-based options.

Evidence from Issue #1224

User Report:

"Unfortunately the documentation is a bit lacking. The interactive code builder is a bit buggy (the API request body doesn't change when you toggle the settings), and it's difficult to get a clear sense of the response payload body shape."

Root Cause (HAR Analysis):
The interactive builder issues simple GET requests with query parameters (e.g., ?respond_with=markdown&with_generated_alt=true), but the server-side code only checks request headers (e.g., x-respond-with, x-with-generated-alt). This causes all UI toggles to be silently ignored.

Example GET Request from UI (non-functional):

GET /https://example.com/?respond_with=markdown&with_generated_alt=true HTTP/1.1

The query params are never mapped to headers, so downstream code sees no options set.

---

✅ Solution

This PR implements a server-side compatibility layer that maps well-known query parameters to their corresponding request headers. This is:

• Low-risk: Existing header-based behavior is completely unchanged
• Backward-compatible: Old clients using headers continue to work
• Minimal: Single middleware addition; no business logic changes

Changes Made

File: src/services/registry.ts

1. Added __mapQueryToHeaders middleware method that translates UI-friendly query params into request headers:

   • ?respond_with=markdown → x-respond-with: markdown
   • ?with_generated_alt=true → x-with-generated-alt: true
   • ?no_cache=true → x-no-cache: true
   • ?wait_for_selector=#content → x-wait-for-selector: #content
   • ?target_selector=.main → x-target-selector: .main
   • ?proxy=https://proxy:3128 → x-proxy-url: https://proxy:3128
   • ?cache_tolerance=120 → x-cache-tolerance: 120
   • ?timeout=30 → x-timeout: 30
   • ?accept=application/json → Accept: application/json
   • Plus support for camelCase variants (e.g., respondWith, withGeneratedAlt)

2. Integrated middleware into koaMiddlewares array (runs after CORS, before body parser)

   • Ensures query params are converted before options are parsed

File: src/api/crawler.ts (Debug support)

3. Added temporary debug logging to the crawl() handler:

   this.logger.debug('Incoming headers', { headers: ctx.request.headers });

   This confirms headers are correctly mapped for testing.

---

🧪 Testing Steps

Test 1: Query Params → Headers Mapping (via curl)

Before PR: Headers would be empty/missing
After PR: Headers populated from query params

# Test 1a: Markdown output via query param
curl -v "http://localhost:8080/https://example.com/?respond_with=markdown&with_generated_alt=true"

# Expected: Server treats as if headers were set:
#   x-respond-with: markdown
#   x-with-generated-alt: true

# Test 1b: JSON output via query param
curl -v "http://localhost:8080/https://example.com/?accept=application/json"

# Expected: Response Content-Type: application/json

# Test 1c: Control test (traditional header-based)
curl -v -H "Accept: application/json" "http://localhost:8080/https://example.com/"

# Expected: Identical behavior to Test 1b

Test 2: Verify Debug Logs

Watch server logs during any request:

[debug] Incoming headers { headers: { 'x-respond-with': 'markdown', 'x-with-generated-alt': 'true', ... } }

Test 3: Interactive Builder Integration

1. Open the interactive builder
2. Toggle "Markdown Output" → Submit GET request
3. Observe URL includes ?respond_with=markdown
4. Confirm response format matches toggle (should now work)

Test 4: Backward Compatibility

All existing header-based clients must continue working unchanged:

# Old clients using headers (must still work)
curl -H "x-respond-with: markdown" "http://localhost:8080/https://example.com/"

# New clients using query params (now works)
curl "http://localhost:8080/https://example.com/?respond_with=markdown"

# Mixed (query params override nothing; headers take precedence if both set)
curl -H "x-respond-with: html" "http://localhost:8080/https://example.com/?respond_with=markdown"

---

🔒 Safety & Design Rationale

Why This Approach?

1. Minimal code change → Lower risk of regressions
2. Middleware-based → Transparent to business logic (snapshot-formatter, LmControl, searcher, crawler)
3. Early in pipeline → Query params converted before any options parsing
4. Error handling → Mapping errors logged but don't fail requests

Error Handling

If query param mapping fails (e.g., malformed value), the error is logged and the request continues:

try {
    // ... mapping logic ...
} catch (err) {
    this.logger.warn(`Failed to map query params to headers: ${err}`, { err });
}
return next();

Performance Impact

Negligible—mapping runs once per request, only iterates over ~10 known query param keys, uses simple string operations.

---

📋 Checklist

• ✅ Code changes applied to src/services/registry.ts
• ✅ Debug logging added to src/api/crawler.ts
• ✅ TypeScript compilation passes (no new errors)
• ✅ Backward compatible (existing header clients unaffected)
• ✅ Error handling included
• ✅ Minimal code footprint

---

🎯 Impact

Before This PR

• Interactive builder toggles are silently ignored
• Users confused by non-functional UI
• Must manually craft curl commands with headers

After This PR

• Interactive builder toggles now work correctly
• GET requests with query params behave identically to header requests
• Better developer experience
• Closes issue Improve documentation / provide llms.txt file #1224

---

🔗 Related Issues

• Resolves Improve documentation / provide llms.txt file #1224 (Interactive builder UI toggles not working)
• HAR evidence: See issue comments for actual request/response trace

---

💡 Future Recommendations (Optional)

Ideal Long-Term Fix

Update the interactive builder frontend to:

1. Preferred: Use fetch() with proper request headers (clearer semantics)
2. Alternative: Continue using query params (server-side mapping now supports this)

This ensures all clients—whether using headers or query params—have consistent behavior.

---

Testing Environment:

• Node.js: ≥18
• Dependencies: npm ci
• Build: npm run build