Remove temporary robots.txt intercept from Worker

[MarketDataApp]

Context

A stale robots.txt (User-agent: * Disallow: /) was cached on Cloudflare's edge via the Worker's cacheEverything: true. The origin (Pages) correctly returns 404, but the edge cache entry has a 7-day TTL (s-maxage=604800) and resisted all purge attempts (zone purge, Worker redeploy, Cache API delete).

A temporary intercept was added to worker/index.js that short-circuits robots.txt requests with a 404 before they hit the cache layer.

Action

After March 3, 2026, the stale cache entry will have expired. Remove the robots.txt intercept block from worker/index.js (lines that check url.pathname.endsWith('/robots.txt')) and redeploy the Worker. Verify that /docs/robots.txt returns 404 naturally from the Pages origin.

Files

• worker/index.js — remove the robots.txt early-return block