Add production Worker routes, cache headers, and SEO fixes

[MarketDataApp]

Summary

• Add production /docs/* routes to Worker
• Remove tracked robots.txt (gitignored but was still committed)
• Add noIndex meta tag to staging builds to preserve crawl budget
• Add aggressive cache headers for static assets via _headers file
• Enable edge caching for Worker subrequests

Changes

• worker/wrangler.toml — added /docs and /docs/* production routes
• worker/index.js — added cf: { cacheEverything: true } to subrequests
• .github/workflows/deploy-docs.yml — generate _headers file with cache rules for assets/images
• docusaurus.config.js — added noIndex: true for non-production builds
• static/robots.txt — removed from git tracking