Improve LLM agent documentation browsing with intermediate loop

[exedev-shelley]

Author: philz42
Channel: #general
Link: https://discord.com/channels/1405685085923049482/1405686123908567141/1471167026914398354

---

Context

Discussion about how LLM agents browse documentation, particularly regarding the OpenClaw prompt and Shelley's browsing capabilities.

amolith: Most agents that read webpages best-effort convert to markdown using something like defuddle. Considering testing the prompt to see how Shelley uses the browser and compare approaches.

amolith: Even with console access, it would likely result in way more tokens than running the page through defuddle or similar.

Issue

The current approach to documentation browsing could be improved with an intermediate LLM loop to optimize token usage and efficiency.

Consider implementing a more token-efficient method for agents to browse and process documentation, potentially by:

• Incorporating a preprocessing step (markdown conversion via defuddle or similar)
• Adding an intermediate LLM loop to better handle the browse-extract-synthesize workflow
• Comparing token efficiency between direct DOM access vs. pre-converted markdown approaches

[philz]

“Browser ask (prompt)” is the question for a single turn subloop with only the browser tool