[Feature Request]: Expose Token Usage for Embeddings and Query Generation in Adaptive Crawling

[jyumay]

What needs to be done?

Exactly as the title says. Just like the llm extraction strategy exposes show_usage(), token usage should be exposed from the adaptive crawler when using the embedding strategy (and really anywhere else they aren't being exposed right now).

What problem does this solve?

Gives the dev a way to track token usage and costs in code rather than having to check multiple API dashboards.

Target users/beneficiaries

No response

Current alternatives/workarounds

None, except expanding the AdaptiveCrawler class and its helpers yourself.

Proposed approach

Surface via the returned CrawlState as usage. I suppose the dictionary should be normalized to the open ai format?

The dict should show separate usage for both query generation and embedding. This is important if someone wants to calculate costs with this usage data later.

[jyumay]

adaptive_crawler_with_usage.py
example_usage.py

Created a class expansion for anyone who wants an immediate fix. Also fixes this issue: https://github.com/unclecode/crawl4ai/issues/1682