[Feature] Feature: ActionTrace System for Unity AI-User Interaction Tracking

[whatevertogo]

Description:
Implement an ActionTrace System for Unity that logs AI and user operations, enabling selective and prioritized access to recent changes. The system should provide the following capabilities:

Priority‑Based Filtering: Assign an importance level to each action, allowing the AI to concentrate on the most relevant recent operations.

Configurable Event Retrieval: Retrieve a customizable number of recent events, avoiding the overhead of processing the entire history.

Controlled Undo Support: Revert changes tracked by the system, with awareness that undo operations themselves carry inherent risk (recently as they cannot be undone).

Annotated Actions: Attach notes or comments to any action, facilitating future multi‑agent collaboration and contextual knowledge sharing.

Goals:

Enhance AI comprehension of ongoing Unity edits without requiring full‑scene or full‑history scans.

Enable efficient, targeted event tracking and retrieval.

Support collaborative annotation for multi‑agent workflows.

Extended Potential:
An ActionTrace System can also serve as a foundation for advanced features such as:

Machine‑learning‑based behavior analysis

Multi‑agent workflow orchestration

Automated summarization of recent activities

[whatevertogo]

ActionTrace can indirectly track the usage of MCP tools, but I'm wondering whether to actively track it. However, I don't want to deduplicate the events.

[Scriptwonder]

ActionTrace can indirectly track the usage of MCP tools, but I'm wondering whether to actively track it. However, I don't want to deduplicate the events.

Interesting feature! MCP Log Listener can be a way to track the usage, and this is for the user to view what past behavior is called, not for the LLM comprehension, right?

I am quite interested in how you define priority of multiple actions in this case. Can you elaborate a little bit on that? Also, you mentioned it could enhance AI comprehension. How does that work? Did you feed extra information to the LLM through a customized tool? I am just thinking about the tradeoff between tokens and gaining extra information here, and does it actually help the comprehension with some useful tests.

[whatevertogo]

ActionTrace is not designed as a traditional usage log. Its purpose is not to replay history for users or to analyze the LLM itself. Instead, it acts as an editor-context extraction layer specifically for LLM consumption. The focus is not on what was recorded, but on which recent changes are most relevant at the current moment.

For prioritization, the system uses staged filtering rather than global ranking.
First, the query is hard-limited to a recent time window. Only actions that occurred within this window are eligible; older events are excluded entirely. This guarantees that all surfaced information is strongly correlated with the user’s current editing intent.

Within that time window, a coarse-grained importance filter is applied. At this stage, the goal is not precise ordering but noise reduction. For example, introducing new code, components, or structural changes is treated as higher importance than modifying inspector properties, while transient actions such as object selection or view changes are usually filtered out. In the experimental phase, these rules are intentionally simple and conservative.

Regarding improved AI understanding, the system does not attempt to give the LLM more raw data. Instead, it provides a semantic-based summary of recent editor activity, distilled from the filtered action set. This summary is intended to capture meaningful intent and change, rather than low-level editor events.

From a token-cost perspective, this approach is more controllable than injecting large scene snapshots or full state descriptions. The tool is only invoked when explicitly called by the user, and currently only in cases where the task genuinely depends on recent editor context. This keeps the cost bounded and the behavior predictable.

Overall, the trade-off is not more tokens for more data, but fewer, higher-signal tokens for a more stable understanding of the user’s current intent. That is the core value of ActionTrace.

[whatevertogo]

ActionTrace is a callable intent-capturing layer that transforms recent editing activity into a clear, semantic narrative, reflecting the user's current focus and likely next steps. This semantic output is not only consumable by LLMs but also serves as a higher-level capability, providing contextual empowerment for other tools or systems.

[whatevertogo]

Please note that this is currently in the experimental phase. This weekend, I will dive headfirst into development and aim to produce an initial demo for the community to discuss and provide feedback on.

[whatevertogo]

[whatevertogo]

It‘s worked