Fix source insertion metadata alignment

[marcelsamyn]

Summary

• ensure new source rows are matched back to their original inputs before post-processing
• prevent inline/blob payload updates from using metadata belonging to a different memory when conflicts occur

Testing

• pnpm lint

---

https://chatgpt.com/codex/tasks/task_e_68cf96b84730832f9ee1be18cace2dc6

[gemini-code-assist]

Summary of Changes

Hello @iamarcel, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical issue where the metadata used for post-processing newly inserted source rows could become misaligned with their original input data. By implementing a robust lookup mechanism, the changes ensure that each inserted source row is correctly associated with its corresponding initial input, thereby preventing incorrect payload updates and maintaining data integrity during the source creation process.

Highlights

• Correct Input-Output Alignment: Introduced a lookup mechanism to accurately match newly inserted source rows with their original input data, ensuring correct metadata is used for post-processing.
• Robust Payload Handling: Modified payload processing to prevent misassociation of metadata by explicitly linking inserted database rows back to their corresponding initial input objects using a unique key.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly addresses a bug where source metadata could be misaligned after bulk insertion with conflicts. By creating a lookup map from the original inputs, you ensure that the post-processing logic correctly associates the inserted database rows with their corresponding input data. This is a solid fix.

I have one suggestion to make the implementation more robust by improving how the lookup key is generated to prevent potential collisions.

[gemini-code-assist]

The current method of creating a lookup key by concatenating userId, type, and externalId with a colon separator is not robust. If userId or externalId were to contain a colon, it could lead to key collisions and incorrect data association. For example, userId: 'user:conversation', type: 'document', externalId: 'ext1' would produce the same key as userId: 'user', type: 'conversation', externalId: 'document:ext1'. This would cause one of the inputs to be overwritten in the lookup map, and could lead to incorrect metadata being associated with an inserted source.

A safer approach is to use a method that guarantees uniqueness, such as JSON stringifying an array of the key components. This will prevent any potential key collisions.

Suggested change

	const makeLookupKey = (
	userId: string,
	type: SourceType,
	externalId: string,
	) => `${userId}:${type}:${externalId}`;
	const makeLookupKey = (
	userId: string,
	type: SourceType,
	externalId: string,
	) => JSON.stringify([userId, type, externalId]);