Skip to content

Use protobuf and msgio for sync#4

Open
cmsegal wants to merge 2 commits intomainfrom
use-protobuf-for-sync
Open

Use protobuf and msgio for sync#4
cmsegal wants to merge 2 commits intomainfrom
use-protobuf-for-sync

Conversation

@cmsegal
Copy link
Collaborator

@cmsegal cmsegal commented Aug 24, 2025

No description provided.

@cmsegal cmsegal force-pushed the use-protobuf-for-sync branch 2 times, most recently from d2719b1 to 8f92163 Compare August 25, 2025 20:25
@pcsegal pcsegal force-pushed the use-protobuf-for-sync branch from 8f92163 to 818d7e0 Compare September 22, 2025 17:57
@pcsegal pcsegal force-pushed the use-protobuf-for-sync branch from 818d7e0 to cd9d316 Compare September 30, 2025 10:11
Copy link
Collaborator

@mgazza mgazza left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent improvement to use Protocol Buffers for sync messages. This provides better performance, type safety, and forward/backward compatibility compared to JSON.

Key improvements:

  • Binary serialization is more efficient than JSON
  • Type-safe enums for message types (SYNC_REQUEST/SYNC_RESPONSE)
  • Uses msgio for proper length-delimited framing
  • Separate read/write deadlines for better stream control
  • Clean conversion functions between internal and protobuf types

The reduction of sync batch size from 100k to 10k is also sensible for better memory management.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants