v0.0.7

Changelog

• 7b7578f Add NotifySplitsFinished API
• 9ca8769 Add SourceSplitter Start/Close methods
• 4b80112 Add more context to source runner error
• 4abc754 Add test helper to get assignments
• 8b8507a Adjust checkpointing contract to include SourceSplitter
• 7f0a188 Close dkv databases in test
• 8fc2a1c Combine SourceSplitter Start and LoadCheckpoint
• 454d3e1 Fix panic possible when closing ReadSourceChannel
• 7ec91de Have Kinesis client return shard type
• a367402 Improve test error
• a662825 Keep track of lastSeenShardID for Kinesis splitter
• 38874e8 Keep track of the HashKeyRanges of assigned shards
• 095883a Make AssignSplits only adds new splits
• ca0caa5 Make all split discovery async
• 896c0a4 Make stateUpdates a more general purpose queue
• de747eb Mark shards finished when read to end
• f62eb39 Pin reduction-protocol to head
• 955b34e Process parent shards before children
• 2f9524c Remove client wrapper and add test-only wrapper
• 1c09765 Repurpose checkpoint events chan for job errors
• 8cbde70 Split SR startup into deploy & assign splits
• cde1f9c Uniformly assign kinesis shards to runners
• bf7ed45 Use latest reduction-go version

---

See the documentation at reduction.dev/docs.

v0.0.6

Changelog

• abdd5c8 Add Kafka SinkWriter
• 5596e59 Add test for Kafka checkpointing
• f797d4c Include partition in kafkapb.Record
• a065dcc Send more fields of Kafka records
• 361afea Start Kafka connector

---

See the documentation at reduction.dev/docs.

v0.0.5

Changelog

• 7478dca Add Dockerfile
• b4fbd18 Add ECS task session script for debugging
• e0a3bdc Add NewReadSourceChannel tests
• d8bb354 Add S3 object storage for checkpoints
• c6070ef Add contract tests for FileStore
• 5180de7 Add curl to Dockerfile
• 8665549 Add err message to retry err logging
• aadc849 Add source reader backoff
• 203956a Add terminal source errors
• c3a1f94 Add test for checkpoints with 2 workers
• 9f23804 Add test workflow
• 7e9c9f9 Add tests for kinesis source reader errs
• 06d3c39 Align release and test workflows
• c625184 Always unmarshall config with params
• e5f16fb Avoid clone copy
• ea7ba93 Avoid setting AWS region and credentials
• f988685 Bring fileu.ReadFile into locations package
• 615e9f1 Build binaries outside of Docker
• 46df5ca Choose storage location type based on path
• 40b64f3 Consolidate EndOfInput checking
• a6dd721 Correctly resolve oneof
• 2f33145 Disable CGO for Docker build
• b48f67c Don't make source readers block on ReadEvents
• 5e51bbf Fix kinesis fake data race
• 3c08651 Give S3FileSystem a prefix
• 7d1679f Handle expired shard iterators
• 555a616 Have registry create assembly
• b6ed613 Initialize DKV s3 filesystem from URI
• d38049b Keep the source reading synchronous
• a01085c Load checkpoint only during job initialization
• 1c0ba8f Make it easier to change interceptor log levels
• 12bbef6 Make register/deregister debug level messages
• b39e373 Merge storage types into locations package
• b62ba54 Move dynamic config to protobuf
• 10c3bab Move guidelines to copilot instrutions
• 0534837 Move s3 cost to objstore package
• d16ef9f Only log "check registry" when changed
• 2713e43 Post process json template params
• d442c46 Read config from s3 URL
• ac39ac3 Refactor ResolveVars
• 1ca73f3 Remove default case from processEvents
• 06b6812 Remove duplicate import
• 225f094 Remove keySpace member from job
• 94493b3 Remove logger dependency from Assembly
• f6db94b Remove registry ownership from assembly
• bfb189c Remove shardList
• bd93ba5 Remove some unnecessary logging
• 9ecc7c2 Resolve a worker host for registration
• a106825 Retry DNS lookup errors
• 7ca44d2 Run gopls modernize
• c286f1d Simplify source reading channel
• 8b94063 Split connector types into different files
• f54114b Store the sequence number (not iterator) as cursor
• bde5ca2 Update to latest version of reduction protocol
• 26da3d9 Upgrade reduction-go dependency
• b17ef28 Use HeadObject S3 API for existence check
• 21d8cd6 Use latest reduction-go version
• 5e6378d Use protobuf files from Go import
• 7cb25f8 Validate the presence of kinesis stream arn

---

See the documentation at reduction.dev/docs.

v0.0.4

Changelog

• f1eea70 Add missing arg in benchmark
• 98e0d66 Remove checkpoint validation panic

---

See the documentation at reduction.dev/docs.

v0.0.3

Changelog

• e0a0364 Add async recovery test
• d299ae8 Add benchmark for table get requests
• fba4379 Add concept of WAL being full
• 52366f8 Adding NeedsTable
• d64e56d Delete shared SST tables
• c0c4a8c Document and enforce WAL concurrency contract
• 8b01a01 Don't return deleted entries from LL Scan
• be0e210 Don't timeout on long WAL replay
• 3e5800f Fix EOF bug for deleted entry not matching prefix
• aba444b Fix bug deleting local files never read or written
• 56bd61d Fix compaction table ref dropping
• 4fd24cd Fix data race for j.assembly
• 1959f1e Fix deleted entry EOF bug
• 08e85a8 Fix issue where WAL could grow indefinitely
• 543aab7 Fix local filesystem delete bug
• 6e82c76 Fix some more data races
• e86223a Improve logging when operator returns error
• f4d7522 Include host for operators in StartOperatorRequest
• 0b08174 Include the LastSeqNum in checkpoint
• b469f0f Modernize benchmark
• 6367713 Only replay WAL entries in the db Partition
• 933dede Only return unique key states per batch
• 9f3232c Remove job's checkpoint files
• 29792d2 Remove obsolete checkpoints
• 03d30f1 Remove ref counting code
• 4219170 Remove unused errChan from operator factory
• ac2eb43 Retry on 503 status forever
• ee9c285 Save the entire table URI in checkpoints
• d51efe8 Show help for dev command without arg
• b5a732e Split dkv_test into two files
• e2c3b77 Use CleanupAdd to rm files instead of ref counting
• 97ab2ad WIP adding deletion policy

---

See the documentation at reduction.dev/docs.

v0.0.2

Changelog

• 9f207d0 Be more consistent with config factories
• f577997 Have SummingHandler use public reduction-go API
• 45a21ed Minor updates to work with TypeScipt SDK
• 41184f6 Remove unnecessary brand methods
• 5557499 Remove unused ParseSourceConfig funcs
• 499af90 Remove unused members in config marshal
• 96160fa Replace internal config with jobconfigpb
• 15483d1 Tidy dependecies
• fc5af87 Update README.md
• 6ea5623 Update for reduction-go API change
• 3f92803 Update tests for v0.0.3 of Go SDK
• f3d6aec Use new rxnsvr package
• 00f404a Use public API for CountInWindowHandler
• bfda18e Use public API for sliding window handler
• 724223b Use the pb JSON config formay

---

See the documentation at reduction.dev/docs.

v0.0.1

Changelog

• 5c0e184 Actually send batches of events to the handler
• fbfd32e Add SourceReaderFactory param to SourceRunner
• f476458 Add script to tag releases
• 8c2c0c5 Add some request metrics
• 49c1403 Add stdio source
• 2eef03a Add testrun command
• e1040db Adjust to new reduction-go API
• 36d3f27 Allow table_test to pass with run -count > 1
• e080486 Avoid redundant timers in batching
• 7828870 Batch KeyEvent requests to handler
• 78d9482 Batch events to handler
• 0d4e6f0 Buffer output channels
• 219946d Centralize source parsing to allow defaults
• 18c723b Configure params for batchers
• 1bbe9a6 Ensure order of key event batch processing
• 7e14159 Extract processEventBatch
• 32877a3 Fix deadlock when waiting on SR alignment
• 0970fbc Fix use of memory:// scheme
• c9d0822 Flush batched events on SR source complete
• 8dfdac0 Hoist batching package out of rpc
• b0b03ea Make EventBatcher generic
• 40fb991 Make snapshot store safe to call concurrently
• 61e9bb8 Move ./bin files to ./vendor so gopls will ignore
• 8a045e6 Move factories out of connect handlers
• ad1c42a Move key event batching out of client
• aed31dd Reduction proof of concept
• 19688c3 Release without --snapshot
• 28a2fb6 Remove extraneous channel
• cd75d8a Remove local module replace directives
• b8ee7e6 Remove original async batching code
• 49d1e56 Remove unused SourceRunner param
• b56755b Rename SinkEvent to SinkRecord
• 7eb8be0 Rename some connect adapter types
• 5955077 Send KeyedEvent to OnEvent
• f976a7d Send kinesispb.Records over the wire to client
• 7e15f75 Send testing.T to the workertest server
• 6446223 Send watermark to handler
• fcbef30 Setup release workflow
• ee67aa5 Stop using rxn.Handler and use OperatorHandler
• ff2ee69 Switch to batching handle event endpoint
• 3159c56 Switch to handler batch endpoint for keying events
• d165f33 Sync workspace
• c210944 Update e2e tests for new state pattern
• fdc9d26 Update for reduction-handler rename
• 190685e Update for removal of job.Marshal method
• e7f53c8 Update for some go client changes
• 99237b4 Use a pointer ref instead of counter
• 68deb9d Use batcher in op client
• 4f06d21 Use context to stop batcher
• 4e06553 Use factory functions to configure clients
• 414e39a Use handler batch event endpoint
• f58540f Use mod=readonly for release
• 2ce5d97 Use new named connector packages
• 74f31c3 Use protobuf oneof over byte prefixes for testrun
• 9fe189d Use sync batcher for operator client

---

See the documentation at reduction.dev/docs.