ReadStateService in zaino not syncing non-finalised state on testnet.

[idky137]

Usage mode

zainod (running the daemon)

Describe the bug

When running zainod on testnet, using the "state" backend to communitcate with zebrad, StateService::spawn() stalls on trying to sync the ReadStateService with the connected Zebrad.

Steps to reproduce

1. Run zebrad on testnet:
   1a. zebrad --config zainod/zebrad.toml start

2. Run zainod, connected to zebrad using the "state" backend (in separate terminal, while zebrad is running).
   2a. zainod generate-config
   2b. zainod start

Expected behavior

ReadStateService syncs with zebra and spawn continues.

Environment

Zaino: https://github.com/idky137/zaino/tree/zebrad_state_service_testnet_config

Zebra: v3.1.0

Configuration

Zebrad config:

# Default configuration for zebrad.
#
# This file can be used as a skeleton for custom configs.
#
# Unspecified fields use default values. Optional fields are Some(field) if the
# field is present and None if it is absent.
#
# This file is generated as an example using zebrad's current defaults.
# You should set only the config options you want to keep, and delete the rest.
# Only a subset of fields are present in the skeleton, since optional values
# whose default is None are omitted.
#
# The config format (including a complete list of sections and fields) is
# documented here:
# https://docs.rs/zebrad/latest/zebrad/config/struct.ZebradConfig.html
#
# CONFIGURATION SOURCES (in order of precedence, highest to lowest):
#
# 1. Environment variables with ZEBRA_ prefix (highest precedence)
#    - Format: ZEBRA_SECTION__KEY (double underscore for nested keys)
#    - Examples:
#      - ZEBRA_NETWORK__NETWORK=Testnet
#      - ZEBRA_RPC__LISTEN_ADDR=127.0.0.1:8232
#      - ZEBRA_STATE__CACHE_DIR=/path/to/cache
#      - ZEBRA_TRACING__FILTER=debug
#      - ZEBRA_METRICS__ENDPOINT_ADDR=0.0.0.0:9999
#
# 2. Configuration file (TOML format)
#    - At the path specified via -c flag, e.g. `zebrad -c myconfig.toml start`, or
#    - At the default path in the user's preference directory (platform-dependent, see below)
#
# 3. Hard-coded defaults (lowest precedence)
#
# The user's preference directory and the default path to the `zebrad` config are platform dependent,
# based on `dirs::preference_dir`, see https://docs.rs/dirs/latest/dirs/fn.preference_dir.html :
#
# | Platform | Value                                 | Example                                        |
# | -------- | ------------------------------------- | ---------------------------------------------- |
# | Linux    | `$XDG_CONFIG_HOME` or `$HOME/.config` | `/home/alice/.config/zebrad.toml`              |
# | macOS    | `$HOME/Library/Preferences`           | `/Users/Alice/Library/Preferences/zebrad.toml` |
# | Windows  | `{FOLDERID_RoamingAppData}`           | `C:\Users\Alice\AppData\Local\zebrad.toml`     |

[consensus]
checkpoint_sync = true

[health]
enforce_on_test_networks = false
min_connected_peers = 1
ready_max_blocks_behind = 2
ready_max_tip_age = "5m"

[mempool]
eviction_memory_time = "1h"
tx_cost_limit = 80000000

[metrics]

[mining]
internal_miner = false

[network]
cache_dir = true
crawl_new_peer_interval = "1m 1s"
initial_mainnet_peers = [
    "dnsseed.z.cash:8233",
    "dnsseed.str4d.xyz:8233",
    "mainnet.seeder.zfnd.org:8233",
]
initial_testnet_peers = [
    "dnsseed.testnet.z.cash:18233",
    "testnet.seeder.zfnd.org:18233",
]
listen_addr = "[::]:8233"
max_connections_per_ip = 1
network = "Testnet"
peerset_initial_target_size = 25

[rpc]
debug_force_finished_sync = false
enable_cookie_auth = false
max_response_body_size = 52428800
parallel_cpu_threads = 0
listen_addr = '127.0.0.1:18232'
indexer_listen_addr = '127.0.0.1:18230'

[state]
delete_old_database = true
ephemeral = false
should_backup_non_finalized_state = true

[sync]
checkpoint_verify_concurrency_limit = 1000
download_concurrency_limit = 50
full_verify_concurrency_limit = 20
parallel_cpu_threads = 0

[tracing]
buffer_limit = 128000
force_use_color = false
use_color = true
use_journald = false


Zainod config:

# Zaino Configuration
#
# Generated with `zainod generate-config`
#
# Configuration sources are layered (highest priority first):
#   1. Environment variables (prefix: ZAINO_)
#   2. TOML configuration file
#   3. Built-in defaults
#
# For detailed documentation, see:
#   https://github.com/zingolabs/zaino

backend = 'state'
zebra_db_path = '/home/idky137/.cache/zebra'
network = 'Testnet'

[grpc_settings]
listen_address = '127.0.0.1:8137'

[validator_settings]
validator_grpc_listen_address = '127.0.0.1:18230'
validator_jsonrpc_listen_address = '127.0.0.1:18232'
validator_user = 'xxxxxx'
validator_password = 'xxxxxx'

[service]
timeout = 30
channel_size = 32
[storage.cache]
capacity = 10000
shard_power = 4

[storage.database]
path = '/home/idky137/.cache/zaino'
size = 128

Logs

Zebrad:

...

2026-03-04T12:53:04.532358Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::sync::gossip: height=Height(3879438) request=AdvertiseBlock(block::Hash("0044416dc051ad856c2b23a7fca6842a9fccb808892df9376245d90721e8bed6")) log_msg="sending committed block broadcast"
2026-03-04T12:53:04.567179Z  INFO {zebrad="930fb7f" net="Test"}:{peer=Out("v4redacted:18233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=0048319b7c7ec817a611bd3154db594601365e48feae99d3698121715d63ca10}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(3879440)
2026-03-04T12:53:04.583456Z  INFO {zebrad="930fb7f" net="Test"}:{peer=Out("v4redacted:18233")}:msg_as_req{msg="inv"}:inbound:download_and_verify{hash=004566129affa3829e65f9fc5842c0999087a1bd07a9b69b587991930550f73d}: zebrad::components::inbound::downloads: downloaded and verified gossiped block height=Height(3879441)
2026-03-04T12:53:04.716902Z  INFO {zebrad="930fb7f" net="Test"}:sync:try_to_sync: zebrad::components::sync: exhausted prospective tip set
2026-03-04T12:53:04.716943Z  INFO {zebrad="930fb7f" net="Test"}:sync: zebrad::components::sync: waiting to restart sync timeout=67s state_tip=Some(Height(3879441))
2026-03-04T12:53:06.348603Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::mempool: resetting mempool: switched best chain, skipped blocks, or activated network upgrade tip_height=Height(3879441)
2026-03-04T12:53:06.348650Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::mempool: activating mempool: Zebra is close to the tip tip_height=Height(3879441)
2026-03-04T12:53:06.348657Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::mempool: re-verifying mempool transactions after a chain fork transactions=1
2026-03-04T12:53:06.548078Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::mempool::gossip: sending mempool transaction broadcast request=AdvertiseTransactionIds(1) changes=1
2026-03-04T12:53:11.566775Z  INFO {zebrad="930fb7f" net="Test"}: zebrad::components::sync::gossip: height=Height(3879441) request=AdvertiseBlock(block::Hash("004566129affa3829e65f9fc5842c0999087a1bd07a9b69b587991930550f73d")) log_msg="sending committed block broadcast"
2026-03-04T12:53:21.135703Z  INFO zebra_rpc::indexer::methods: failed to send non-finalized state change, dropping task error=SendError { .. }
2026-03-04T12:53:35.013505Z  INFO zebra_rpc::indexer::methods: failed to send non-finalized state change, dropping task error=SendError { .. }
2026-03-04T12:53:35.013581Z  INFO zebra_rpc::indexer::methods: failed to send non-finalized state change, dropping task error=SendError { .. }
2026-03-04T12:53:35.013651Z  INFO zebra_rpc::indexer::methods: failed to send non-finalized state change, dropping task error=SendError { .. }


Zainod:

...

2026-03-04T12:53:21.125468641Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:21.128258221Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:21.130404434Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:21.132917671Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:21.135320541Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:21.135509844Z  WARN h2::proto::streams::streams: locally-reset streams reached limit (1024)
2026-03-04T12:53:21.135579824Z  WARN sync: zebra_rpc::sync: failed to subscribe to non-finalized state changes err=Status { code: ResourceExhausted, message: "h2 protocol error: http2 error", source: Some(tonic::transport::Error(Transport, hyper::Error(Http2, Error { kind: GoAway(b"too_many_internal_resets", ENHANCE_YOUR_CALM, Library) }))) }
2026-03-04T12:53:21.43939792Z  INFO zaino_state::backends::state: got blockchain info!
2026-03-04T12:53:21.439778749Z  INFO zaino_state::backends::state: got tip!
2026-03-04T12:53:21.439807452Z  INFO zaino_state::backends::state:  - ReadStateService syncing with Zebra. Syncer chain height: 3879342, Validator chain height: 3879441
2026-03-04T12:53:22.44417935Z  INFO zaino_state::backends::state: got blockchain info!
2026-03-04T12:53:22.444521066Z  INFO zaino_state::backends::state: got tip!
2026-03-04T12:53:22.444532938Z  INFO zaino_state::backends::state:  - ReadStateService syncing with Zebra. Syncer chain height: 3879342, Validator chain height: 3879441
2026-03-04T12:53:23.447744642Z  INFO zaino_state::backends::state: got blockchain info!
2026-03-04T12:53:23.448087359Z  INFO zaino_state::backends::state: got tip!
2026-03-04T12:53:23.448117065Z  INFO zaino_state::backends::state:  - ReadStateService syncing with Zebra. Syncer chain height: 3879342, Validator chain height: 3879441
2026-03-04T12:53:24.452040279Z  INFO zaino_state::backends::state: got blockchain info!
2026-03-04T12:53:24.452276859Z  INFO zaino_state::backends::state: got tip!
2026-03-04T12:53:24.452289523Z  INFO zaino_state::backends::state:  - ReadStateService syncing with Zebra. Syncer chain height: 3879342, Validator chain height: 3879441
2026-03-04T12:53:25.456187686Z  INFO zaino_state::backends::state: got blockchain info!
2026-03-04T12:53:25.456470653Z  INFO zaino_state::backends::state: got tip!
2026-03-04T12:53:25.456495208Z  INFO zaino_state::backends::state:  - ReadStateService syncing with Zebra. Syncer chain height: 3879342, Validator chain height: 3879441
2026-03-04T12:53:26.14102612Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")
2026-03-04T12:53:26.144065535Z  WARN sync: zebra_rpc::sync: failed to commit block to non-finalized state error=NotReadyToBeCommitted hash=block::Hash("002b96ec125138c9b31d38f2c8c989c72b468296c83428027cb919c2bc9a690c")

Additional context

This is happening during StateService::spawn():

zaino/zaino-state/src/backends/state.rs

Line 235 in 7da1727

// Wait for ReadStateService to catch up to primary database:

The height the ReadStateService syncer is reporting is 99 blocks behind what the validator is reporting. The likely cause is that the ReadStateService in zainod is failing to sync the non-finalised state from the backing zebrad.

This is also related to code in zebra:
https://github.com/ZcashFoundation/zebra/blob/de9a52cbd646364c35b8e1b9db32c9f8af31160d/zebra-rpc/src/sync.rs#L288

https://github.com/ZcashFoundation/zebra/blob/de9a52cbd646364c35b8e1b9db32c9f8af31160d/zebra-rpc/src/sync.rs#L136