Clickhouse exporter logs table on error

[mheffner]

We should include metadata about the connection and table when logging this error to help identify the source:

Failed to export to Clickhouse: ...

[garysassano]

I'd like to expand on this. We should emit a JSON string in the body field so HyperDX can parse it and allow filtering by the fields.

Right now the error logging dumps everything into an unstructured string in the body field. If we emit JSON instead, HyperDX will automatically parse it and make all the fields queryable.

Current Body Field (Unstructured)

Failed to export to Clickhouse: Http(Parts { status: 500, version: HTTP/1.1, headers: {"date": "Sun, 27 Jul 2025 23:38:40 GMT", "connection": "Keep-Alive", "content-type": "text/plain; charset=UTF-8", "access-control-expose-headers": "X-ClickHouse-Query-Id,X-ClickHouse-Summary,X-ClickHouse-Server-Display-Name,X-ClickHouse-Format,X-ClickHouse-Timezone,X-ClickHouse-Exception-Code", "x-clickhouse-server-display-name": "clickhouse-cloud", "transfer-encoding": "chunked", "x-clickhouse-query-id": "83dd50f8-6d22-4544-afef-504482e68357", "x-clickhouse-timezone": "UTC", "x-clickhouse-exception-code": "497", "keep-alive": "timeout=10, max=9999", "x-clickhouse-summary": "{\"read_rows\":\"0\",\"read_bytes\":\"0\",\"written_rows\":\"0\",\"written_bytes\":\"0\",\"total_rows_to_read\":\"0\",\"result_rows\":\"0\",\"result_bytes\":\"0\",\"elapsed_ns\":\"3685543\"}"} }, None)

What I'm Thinking For JSON Body

Instead of a plain string, emit JSON in the body field:

{
  "message": "Failed to export to ClickHouse",
  "error_type": "clickhouse_export_failed",
  "clickhouse": {
    "operation": "export",
    "endpoint": "https://your-cluster.clickhouse.cloud:8443",
    "database": "your_database", 
    "table": "your_table",
    "user": "your_username",
    "exception_code": 497,
    "query_id": "83dd50f8-6d22-4544-afef-504482e68357",
    "server_name": "clickhouse-cloud",
    "timezone": "UTC",
    "summary": {
      "read_rows": 0,
      "read_bytes": 0,
      "written_rows": 0,
      "written_bytes": 0,
      "elapsed_ns": 3685543
    }
  },
  "http": {
    "status_code": 500,
    "version": "HTTP/1.1"
  },
  "raw_error": "Http(Parts { status: 500, ... })"
}

Why I Think JSON in Body Would Help

1. HyperDX Auto-Parsing: HyperDX will automatically parse JSON and make fields filterable
2. Easy Querying: We could filter by clickhouse.exception_code = 497 or http.status_code >= 500
3. Better Alerting: Set up alerts on specific error types like permission failures vs server errors
4. Analytics: Aggregate errors by database, table, or user to identify patterns
5. Faster Debugging: Quickly find all failures for a specific query ID or server

Examples of How We Could Filter in HyperDX

With JSON in the body field, HyperDX would let us filter like:

// Find all permission errors
clickhouse.exception_code = 497

// Find errors by specific table
clickhouse.table = "users"

// Track slow operations 
clickhouse.summary.elapsed_ns > 1000000000

// Monitor specific server issues
clickhouse.server_name = "clickhouse-cloud" AND http.status_code >= 500

This way we get all the connection context and ClickHouse metadata in a queryable format without changing how OpenTelemetry logging works - just emit JSON instead of plain text in the body field.

[rjenkins]

re: JSON body sounds like a good suggestion to me and the above benefits listed are quite nice, ty @garysassano . Probably best to make it configurable for other users who want plain text. On that topic, iirc the rust tracing library that we're using for logging has multiple options for "subscribers", one of which is a JSON subscriber, so theoretically it should be quite easy to add (if it isn't already "there" and just needs to be exposed and toggled on with a cli arg).

To enable JSON output, you typically add the json feature flag to your tracing-subscriber dependency in Cargo.toml and then configure the fmt::format::Json formatter in your subscriber setup.