Observability Data Generator (TypeScript/Node.js)

Small two‑service TypeScript/Node.js app to generate OpenTelemetry traces, metrics, and logs across REST, Kafka (request‑reply and fire‑and‑forget), gRPC, PostgreSQL, and Elasticsearch.

Modules

• Gateway service [HTTP 9980]
• Processor service [HTTP 9981, gRPC 9090]

Quick start (Docker)

./build.sh
docker-compose up
# Stop
docker-compose down
# Clean volumes
docker-compose down -v

Telemetry flow (OTLP over HTTP)

This app produces telemetry only; it does not store or display traces/metrics/logs. It relies on an OpenTelemetry Collector (or a vendor endpoint) to receive and forward/export that data.

• Transport: OTLP over HTTP (http/protobuf), not OTLP/gRPC. The Node.js SDK uses OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.

Default wiring (Docker Compose):

• gateway-service and processor-service export OTLP/HTTP to otel-test-collector:5318 inside the compose network.
• The local test collector writes a copy to otel-logs/telemetry.log and proxies everything to the host at http://host.docker.internal:4318.
• You can change the proxy target in otel-collector-config.yaml (exporter otlphttp/production).

Send data to your own collector/vendor:

• Option A (direct): set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, and OTEL_EXPORTER_OTLP_LOGS_ENDPOINT on each service to your collector URLs (HTTP/4318) and keep OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf.
• Option B (proxy via local collector — default): keep services pointed at otel-test-collector:5318 and change exporters.otlphttp/production.endpoint in otel-collector-config.yaml to your vendor ingest URL.
• Option C (dev‑only logs): keep the default and tail otel-logs/telemetry.log for quick local verification.

Quick verification:

docker-compose logs otel-test-collector
# and
tail -f otel-logs/telemetry.log

Ports

• Gateway HTTP: 9980
• Processor HTTP: 9981
• Processor gRPC: 9090
• Kafka broker: 9092
• PostgreSQL: 5432
• Elasticsearch: 9200 (HTTP), 9300 (transport)
• OTLP HTTP receiver on the local test collector (inside Docker): 5318
• Default OTLP HTTP endpoint on your host/vendor collector: 4318

Endpoints (gateway 9980)

• /rest-chain — REST call to processor
• /kafka-rr — Kafka request‑reply
• /kafka-ff — Kafka fire‑and‑forget
• /db-ops — PostgreSQL write + read
• /es-ops — Elasticsearch index + get
• /grpc — gRPC call to processor

Configuration (minimal)

• KAFKA_BROKERS — e.g. kafka:29092 in Docker or localhost:9092 locally
• PROCESSOR_URL — e.g. http://processor-service:9981
• PROCESSOR_GRPC_URL — e.g. processor-service:9090
• POSTGRES_* — POSTGRES_HOST, POSTGRES_PORT, POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD
• ELASTICSEARCH_NODE — e.g. http://elasticsearch:9200
• Optional OTEL: OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_METRICS_ENDPOINT, OTEL_EXPORTER_OTLP_LOGS_ENDPOINT, OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, OTEL_RESOURCE_ATTRIBUTES

Kafka topics:

• gateway-processor-request — request‑reply requests
• gateway-processor-reply — request‑reply responses
• gateway-processor-fire-forget — fire‑and‑forget events

Run locally (dev)

# Gateway
cd gateway-service
npm install
npm run dev

# In another shell — Processor
cd processor-service
npm install
npm run dev

Ensure PostgreSQL, Kafka, and Elasticsearch are available or adjust env vars.

OpenTelemetry notes

• Auto‑instrumentation via @opentelemetry/sdk-node and friends; OTLP/HTTP exporters for traces, metrics, and logs.
• Elasticsearch is instrumented through @elastic/opentelemetry-node (works with @elastic/elasticsearch v8+). The community @opentelemetry/instrumentation-elasticsearch targets the deprecated elasticsearch v16.x and is not used here.

Troubleshooting (quick)

• Ports free: 9980, 9981, 9090, 9092, 5432, 9200, 5318
• Logs: docker-compose logs -f
• Kafka topics: use console consumer tools
• ES health: http://localhost:9200
• gRPC test: grpcurl -plaintext -d '{"message":"test"}' localhost:9090 processor.ProcessorService/ProcessData
• OTEL export: verify docker-compose logs otel-test-collector

Verification

docker-compose logs otel-test-collector

# or

tail -f otel-logs/telemetry.log