AWS Lambda Rotel + Python + Clickhouse

This AWS Lambda function showcases the integration of the Rotel Lambda Extension with Clickhouse to store OpenTelemetry traces and logs. Application traces and Lambda output logs are sent directly to Clickhouse from AWS Lambda, so you can skip Cloudwatch entirely. This example leverages the new Clickhouse JSON data type to store key/value telemetry attributes efficiently.

The result is that you can query large amounts of trace and log data in Clickhouse:

For the purpose of demonstrating multiple exporters, metrics are dropped by sending them to the blackhole exporter. The Clickhouse exporter also supports metrics, so you could also export metrics to Clickhouse.

Functionality

The function accepts a US zip code as input and returns relevant geospatial data in JSON format.

Integration

To see how this integration works, check out the following:

• Makefile: for how the Rotel Lambda layer is created
• rotel-clickhouse.env: configuration for multiple exporters (Clickhouse + Blackhole)
• location.py: Python OpenTelemetry instrumentation

Migration

To create the tables required for this example, follow these instructions located here. Make sure to specify --enable-json to enable the new JSON data type when creating the tables.

Deployment Instructions

Note: Requires the AWS CLI to be installed.

Bundle function

make bundle

Deploy function

Required environment variables:

• AWS_DEFAULT_REGION: (defaults to us-east-1)
• AWS_ROLE_ARN: (Lambda execution role)
• CLICKHOUSE_ENDPOINT: Endpoint of your Clickhouse cluster/service (eg. https://xxxxxx.clickhouse.cloud:8443)
• CLICKHOUSE_PASSWORD: Password for Clickhouse, the user defaults to default
• AWS CLI credentials: Credentials required for executing the CLI and creating a function

make deploy

Invoke function

Required environment variables:

• ZIP_CODE: zip code to lookup (defaults to 90210)
• AWS CLI credentials: Credentials required for executing the CLI and invoking a function

make invoke

Delete function

If you need to change the configuration, you'll need to delete the function before recreating it.

Required environment variables:

• AWS CLI credentials: Credentials required for executing the CLI and deleting a function

make delete

Example Response

{
  "zipCode": "90210",
  "country": "United States",
  "countryAbbreviation": "US",
  "place": "Beverly Hills",
  "state": "California",
  "stateAbbreviation": "CA",
  "latitude": "34.0901",
  "longitude": "-118.4065"
}