diff --git a/docs/best-practices/worker.mdx b/docs/best-practices/worker.mdx index f9adb04905..d3aa61e55b 100644 --- a/docs/best-practices/worker.mdx +++ b/docs/best-practices/worker.mdx @@ -59,7 +59,6 @@ production-ready Worker image: {/* SNIPSTART oms-dockerfile-worker */} [Dockerfile](https://github.com/temporalio/reference-app-orders-go/blob/main/Dockerfile) - ```Dockerfile FROM golang:1.24.1 AS oms-builder @@ -79,7 +78,6 @@ RUN --mount=type=cache,target=/go/pkg/mod \ FROM busybox AS oms-worker ``` - {/* SNIPEND oms-dockerfile-worker */} This Dockerfile uses a multi-stage build pattern with two stages: @@ -102,27 +100,25 @@ uses environment variables to configure the Worker: {/* SNIPSTART oms-billing-worker-deployment {"selectedLines": ["20-35"]} */} [deployments/k8s/billing-worker-deployment.yaml](https://github.com/temporalio/reference-app-orders-go/blob/main/deployments/k8s/billing-worker-deployment.yaml) - ```yaml # ... -spec: - containers: - - args: - - -k - - supersecretkey - - -s - - billing - env: - - name: FRAUD_API_URL - value: http://billing-api:8084 - - name: TEMPORAL_ADDRESS - value: temporal-frontend.temporal:7233 - image: ghcr.io/temporalio/reference-app-orders-go-worker:latest - name: billing-worker - imagePullPolicy: Always - enableServiceLinks: false + spec: + containers: + - args: + - -k + - supersecretkey + - -s + - billing + env: + - name: FRAUD_API_URL + value: http://billing-api:8084 + - name: TEMPORAL_ADDRESS + value: temporal-frontend.temporal:7233 + image: ghcr.io/temporalio/reference-app-orders-go-worker:latest + name: billing-worker + imagePullPolicy: Always + enableServiceLinks: false ``` - {/* SNIPEND */} ### Separate Task Queues logically @@ -150,7 +146,6 @@ Worker reference to avoid this issue. {/* SNIPSTART oms-billing-worker-go {"selectedLines": ["12-23"]} */} [app/billing/worker.go](https://github.com/temporalio/reference-app-orders-go/blob/main/app/billing/worker.go) - ```go // ... // RunWorker runs a Workflow and Activity worker for the Billing system. @@ -166,7 +161,6 @@ func RunWorker(ctx context.Context, config config.AppConfig, client client.Clien return w.Run(temporalutil.WorkerInterruptFromContext(ctx)) } ``` - {/* SNIPEND */} ### Use Worker Versioning to safely deploy new Workflow code diff --git a/docs/develop/dotnet/observability.mdx b/docs/develop/dotnet/observability.mdx index 042bb2961c..cbf9b08b2a 100644 --- a/docs/develop/dotnet/observability.mdx +++ b/docs/develop/dotnet/observability.mdx @@ -165,8 +165,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). -- A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/environment-configuration.mdx b/docs/develop/environment-configuration.mdx index 30a3398500..347cc4e1ee 100644 --- a/docs/develop/environment-configuration.mdx +++ b/docs/develop/environment-configuration.mdx @@ -335,7 +335,6 @@ To load the `default` profile along with any environment variables in TypeScript {/* SNIPSTART typescript-env-config-load-default-profile {"highlightedLines": "17-19,28-29"} */} [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) - ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; @@ -378,7 +377,6 @@ main().catch((err) => { process.exit(1); }); ``` - {/* SNIPEND */} @@ -693,7 +691,6 @@ To load a specific profile from a custom path in TypeScript, use the `loadClient {/* SNIPSTART typescript-env-config-load-default-profile {"highlightedLines": "17-19,28-29"} */} [env-config/src/load-from-file.ts](https://github.com/temporalio/samples-typescript/blob/main/env-config/src/load-from-file.ts) - ```ts {17-19,28-29} import { Connection, Client } from '@temporalio/client'; import { loadClientConnectConfig } from '@temporalio/envconfig'; @@ -736,7 +733,6 @@ main().catch((err) => { process.exit(1); }); ``` - {/* SNIPEND */} diff --git a/docs/develop/go/observability.mdx b/docs/develop/go/observability.mdx index 0158d3d4c0..7cbf7017b5 100644 --- a/docs/develop/go/observability.mdx +++ b/docs/develop/go/observability.mdx @@ -304,8 +304,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). - - A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/go/temporal-client.mdx b/docs/develop/go/temporal-client.mdx index 922d04f2d1..545fe4ad57 100644 --- a/docs/develop/go/temporal-client.mdx +++ b/docs/develop/go/temporal-client.mdx @@ -207,7 +207,6 @@ variables or a configuration file, and then override specific options in code. {/* SNIPSTART samples-apps-go-yourapp-gateway {"selectedLines": ["1-23", "32"]} */} [sample-apps/go/yourapp/gateway/main.go](https://github.com/temporalio/documentation/blob/main/sample-apps/go/yourapp/gateway/main.go) - ```go package main @@ -235,7 +234,6 @@ func main() { // ... } ``` - {/* SNIPEND */} diff --git a/docs/develop/java/observability.mdx b/docs/develop/java/observability.mdx index 11363a6e0f..65757d8307 100644 --- a/docs/develop/java/observability.mdx +++ b/docs/develop/java/observability.mdx @@ -176,8 +176,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). - - A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/java/temporal-client.mdx b/docs/develop/java/temporal-client.mdx index 1e7a4ed224..8c509841e6 100644 --- a/docs/develop/java/temporal-client.mdx +++ b/docs/develop/java/temporal-client.mdx @@ -58,9 +58,7 @@ Use the `newLocalServiceStubs` method to create a stub that points to the Tempor to create a Temporal Client. - [sample-apps/java/client/devserver-client-sample/src/main/java/clientsample/YourCallerApp.java](https://github.com/temporalio/documentation/blob/main/sample-apps/java/client/devserver-client-sample/src/main/java/clientsample/YourCallerApp.java) - ```java {3,7} // ... // Create an instance that connects to a Temporal Service running on the local @@ -71,7 +69,6 @@ to create a Temporal Client. // This application uses the Client to communicate with the local Temporal Service WorkflowClient client = WorkflowClient.newInstance(serviceStub); ``` - When you create a new Client with an instance of `newLocalServiceStubs`, the Client connects to the default local port @@ -273,9 +270,7 @@ code. This is convenient for local development and testing. You can also load a variables or a configuration file, and then override specific options in code. - [sample-apps/java/client/devserver-namespace-client-sample/src/main/java/clientsample/YourCallerApp.java](https://github.com/temporalio/documentation/blob/main/sample-apps/java/client/devserver-namespace-client-sample/src/main/java/clientsample/YourCallerApp.java) - ```java // ... // Add the Namespace as a Client Option @@ -288,7 +283,6 @@ variables or a configuration file, and then override specific options in code. // This application uses the Client to communicate with the Temporal Service WorkflowClient client = WorkflowClient.newInstance(service, clientOptions); ``` - @@ -552,9 +546,7 @@ When you use a remote service, you don't use the `newLocalServicesStubs` conveni connection details as stub configuration options: - [sample-apps/java/client/cloudserver-client-sample/src/main/java/clientsample/YourCallerApp.java](https://github.com/temporalio/documentation/blob/main/sample-apps/java/client/cloudserver-client-sample/src/main/java/clientsample/YourCallerApp.java) - ```java // ... // Set the Service Stub options (SSL context and gRPC endpoint) @@ -567,7 +559,6 @@ connection details as stub configuration options: // Create a stub that accesses a Temporal Service WorkflowServiceStubs serviceStub = WorkflowServiceStubs.newServiceStubs(stubsOptions); ``` - Each Temporal Cloud service Client has four prerequisites. @@ -584,9 +575,7 @@ information configures a service stub for Temporal Cloud. Add the Namespace to y the new Client: - [sample-apps/java/client/cloudserver-client-sample/src/main/java/clientsample/YourCallerApp.java](https://github.com/temporalio/documentation/blob/main/sample-apps/java/client/cloudserver-client-sample/src/main/java/clientsample/YourCallerApp.java) - ```java // ... // Generate an SSL context @@ -614,7 +603,6 @@ the new Client: // This application uses the Client to communicate with the Temporal Service WorkflowClient client = WorkflowClient.newInstance(serviceStub, clientOptions); ``` - diff --git a/docs/develop/php/observability.mdx b/docs/develop/php/observability.mdx index ea2dee589c..2a526ed611 100644 --- a/docs/develop/php/observability.mdx +++ b/docs/develop/php/observability.mdx @@ -113,8 +113,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). - - A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/php/temporal-clients.mdx b/docs/develop/php/temporal-clients.mdx index 2a7d54e18e..9e5b9c9e2f 100644 --- a/docs/develop/php/temporal-clients.mdx +++ b/docs/develop/php/temporal-clients.mdx @@ -45,7 +45,7 @@ However, it is acceptable and common to use a Temporal Client inside an Activity ::: When you are running a Temporal Service locally (such as the [Temporal CLI](https://docs.temporal.io/cli/server#start-dev)), the number of connection options you must provide is minimal. -Many SDKs default to the local host or IP address and port that Temporalite and [Docker Compose](https://github.com/temporalio/docker-compose) serve (`127.0.0.1:7233`). +Many SDKs default to `127.0.0.1:7233`. In the PHP SDK, different client classes are responsible for different functional areas. The [`ServiceClient`](https://php.temporal.io/classes/Temporal-Client-GRPC-ServiceClient.html) is responsible for the low-level API and connection to the Temporal Service. diff --git a/docs/develop/python/observability.mdx b/docs/develop/python/observability.mdx index ff67b51ea2..62ad97f4f4 100644 --- a/docs/develop/python/observability.mdx +++ b/docs/develop/python/observability.mdx @@ -146,8 +146,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). - - A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/ruby/observability.mdx b/docs/develop/ruby/observability.mdx index 571d04cdff..fa3cf42edd 100644 --- a/docs/develop/ruby/observability.mdx +++ b/docs/develop/ruby/observability.mdx @@ -140,8 +140,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). -- A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/develop/typescript/core-application.mdx b/docs/develop/typescript/core-application.mdx index d5a513538f..50a855150a 100644 --- a/docs/develop/typescript/core-application.mdx +++ b/docs/develop/typescript/core-application.mdx @@ -216,7 +216,7 @@ However, it is acceptable and common to use a Temporal Client inside an Activity ::: When you are running a Temporal Service locally (such as the [Temporal CLI](https://docs.temporal.io/cli/server#start-dev)), the number of connection options you must provide is minimal. -Many SDKs default to the local host or IP address and port that Temporalite and [Docker Compose](https://github.com/temporalio/docker-compose) serve (`127.0.0.1:7233`). +Many SDKs default to `127.0.0.1:7233`. Creating a [Connection](https://typescript.temporal.io/api/classes/client.Connection) connects to the Temporal Service, and you can pass the `Connection` instance when creating the [Client](https://typescript.temporal.io/api/classes/client.Client#connection). diff --git a/docs/develop/typescript/integrations/ai-sdk.mdx b/docs/develop/typescript/integrations/ai-sdk.mdx index 23644028a0..e3a7d3aea0 100644 --- a/docs/develop/typescript/integrations/ai-sdk.mdx +++ b/docs/develop/typescript/integrations/ai-sdk.mdx @@ -159,9 +159,7 @@ For example, if you want to call an external API to get the weather, you would i from the tool function. The following is an example of an Activity that gets the weather for a given location: - [ai-sdk/src/activities.ts](https://github.com/temporalio/samples-typescript/blob/main/ai-sdk/src/activities.ts) - ```ts export async function getWeather(input: { location: string; @@ -174,7 +172,6 @@ export async function getWeather(input: { }; } ``` - Then in your agent implementation, provide the tool to the model using the `tools` option and instruct the model to use diff --git a/docs/develop/typescript/observability.mdx b/docs/develop/typescript/observability.mdx index 95ad51d595..1e82fcea17 100644 --- a/docs/develop/typescript/observability.mdx +++ b/docs/develop/typescript/observability.mdx @@ -444,8 +444,7 @@ However, sometimes you'll want to retrieve one or more Workflow Executions based You can do this with [Search Attributes](/search-attribute). - [Default Search Attributes](/search-attribute#default-search-attribute) like `WorkflowType`, `StartTime` and `ExecutionStatus` are automatically added to Workflow Executions. -- _Custom Search Attributes_ can contain their own domain-specific data (like `customerId` or `numItems`). - - A few [generic Custom Search Attributes](/search-attribute#custom-search-attribute) like `CustomKeywordField` and `CustomIntField` are created by default in Temporal's [Docker Compose](https://github.com/temporalio/docker-compose). +- [Custom Search Attributes](/search-attribute#custom-search-attribute) can contain their own domain-specific data (like `customerId` or `numItems`). The steps to using custom Search Attributes are: diff --git a/docs/production-deployment/self-hosted-guide/defaults.mdx b/docs/production-deployment/self-hosted-guide/defaults.mdx index 03f9c4498b..143d1c2aa8 100644 --- a/docs/production-deployment/self-hosted-guide/defaults.mdx +++ b/docs/production-deployment/self-hosted-guide/defaults.mdx @@ -31,7 +31,7 @@ These limits might apply specifically to each Workflow Execution and do not pert ::: - **Identifiers:** By default, the maximum length for identifiers (such as Workflow Id, Workflow Type, and Task Queue name) is 1000 characters. - - This is configurable with the `limit.maxIDLength` dynamic config variable, set to 255 in [this SQL example](https://github.com/temporalio/docker-compose/blob/93d382ef9133e4cde8ce311de5153cd0cc9fbd0c/dynamicconfig/development-sql.yaml#L1-L2). + - This is configurable with the `limit.maxIDLength` dynamic config variable, set to 255 in [this SQL example](https://github.com/temporalio/samples-server/blob/main/compose/dynamicconfig/development-sql.yaml). - The character format is UTF-8. - **gRPC:** gRPC has a limit of 4 MB for [each message received](https://github.com/grpc/grpc/blob/v1.36.2/include/grpc/impl/codegen/grpc_types.h#L466). - **Event batch size:** The `DefaultTransactionSizeLimit` limit is [4 MB](https://github.com/temporalio/temporal/pull/1363). diff --git a/docs/production-deployment/self-hosted-guide/deployment.mdx b/docs/production-deployment/self-hosted-guide/deployment.mdx index 115c175707..a069aa8f51 100644 --- a/docs/production-deployment/self-hosted-guide/deployment.mdx +++ b/docs/production-deployment/self-hosted-guide/deployment.mdx @@ -2,7 +2,9 @@ id: deployment title: Deploying a Temporal Service sidebar_label: Deployment -description: Deploy a Temporal Service using Docker, Kubernetes, or from scratch. Requires a database such as Apache Cassandra, MySQL, or PostgreSQL. Customize setup for your infrastructure and tooling. +description: + Deploy a Temporal Service using Docker, Kubernetes, or from scratch. Requires a database such as Apache Cassandra, + MySQL, or PostgreSQL. Customize setup for your infrastructure and tooling. slug: /self-hosted-guide/deployment toc_max_heading_level: 4 keywords: @@ -15,88 +17,131 @@ tags: - Self-hosting --- -There are many ways to self-host a [Temporal Service](/temporal-service). -The right way for you depends entirely on your use case and where you plan to run it. +There are many ways to self-host a [Temporal Service](/temporal-service). The right way for you depends entirely on your +use case and where you plan to run it. + +This page provides instructions for deploying a Temporal Service for sustained workloads that exceed what the +[development server](/cli#start-dev-server) is designed to handle. For local development or testing, you can use the +Temporal CLI to [start a local development Temporal Service](/cli#start-dev-server). :::warning Temporal Hosts Should Not Be Exposed to the Open Internet In self-hosted deployments, the Temporal Service is a critical control and persistence component and should be secured similarly to a database. Temporal services should run on hosts that are not accessible from the public internet, with network configurations that restrict access to trusted internal networks only. + ::: For step-by-step guides on deploying and configuring Temporal, refer to our [Infrastructure tutorials](https://learn.temporal.io/tutorials/infrastructure/). -### Minimum requirements +## Use Docker Compose + +You can run a Temporal Service in [Docker](https://docs.docker.com/engine/install) containers using +[Docker Compose](https://docs.docker.com/compose/install). + +### Prerequisites + +- You have Docker Compose installed. +- Docker is running and the daemon is available. +- Git is installed and available. + +### Procedure + +1. Clone the [temporalio/samples-server](https://github.com/temporalio/samples-server) repository. -The Temporal Server depends on a database. +2. Change into the `compose` directory. -Supported databases include the following: + ``` + cd samples-server/compose + ``` -- [Apache Cassandra](/self-hosted-guide/visibility#cassandra) -- [MySQL](/self-hosted-guide/visibility#mysql) -- [PostgreSQL](/self-hosted-guide/visibility#postgresql) -- [SQLite](/self-hosted-guide/visibility#sqlite) +3. Run the `docker compose up` command. This uses the default configuration from the `docker-compose.yaml` file, which + includes a PostgreSQL database, an Elasticsearch instance, and exposes the Temporal gRPC Frontend on port 7233. -### Docker & Docker Compose + ``` + docker compose up + ``` -You can run a Temporal Service in [Docker](https://docs.docker.com/engine/install) containers using [Docker Compose](https://docs.docker.com/compose/install). + The Temporal Web UI will be available at `http://localhost:8233`. -If you have Docker and Docker Compose installed, all you need to do is clone the [temporalio/docker-compose](https://github.com/temporalio/docker-compose) repo and run the `docker compose up` command from its root. +4. (Optional) Review + [the additional configuration options](https://github.com/temporalio/samples-server/tree/main/compose#other-configuration-files) + available in the samples-server repository and use `docker compose up` with the corresponding configuration file to + try them out. The configurations include different databases, visibility stores, and TLS settings. -The `temporalio/docker-compose` repo comes loaded with a variety of configuration templates that enable you to try all three databases that the Temporal Platform supports (PostgreSQL, MySQL, Cassandra). -It also enables you to try [Advanced Visibility](/visibility#advanced-visibility) using [Search Attributes](/search-attribute), emit metrics, and even play with the [Archival](/temporal-service/archival) feature. -The Docker images in this repo are produced using the Temporal Server [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script. -This script defaults to creating images that run all the Temporal Server services in a single process. -You can use this script as a starting point for producing your own images. +## Use Temporal Server binaries -The following commands start and run a Temporal Service in Docker using the default configuration: +You can run a complete Temporal Server by deploying two Go binaries -- the +[core Temporal Server](https://github.com/temporalio/temporal/releases/), and the +[Temporal UI Server](https://github.com/temporalio/ui-server/releases). -```bash -git clone https://github.com/temporalio/docker-compose.git -cd docker-compose -docker compose up -``` +Each service can be deployed separately. Refer to +[How to Configure a Temporal Service without a Proxy](https://learn.temporal.io/tutorials/infrastructure/configuring-sqlite-binary/) +to deploy each service using `systemd`. If you need to run the Temporal Server behind a reverse proxy, refer to our +tutorials to deploy the Temporal Service behind an +[Nginx reverse proxy](https://learn.temporal.io/tutorials/infrastructure/nginx-sqlite-binary/) or an +[Envoy edge proxy](https://learn.temporal.io/tutorials/infrastructure/envoy-sqlite-binary/). -Local [Temporal Clients](/encyclopedia/temporal-sdks#temporal-client) and [Workers](/workers) can connect to the Temporal Service running in Docker at 127.0.0.1:7233 (default connection for most SDKs) and the Temporal Web UI at 127.0.0.1:8080. +### Configuration templating -To try other configurations (different dependencies and databases), or to try a custom Docker image, follow the [temporalio/docker-compose README](https://github.com/temporalio/docker-compose/blob/main/README.md). +Configuration templating is how the Temporal Server turns a template config file into the final `config.yaml` it runs +with. It lets you reuse one template across environments by filling in values from environment variables. For example, +database endpoints, TLS paths, or feature flags. -### Temporal Server binaries +If you are **not** using a custom config template, you can skip this section. The default configuration is rendered +automatically by the server and embedded in the binary. -You can run a complete Temporal Server by deploying just two Go binaries -- the [core Temporal Server](https://github.com/temporalio/temporal/releases/), and the [Temporal UI Server](https://github.com/temporalio/ui-server/releases). -Refer to our [tutorial site](https://learn.temporal.io/) for more details on how to deploy Temporal binaries behind an [Nginx reverse proxy](https://learn.temporal.io/tutorials/infrastructure/nginx-sqlite-binary/) or an [Envoy edge proxy](https://learn.temporal.io/tutorials/infrastructure/envoy-sqlite-binary/). +#### Template compatibility -Each service can also be deployed separately. -For example, if you are using Kubernetes, you could have one service per pod, so they can scale independently in the future. +If you use a custom configuration template, be aware of the following: -In Docker, you could run each service in its own container, using the `SERVICES` flag to specify the service: +- The server renders templates with embedded `sprig`, so any `dockerize`-specific syntax or helpers will fail +- Some template syntax differs, particularly `.Env` and `default` function usage. +- Refer to the [sprig documentation](http://masterminds.github.io/sprig/) for supported template functions +- Use `temporal-server render-config` to verify your templates render correctly -```bash -docker run - # persistence/schema setup flags omitted - -e SERVICES=history \ -- Spin up one or more: history, matching, worker, frontend - -e LOG_LEVEL=debug,info \ -- Logging level - -e DYNAMIC_CONFIG_FILE_PATH=config/foo.yaml -- Dynamic config file to be watched - temporalio/server: -``` +#### Helm Chart configuration -The environment variables supported by the Temporal Docker images are documented [on Docker Hub](https://hub.docker.com/r/temporalio/auto-setup). +When deploying with Helm charts versions 0.73.1 or later, you may need to adjust the following configuration options +depending on the images you are using. -Each Temporal Server release ships an [Auto Setup](https://temporal.io/blog/auto-setup) Docker image that includes an [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script. -We recommend using this script for initial schema setup of each supported database. +| Configuration Option | Description | Default | +| ---------------------------- | ------------------------------------------------------------------------- | ------------- | +| `server.useEntrypointScript` | Whether to use entrypoint script that autodetects `dockerize` vs `sprig`. | `false` | +| `server.configMapsToMount` | Which config template to mount: `"dockerize"`, `"sprig"`, or `"both"`. | `"dockerize"` | +| `server.setConfigFilePath` | Set `TEMPORAL_SERVER_CONFIG_FILE_PATH` environment variable. | `false` | -### Importing the Server package +Refer to the following guidelines to determine if you need to adjust the configuration options: -The Temporal Server is a standalone Go application that can be [imported](/references/server-options) into another project as a library. -This is useful for testing, development, and building customized servers. +- The default settings work if you are only using pre-1.30 images with 0.73.1 or later Helm chart. +- If you are using 1.30+ images with 0.73.1 or later Helm chart, you need to set `server.configMapsToMount` to `"sprig"` + and `server.setConfigFilePath` to `true`. Keep the `server.useEntrypointScript` as `false`. +- If you need use the Helm chart with both pre-1.30 and 1.30+ images, you need to set `server.configMapsToMount` to + `"both"` and `server.useEntrypointScript` to `true`. Keep the `server.setConfigFilePath` as `false`. -You can pass custom plugins or other customizations through the [Server Options](/references/server-options), then build and run a binary that contains your customizations. +## Import the Server package -See [Embedded server](/self-hosted-guide/embedded-server) for details on running Temporal in-process with SQLite, or the [samples-server repository](https://github.com/temporalio/samples-server) for examples of server extensibility (authorization, metrics, TLS). +The Temporal Server is a standalone Go application that can be [imported](/references/server-options) into another +project. -This requires Go v1.19 or later, as specified in the Temporal Server [Build prerequisites](https://github.com/temporalio/temporal/blob/main/CONTRIBUTING.md#build-prerequisites). +You might want to do this to pass custom plugins or any other customizations through the +[Server Options](/references/server-options). Then you can build and run a binary that contains your customizations. -### Helm charts +This requires Go v1.19 or later, as specified in the Temporal Server +[Build prerequisites](https://github.com/temporalio/temporal/blob/main/CONTRIBUTING.md#build-prerequisites). -[Temporal Helm charts](https://github.com/temporalio/helm-charts) enable you to get a Temporal Service running on [Kubernetes](https://kubernetes.io/) by deploying the Temporal Server services to individual pods and connecting them to your existing database and Elasticsearch instances. +## Use Helm charts -The Temporal Helm charts repo contains [extensive documentation](https://github.com/temporalio/helm-charts/blob/main/README.md) about Kubernetes deployments. +[Temporal Helm charts](https://github.com/temporalio/helm-charts) enable you to get a Temporal Service running on +[Kubernetes](https://kubernetes.io/) by deploying the Temporal Server services to individual pods and connecting them to +your existing database and Elasticsearch instances. + +The Temporal Helm charts repo contains +[extensive documentation](https://github.com/temporalio/helm-charts/blob/main/README.md) about Kubernetes deployments. + +:::caution Helm Chart version compatibility + +If you are using Temporal Server images 1.30+, you must upgrade to Helm chart version 0.73.1 or later. + +Helm chart versions below 0.73.1 are **not compatible** with `server` and `admin-tools` images **version 1.30 and +later**. You **cannot** override old chart versions with newer images. + +::: diff --git a/docs/production-deployment/self-hosted-guide/namespaces.mdx b/docs/production-deployment/self-hosted-guide/namespaces.mdx index d625f40749..6e2238a2e0 100644 --- a/docs/production-deployment/self-hosted-guide/namespaces.mdx +++ b/docs/production-deployment/self-hosted-guide/namespaces.mdx @@ -2,7 +2,8 @@ id: namespaces title: Managing Namespaces sidebar_label: Namespaces -description: How to create and manage Namespaces in open source Temporal, including registration, configuration, and security. +description: + How to create and manage Namespaces in open source Temporal, including registration, configuration, and security. slug: /self-hosted-guide/namespaces keywords: - namespaces @@ -14,18 +15,20 @@ tags: --- :::info Open source Temporal -This page covers namespace operations for **open source Temporal**. -For core namespace concepts, see [Temporal Namespace](/namespaces). -For Temporal Cloud, see [Temporal Cloud Namespaces](/cloud/namespaces). + +This page covers namespace operations for **open source Temporal**. For core namespace concepts, see +[Temporal Namespace](/namespaces). For Temporal Cloud, see [Temporal Cloud Namespaces](/cloud/namespaces). + ::: -A [Namespace](/namespaces) is a unit of isolation within the Temporal Platform. -Before you can run Workflows, you must register at least one Namespace with your Temporal Service. +A [Namespace](/namespaces) is a unit of isolation within the Temporal Platform. Before you can run Workflows, you must +register at least one Namespace with your Temporal Service. ## Create a Namespace -Registering a Namespace creates it on the Temporal Service. -When you register a Namespace, you must set a [Retention Period](/temporal-service/temporal-server#retention-period) that determines how long closed Workflow execution history is kept. +Registering a Namespace creates it on the Temporal Service. When you register a Namespace, you must set a +[Retention Period](/temporal-service/temporal-server#retention-period) that determines how long closed Workflow +execution history is kept. You can create Namespaces using: @@ -36,32 +39,34 @@ You can create Namespaces using: ### The default Namespace -If no Namespace is specified, SDKs and CLI use the `default` Namespace. -You must register this Namespace before using it. +If no Namespace is specified, SDKs and CLI use the `default` Namespace. You must register this Namespace before using +it. -When deploying with Docker Compose or the [auto-setup image](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh), the `default` Namespace is created automatically. +For local development, the [`temporal server start-dev`](/cli/server#start-dev) command automatically creates the +`default` Namespace. -When deploying with [Helm charts](https://github.com/temporalio/helm-charts), create it manually: +For all other deployment methods, create the `default` Namespace manually using the Temporal CLI: ```bash temporal operator namespace create --namespace default ``` -Namespace registration takes up to 15 seconds to complete. -Wait for this process to finish before making calls to the Namespace. +Namespace registration takes up to 15 seconds to complete. Wait for this process to finish before making calls to the +Namespace. ## Manage Namespaces Common namespace management operations: -| Operation | CLI Command | Description | -|-----------|-------------|-------------| -| List | [`temporal operator namespace list`](/cli/operator#list) | List all registered Namespaces | -| Describe | [`temporal operator namespace describe`](/cli/operator#describe) | Get details for a Namespace | -| Update | [`temporal operator namespace update`](/cli/operator#update) | Update Namespace configuration | -| Delete | [`temporal operator namespace delete`](/cli/operator#delete) | Delete a Namespace and all its data | +| Operation | CLI Command | Description | +| --------- | ---------------------------------------------------------------- | ----------------------------------- | +| List | [`temporal operator namespace list`](/cli/operator#list) | List all registered Namespaces | +| Describe | [`temporal operator namespace describe`](/cli/operator#describe) | Get details for a Namespace | +| Update | [`temporal operator namespace update`](/cli/operator#update) | Update Namespace configuration | +| Delete | [`temporal operator namespace delete`](/cli/operator#delete) | Delete a Namespace and all its data | For SDK-based namespace management: + - [Go SDK namespace management](/develop/go/namespaces#manage-namespaces) - [Java SDK namespace management](/develop/java/namespaces#manage-namespaces) - [TypeScript SDK namespace management](/develop/typescript/namespaces#manage-namespaces) @@ -73,12 +78,15 @@ For SDK-based namespace management: ## Security -Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to control who can create, update, or deprecate Namespaces. +Use a custom [Authorizer](/self-hosted-guide/security#authorizer-plugin) on your Frontend Service to control who can +create, update, or deprecate Namespaces. Without an Authorizer configured, Temporal uses the `nopAuthority` authorizer that allows all API calls unconditionally. -For Temporal Cloud, [role-based access controls](/cloud/users#namespace-level-permissions) provide namespace-level authorization without custom configuration. +For Temporal Cloud, [role-based access controls](/cloud/users#namespace-level-permissions) provide namespace-level +authorization without custom configuration. ## Best practices -For namespace naming conventions, organizational patterns, and production safeguards, see [Namespace Best Practices](/best-practices/managing-namespace). +For namespace naming conventions, organizational patterns, and production safeguards, see +[Namespace Best Practices](/best-practices/managing-namespace). diff --git a/docs/production-deployment/self-hosted-guide/upgrade-server.mdx b/docs/production-deployment/self-hosted-guide/upgrade-server.mdx index 7e9f821e7d..5f3d4cad15 100644 --- a/docs/production-deployment/self-hosted-guide/upgrade-server.mdx +++ b/docs/production-deployment/self-hosted-guide/upgrade-server.mdx @@ -2,7 +2,9 @@ id: upgrade-server title: Upgrade the Temporal Server sidebar_label: Upgrade server -description: Upgrade your Temporal Server effectively. Follow our step-by-step guide for seamless upgrades, ensuring data compatibility and schema alignment. +description: + Upgrade your Temporal Server effectively. Follow our step-by-step guide for seamless upgrades, ensuring data + compatibility and schema alignment. slug: /self-hosted-guide/upgrade-server toc_max_heading_level: 4 keywords: @@ -14,65 +16,73 @@ tags: ## How to upgrade the Temporal Server version {#upgrade-server} -If a newer version of the [Temporal Server](/temporal-service/temporal-server) is available, a notification appears in the Temporal Web UI. +If a newer version of the [Temporal Server](/temporal-service/temporal-server) is available, a notification appears in +the Temporal Web UI. :::info -If you are using a version that is older than 1.0.0, reach out to us at [community.temporal.io](http://community.temporal.io) to ask how to upgrade. +If you are using a version that is older than 1.0.0, reach out to us at +[community.temporal.io](http://community.temporal.io) to ask how to upgrade. ::: -First check to see if an upgrade to the database schema is required for the version you wish to upgrade to. -If a database schema upgrade is required, it will be called out directly in the [release notes](https://github.com/temporalio/temporal/releases). -Some releases require changes to the schema, and some do not. -We ensure that any consecutive versions are compatible in terms of database schema upgrades, features, and system behavior; however there is no guarantee that there is compatibility between _any_ two non-consecutive versions. +First check to see if an upgrade to the database schema is required for the version you wish to upgrade to. If a +database schema upgrade is required, it will be called out directly in the +[release notes](https://github.com/temporalio/temporal/releases). Some releases require changes to the schema, and some +do not. We ensure that any consecutive versions are compatible in terms of database schema upgrades, features, and +system behavior; however there is no guarantee that there is compatibility between _any_ two non-consecutive versions. ### Key considerations When upgrading the Temporal Server, there are two key considerations to keep in mind: -1. **Sequential Upgrades:** Temporal Server should be upgraded sequentially. - That is, if you're on version \(v1.n.x\), your next upgrade should be to \(v1.n+1.x\) or the closest available subsequent version. - This sequence should be repeated until your desired version is reached. +1. **Sequential Upgrades:** Temporal Server should be upgraded sequentially. That is, if you're on version \(v1.n.x\), + your next upgrade should be to \(v1.n+1.x\) or the closest available subsequent version. This sequence should be + repeated until your desired version is reached. -2. **Data Compatibility:** During an upgrade, the Temporal Server either updates or restructures the existing version data to match the data format of the newer version. - Temporal Server ensures backward compatibility only between two successive minor versions. - Consequently, skipping versions during an upgrade may lead to older data formats becoming unreadable. - If the previous data format cannot be interpreted and converted to the newer format, the upgrade process will be unsuccessful. +2. **Data Compatibility:** During an upgrade, the Temporal Server either updates or restructures the existing version + data to match the data format of the newer version. Temporal Server ensures backward compatibility only between two + successive minor versions. Consequently, skipping versions during an upgrade may lead to older data formats becoming + unreadable. If the previous data format cannot be interpreted and converted to the newer format, the upgrade process + will be unsuccessful. ### Step-by-Step Upgrade Procedure: -Upgrading the Temporal Server requires a methodical approach to ensure data integrity, compatibility, and seamless transition between versions. -The following documentation outlines the step-by-step process to successfully upgrade your Temporal Server. +Upgrading the Temporal Server requires a methodical approach to ensure data integrity, compatibility, and seamless +transition between versions. The following documentation outlines the step-by-step process to successfully upgrade your +Temporal Server. When upgrading your Temporal Server version, ensure that you upgrade sequentially. -1. **Upgrade Database Schema:** Before initiating the Temporal Server upgrade, use one of the recommended upgrade tools to update your database schema. - This ensures it is aligned with the version of Temporal Server you aim to upgrade to. -2. **Upgrade Temporal Server:** Once the database schema is updated, proceed to upgrade the Temporal Server deployment to the next sequential version. -3. **Iterative Upgrades** (optional): Continue this process (steps 1 and 2) iteratively until you reach the desired Temporal Server version. +1. **Upgrade Database Schema:** Before initiating the Temporal Server upgrade, use one of the recommended upgrade tools + to update your database schema. This ensures it is aligned with the version of Temporal Server you aim to upgrade to. +2. **Upgrade Temporal Server:** Once the database schema is updated, proceed to upgrade the Temporal Server deployment + to the next sequential version. +3. **Iterative Upgrades** (optional): Continue this process (steps 1 and 2) iteratively until you reach the desired + Temporal Server version. -By adhering to the above guidelines and following the step-by-step procedure, you can ensure a smooth and successful upgrade of your Temporal Server. +By adhering to the above guidelines and following the step-by-step procedure, you can ensure a smooth and successful +upgrade of your Temporal Server. The Temporal Server upgrade updates or rewrites the old version data with the format introduced in the newer version. -Because Temporal Server guarantees backward compatibility between two consecutive minor versions, and because older versions of the code are eventually removed from the code base, skipping versions when upgrading might cause older formats to become unrecognizable. -If the old format of the data can't be read to be rewritten to the new format, the upgrades fail. +Because Temporal Server guarantees backward compatibility between two consecutive minor versions, and because older +versions of the code are eventually removed from the code base, skipping versions when upgrading might cause older +formats to become unrecognizable. If the old format of the data can't be read to be rewritten to the new format, the +upgrades fail. -Check the [Temporal Server releases](https://github.com/temporalio/temporal/releases) and follow these releases in order. -You can skip patch versions; use the latest patch of a minor version when upgrading. +Check the [Temporal Server releases](https://github.com/temporalio/temporal/releases) and follow these releases in +order. You can skip patch versions; use the latest patch of a minor version when upgrading. -Also, be aware that each upgrade requires the History Service to load all Shards and update the Shard metadata, so allow approximately 10 minutes on each version for these processes to complete before upgrading to the next version. +Also, be aware that each upgrade requires the History Service to load all Shards and update the Shard metadata, so allow +approximately 10 minutes on each version for these processes to complete before upgrading to the next version. -Use one of the upgrade tools to upgrade your database schema to be compatible with the Temporal Server version being upgraded to. - -If you are using a schema tools version prior to Temporal Server v1.8.0, we strongly recommend _never_ using the "dryrun" (`-y`, or `--dryrun`) options in any of your schema update commands. -Using this option might lead to potential loss of data, as when using it will create a new database and drop your -existing one. -This flag was removed in the 1.8.0 release. +Use one of the upgrade tools to upgrade your database schema to be compatible with the Temporal Server version being +upgraded to. ### Upgrade Cassandra schema -If you are using Cassandra for your Temporal Service's persistence, use the `temporal-cassandra-tool` to upgrade both the default Persistence and Visibility schemas. +If you are using Cassandra for your Temporal Service's persistence, use the `temporal-cassandra-tool` to upgrade both +the default Persistence and Visibility schemas. **Example default schema upgrade:** @@ -104,9 +114,27 @@ temporal_v1.2.1 $ temporal-cassandra-tool \ --schema-dir ./schema/cassandra/visibility/versioned ``` +### Upgrade Elasticsearch schema + +If you are using Elasticsearch for your Temporal Service's Visibility, use the `temporal-elasticsearch-tool` to upgrade +the schema. + +**Example schema upgrade:** + +```bash +echo "Updating index mappings: $ES_VISIBILITY_INDEX" +temporal-elasticsearch-tool \ + --endpoint "$ES_SCHEME://$ES_HOST:$ES_PORT" \ + --user "$ES_USER" \ + --password "$ES_PWD" \ + update-schema \ + --index "$ES_VISIBILITY_INDEX" +``` + ### Upgrade PostgreSQL or MySQL schema -If you are using MySQL or PostgreSQL use the `temporal-sql-tool`, which works similarly to the `temporal-cassandra-tool`. +If you are using MySQL or PostgreSQL use the `temporal-sql-tool`, which works similarly to the +`temporal-cassandra-tool`. Refer to this [Makefile](https://github.com/temporalio/temporal/blob/v1.4.1/Makefile#L367-L383) for context. @@ -136,7 +164,9 @@ Refer to this [Makefile](https://github.com/temporalio/temporal/blob/v1.4.1/Make --ep localhost -p 5432 -u temporal -pw temporal --pl postgres --db temporal_visibility update-schema -d ./schema/postgresql/v96/visibility/versioned ``` -If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your PostgreSQL version first, and then run `temporal-sql-tool` with the `postgres12` plugin, as shown in the following example: +If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility features with Temporal Server v1.20, +upgrade your PostgreSQL version first, and then run `temporal-sql-tool` with the `postgres12` plugin, as shown in the +following example: ```bash ./temporal-sql-tool \ @@ -174,7 +204,8 @@ If you're upgrading PostgreSQL to v12 or later to enable advanced Visibility fea --ep localhost -p 3036 -u root -pw root --pl mysql --db temporal_visibility update-schema -d ./schema/mysql/v57/visibility/versioned/ ``` -If you're upgrading MySQL to v8.0.17 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade your MySQL version first, and then run `temporal-sql-tool` with the `mysql8` plugin, as shown in the following example: +If you're upgrading MySQL to v8.0.17 or later to enable advanced Visibility features with Temporal Server v1.20, upgrade +your MySQL version first, and then run `temporal-sql-tool` with the `mysql8` plugin, as shown in the following example: ```bash ./temporal-sql-tool \ @@ -192,6 +223,7 @@ We recommend preparing a staging Temporal Service and then do the following to v 1. Create some simulation load on the staging Temporal Service. 2. Upgrade the database schema in the staging Temporal Service. -3. Wait and observe for a few minutes to verify that there is no unstable behavior from both the server and the simulation load logic. +3. Wait and observe for a few minutes to verify that there is no unstable behavior from both the server and the + simulation load logic. 4. Upgrade the server. 5. Now do the same to the live environment Temporal Service. diff --git a/docs/production-deployment/self-hosted-guide/visibility.mdx b/docs/production-deployment/self-hosted-guide/visibility.mdx index 5bcedea8ee..9f59a176af 100644 --- a/docs/production-deployment/self-hosted-guide/visibility.mdx +++ b/docs/production-deployment/self-hosted-guide/visibility.mdx @@ -2,7 +2,10 @@ id: visibility title: Self-hosted Visibility feature setup sidebar_label: Visibility -description: A Visibility store is essential for your Temporal Service, supporting features like batch operations and custom Search Attributes for filtering Workflow Executions. Upgrade to PostgreSQL 12, MySQL 8.0.17, or SQLite 3.31.0 with Temporal Server 1.20+ for advanced Visibility +description: + A Visibility store is essential for your Temporal Service, supporting features like batch operations and custom Search + Attributes for filtering Workflow Executions. Upgrade to PostgreSQL 12, MySQL 8.0.17, or SQLite 3.31.0 with Temporal + Server 1.20+ for advanced Visibility slug: /self-hosted-guide/visibility toc_max_heading_level: 4 keywords: @@ -16,45 +19,53 @@ tags: - Visibility --- -A [Visibility](/temporal-service/visibility) store is set up as a part of your [Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions that exist on your Temporal Service. +A [Visibility](/temporal-service/visibility) store is set up as a part of your +[Persistence store](/temporal-service/persistence) to enable listing and filtering details about Workflow Executions +that exist on your Temporal Service. -A Visibility store is required in a Temporal Service setup because it is used by Temporal Web UI and CLI to pull Workflow Execution data and enables features like batch operations on a group of Workflow Executions. +A Visibility store is required in a Temporal Service setup because it is used by Temporal Web UI and CLI to pull +Workflow Execution data and enables features like batch operations on a group of Workflow Executions. -With the Visibility store, you can use [List Filters](/list-filter) with [Search Attributes](/search-attribute) to list and filter Workflow Executions that you want to review. +With the Visibility store, you can use [List Filters](/list-filter) with [Search Attributes](/search-attribute) to list +and filter Workflow Executions that you want to review. -Setting up [advanced Visibility](/visibility#advanced-visibility) enables access to creating and using multiple custom Search Attributes with your List Filters. +Setting up [advanced Visibility](/visibility#advanced-visibility) enables access to creating and using multiple custom +Search Attributes with your List Filters. For details, see [Search Attributes](/search-attribute). -Note that if you use MySQL, PostgreSQL, or SQLite as your Visibility store, Temporal Server version 1.20 and later supports advanced Visibility features on MySQL (version 8.0.17 and later), PostgreSQL (version 12 and later) and SQLite (v3.31.0 and later), in addition to Elasticsearch. +Note that if you use MySQL, PostgreSQL, or SQLite as your Visibility store, Temporal Server version 1.20 and later +supports advanced Visibility features on MySQL (version 8.0.17 and later), PostgreSQL (version 12 and later) and SQLite +(v3.31.0 and later), in addition to Elasticsearch. To enable advanced Visibility on your SQL databases, ensure that you do the following: - [Upgrade your Temporal Server](/self-hosted-guide/upgrade-server#upgrade-server) to version 1.20 or later. -- [Update your database schemas](/self-hosted-guide/upgrade-server#upgrade-server) for MySQL to version 8.0.17 (or later), PostgreSQL to version 12 (or later), or SQLite to v3.31.0 (or later). +- [Update your database schemas](/self-hosted-guide/upgrade-server#upgrade-server) for MySQL to version 8.0.17 (or + later), PostgreSQL to version 12 (or later), or SQLite to v3.31.0 (or later). -Beginning with Temporal Server v1.21, you can set up a secondary Visibility store in your Temporal Service to enable [Dual Visibility](/dual-visibility). -This is useful for migrating your Visibility store database. +Beginning with Temporal Server v1.21, you can set up a secondary Visibility store in your Temporal Service to enable +[Dual Visibility](/dual-visibility). This is useful for migrating your Visibility store database. #### Supported databases The following databases are supported as Visibility stores: -- [MySQL](#mysql) v5.7 and later. - Use v8.0.17 (or later) with Temporal Server v1.20 or later for advanced Visibility capabilities. - Because standard Visibility is deprecated beginning with Temporal Server v1.21, support for older versions of MySQL will be dropped. -- [PostgreSQL](#postgresql) v9.6 and later. - Use v12 (or later) with Temporal Server v1.20 or later for advanced Visibility capabilities. - Because standard Visibility is deprecated beginning with Temporal Server v1.21, support for older versions of PostgreSQL will be dropped. +- [MySQL](#mysql) v5.7 and later. Use v8.0.17 (or later) with Temporal Server v1.20 or later for advanced Visibility + capabilities. Because standard Visibility is deprecated beginning with Temporal Server v1.21, support for older + versions of MySQL will be dropped. +- [PostgreSQL](#postgresql) v9.6 and later. Use v12 (or later) with Temporal Server v1.20 or later for advanced + Visibility capabilities. Because standard Visibility is deprecated beginning with Temporal Server v1.21, support for + older versions of PostgreSQL will be dropped. - [SQLite](#sqlite) v3.31.0 and later for advanced Visibility capabilities. -- [Cassandra](#cassandra). - Support for Cassandra as a Visibility database is deprecated beginning with Temporal Server v1.21. - For information on migrating from Cassandra to any of the supported databases, see [Migrating Visibility database](#migrating-visibility-database). -- [Elasticsearch](#elasticsearch) supported versions. - We recommend operating a Temporal Service with Elasticsearch as your Visibility store for any use case that spawns more than a few Workflow Executions. +- [Cassandra](#cassandra). Support for Cassandra as a Visibility database is deprecated beginning with Temporal Server + v1.21. For information on migrating from Cassandra to any of the supported databases, see + [Migrating Visibility database](#migrating-visibility-database). +- [Elasticsearch](#elasticsearch) supported versions. We recommend operating a Temporal Service with Elasticsearch as + your Visibility store for any use case that spawns more than a few Workflow Executions. -You can use any combination of the supported databases for your Persistence and Visibility stores. -For updates, check [Server release notes](https://github.com/temporalio/temporal/releases). +You can use any combination of the supported databases for your Persistence and Visibility stores. For updates, check +[Server release notes](https://github.com/temporalio/temporal/releases). ## How to set up MySQL Visibility store {#mysql} @@ -66,16 +77,20 @@ For updates, check [Server release notes](https://github.com/temporalio/temporal ::: -You can set MySQL as your [Visibility store](/temporal-service/visibility). -Verify [supported versions](/self-hosted-guide/visibility) before you proceed. +You can set MySQL as your [Visibility store](/temporal-service/visibility). Verify +[supported versions](/self-hosted-guide/visibility) before you proceed. -If using MySQL v8.0.17 or later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. +If using MySQL v8.0.17 or later as your Visibility store with Temporal Server v1.20 and later, any +[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a +Namespace in that Temporal Service. -**Persistence configuration** +### Persistence configuration -Set your MySQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. +Set your MySQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then +define the Visibility store configuration under `datastores`. -The following example shows how to set a Visibility store `mysql-visibility` and define the datastore configuration in your Temporal Service configuration YAML. +The following example shows how to set a Visibility store `mysql-visibility` and define the datastore configuration in +your Temporal Service configuration YAML. ```yaml #... @@ -100,50 +115,53 @@ persistence: #... ``` -For details on the configuration parameters and values, see [Temporal Service configuration](/references/configuration#sql). +For details on the configuration parameters and values, see +[Temporal Service configuration](/references/configuration#sql). -To enable advanced Visibility features on your MySQL Visibility store, upgrade to MySQL v8.0.17 or later with Temporal Server v1.20 or later. -See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) on how to upgrade your Temporal Server and database schemas. +To enable advanced Visibility features on your MySQL Visibility store, upgrade to MySQL v8.0.17 or later with Temporal +Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) on how to upgrade your +Temporal Server and database schemas. -For example configuration templates, see [MySQL Visibility store configuration](https://github.com/temporalio/temporal/blob/main/config/development-mysql8.yaml). +For example configuration templates, see +[MySQL Visibility store configuration](https://github.com/temporalio/temporal/blob/main/config/development-mysql8.yaml). -**Database schema and setup** +### Database schema and setup -Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions): +Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for +[MySQL v8.0.17 and later](https://github.com/temporalio/temporal/tree/main/schema/mysql/v8/visibility). -- [MySQL v8.0.17 and later](https://github.com/temporalio/temporal/tree/main/schema/mysql/v8/visibility) +The following example shows how to set up your MySQL as both your persistence and Visibility store using +`temporal-sql-tool`. Refer to the +[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples +with different databases. -The following example shows how the [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script sets up your Visibility store. +{/* SNIPSTART compose-mysql-setup */} +[compose/scripts/setup-mysql.sh](https://github.com/temporalio/samples-server/blob/docker-image-restructure-snipsync/compose/scripts/setup-mysql.sh) +```sh +set -eu -```bash -#... -# set your MySQL environment variables -: "${DBNAME:=temporal}" -: "${VISIBILITY_DBNAME:=temporal_visibility}" -: "${DB_PORT:=}" -: "${MYSQL_SEEDS:=}" -: "${MYSQL_USER:=}" -: "${MYSQL_PWD:=}" -: "${MYSQL_TX_ISOLATION_COMPAT:=false}" +echo 'Starting MySQL schema setup...' +echo 'Waiting for MySQL port to be available...' +nc -z -w 10 mysql 3306 +echo 'MySQL port is available' -#... -# set connection details -#... -# set up MySQL schema -setup_mysql_schema() { - #... - # use valid schema for the version of the database you want to set up for Visibility - VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/mysql/${MYSQL_VERSION_DIR}/visibility/versioned - if [[ ${SKIP_DB_CREATE} != true ]]; then - temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" create - fi - temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" setup-schema -v 0.0 - temporal-sql-tool --ep "${MYSQL_SEEDS}" -u "${MYSQL_USER}" -p "${DB_PORT}" "${MYSQL_CONNECT_ATTR[@]}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}" -#... -} +# Create and setup temporal database +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal create +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal setup-schema -v 0.0 +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned + +# Create and setup visibility database +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal_visibility create +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal_visibility setup-schema -v 0.0 +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal_visibility update-schema -d /etc/temporal/schema/mysql/v8/visibility/versioned + +echo 'MySQL schema setup complete' ``` +{/* SNIPEND */} -Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) to run the setup. +Note that the script uses +[temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) +to run the setup. ## How to set up PostgreSQL Visibility store {#postgresql} @@ -151,20 +169,25 @@ Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temp - PostgreSQL v9.6 and later. - With Temporal Service version 1.20 and later, advanced Visibility is available on PostgreSQL v12 and later. -- Support for PostgreSQL v9.6 through v11 will be deprecated for all Temporal Server versions after v1.20; we recommend upgrading to PostgreSQL 12 or later. +- Support for PostgreSQL v9.6 through v11 will be deprecated for all Temporal Server versions after v1.20; we recommend + upgrading to PostgreSQL 12 or later. ::: -You can set PostgreSQL as your [Visibility store](/temporal-service/visibility). -Verify [supported versions](/self-hosted-guide/visibility) before you proceed. +You can set PostgreSQL as your [Visibility store](/temporal-service/visibility). Verify +[supported versions](/self-hosted-guide/visibility) before you proceed. -If using PostgreSQL v12 or later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. +If using PostgreSQL v12 or later as your Visibility store with Temporal Server v1.20 and later, any +[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a +Namespace in that Temporal Service. -**Persistence configuration** +### Persistence configuration -Set your PostgreSQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. +Set your PostgreSQL Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then +define the Visibility store configuration under `datastores`. -The following example shows how to set a Visibility store `postgres-visibility` and define the datastore configuration in your Temporal Service configuration YAML. +The following example shows how to set a Visibility store `postgres-visibility` and define the datastore configuration +in your Temporal Service configuration YAML. ```yaml #... @@ -189,43 +212,47 @@ persistence: #... ``` -To enable advanced Visibility features on your PostgreSQL Visibility store, upgrade to PostgreSQL v12 or later with Temporal Server v1.20 or later. -See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) for details on how to upgrade your Temporal Server and database schemas. +To enable advanced Visibility features on your PostgreSQL Visibility store, upgrade to PostgreSQL v12 or later with +Temporal Server v1.20 or later. See [Upgrade Server](/self-hosted-guide/upgrade-server#upgrade-server) for details on +how to upgrade your Temporal Server and database schemas. -**Database schema and setup** +### Database schema and setup -Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions): +Visibility data is stored in a database table called `executions_visibility` and must be created using the schema for +[PostgreSQL v12 and later](https://github.com/temporalio/temporal/tree/main/schema/postgresql/v12/visibility) -- [PostgreSQL v12 and later](https://github.com/temporalio/temporal/tree/main/schema/postgresql/v12/visibility) +The following example shows how to set up your PostgreSQL as both persistence and Visibility store using +`temporal-sql-tool`. Refer to the +[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts) for more examples +with different databases. -The following example shows how the [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script sets up your PostgreSQL Visibility store. +{/* SNIPSTART compose-postgres-setup */} +[compose/scripts/setup-postgres.sh](https://github.com/temporalio/samples-server/blob/docker-image-restructure-snipsync/compose/scripts/setup-postgres.sh) +```sh +set -eu -```bash -#... -# set your PostgreSQL environment variables -: "${DBNAME:=temporal}" -: "${VISIBILITY_DBNAME:=temporal_visibility}" -: "${DB_PORT:=}" -: "${POSTGRES_SEEDS:=}" -: "${POSTGRES_USER:=}" -: "${POSTGRES_PWD:=}" +echo 'Starting PostgreSQL schema setup...' +echo 'Waiting for PostgreSQL port to be available...' +nc -z -w 10 postgresql 5432 +echo 'PostgreSQL port is available' -#... set connection details -# set up PostgreSQL schema -setup_postgres_schema() { - #... +# Create and setup temporal database +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal create +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal setup-schema -v 0.0 +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal update-schema -d /etc/temporal/schema/postgresql/v12/temporal/versioned - # use valid schema for the version of the database you want to set up for Visibility - VISIBILITY_SCHEMA_DIR=${TEMPORAL_HOME}/schema/postgresql/${POSTGRES_VERSION_DIR}/visibility/versioned - if [[ ${VISIBILITY_DBNAME} != "${POSTGRES_USER}" && ${SKIP_DB_CREATE} != true ]]; then - temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${VISIBILITY_DBNAME}" create - fi - temporal-sql-tool --plugin postgres --ep "${POSTGRES_SEEDS}" -u "${POSTGRES_USER}" -p "${DB_PORT}" --db "${VISIBILITY_DBNAME}" update-schema -d "${VISIBILITY_SCHEMA_DIR}" - #... -} +# Create and setup visibility database +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal_visibility create +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal_visibility setup-schema -v 0.0 +temporal-sql-tool --plugin postgres12 --ep postgresql -u temporal -p 5432 --db temporal_visibility update-schema -d /etc/temporal/schema/postgresql/v12/visibility/versioned + +echo 'PostgreSQL schema setup complete' ``` +{/* SNIPEND */} -Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) to run the setup. +Note that the script uses +[temporal-sql-tool](https://github.com/temporalio/temporal/blob/3b982585bf0124839e697952df4bba01fe4d9543/tools/sql/main.go) +to run the setup. ## How to set up SQLite Visibility store {#sqlite} @@ -235,21 +262,27 @@ Note that the script uses [temporal-sql-tool](https://github.com/temporalio/temp ::: -You can set SQLite as your [Visibility store](/temporal-service/visibility). -Verify [supported versions](/self-hosted-guide/visibility) before you proceed. +You can set SQLite as your [Visibility store](/temporal-service/visibility). Verify +[supported versions](/self-hosted-guide/visibility) before you proceed. -Temporal supports only an in-memory database with SQLite; this means that the database is automatically created when Temporal Server starts and is destroyed when Temporal Server stops. +Temporal supports only an in-memory database with SQLite; this means that the database is automatically created when +Temporal Server starts and is destroyed when Temporal Server stops. You can change the configuration to use a file-based database so that it is preserved when Temporal Server stops. -However, if you use a file-based SQLite database, upgrading your database schema to enable advanced Visibility features is not supported; in this case, you must delete the database and create it again to upgrade. +However, if you use a file-based SQLite database, upgrading your database schema to enable advanced Visibility features +is not supported; in this case, you must delete the database and create it again to upgrade. -If using SQLite v3.31.0 and later as your Visibility store with Temporal Server v1.20 and later, any [custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a Namespace in that Temporal Service. +If using SQLite v3.31.0 and later as your Visibility store with Temporal Server v1.20 and later, any +[custom Search Attributes](/search-attribute#custom-search-attribute) that you create must be associated with a +Namespace in that Temporal Service. -**Persistence configuration** +### Persistence configuration -Set your SQLite Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. +Set your SQLite Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then +define the Visibility store configuration under `datastores`. -The following example shows how to set a Visibility store `sqlite-visibility` and define the datastore configuration in your Temporal Service configuration YAML. +The following example shows how to set a Visibility store `sqlite-visibility` and define the datastore configuration in +your Temporal Service configuration YAML. ```yaml persistence: @@ -283,36 +316,45 @@ persistence: SQLite (v3.31.0 and later) has advanced Visibility enabled by default. -**Database schema and setup** +### Database schema and setup -Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions) in https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql. +Visibility data is stored in a database table called `executions_visibility` that must be set up according to the +schemas defined (by supported versions) in +https://github.com/temporalio/temporal/blob/main/schema/sqlite/v3/visibility/schema.sql. -For an example of setting up the SQLite schema, see [Temporalite](https://github.com/temporalio/temporalite/blob/main/server.go) setup. +For an example of setting up the SQLite schema, see +[Temporalite](https://github.com/temporalio/temporalite/blob/main/server.go) setup. ## How to set up Cassandra Visibility store {#cassandra} :::tip Support, stability, and dependency info -- Support for Cassandra as a Visibility database is deprecated beginning with Temporal Server v1.21. For updates, check the [Temporal Server release notes](https://github.com/temporalio/temporal/releases). +- Support for Cassandra as a Visibility database is deprecated beginning with Temporal Server v1.21. For updates, check + the [Temporal Server release notes](https://github.com/temporalio/temporal/releases). - We recommend migrating from Cassandra to any of the other supported databases for Visibility. ::: -You can set Cassandra as your [Visibility store](/temporal-service/visibility). -Verify [supported versions](/self-hosted-guide/visibility) before you proceed. +You can set Cassandra as your [Visibility store](/temporal-service/visibility). Verify +[supported versions](/self-hosted-guide/visibility) before you proceed. Advanced Visibility is not supported with Cassandra. -To enable advanced Visibility features, use any of the supported databases, such as MySQL, PostgreSQL, SQLite, or Elasticsearch, as your Visibility store. -We recommend using Elasticsearch for any Temporal Service setup that handles more than a few Workflow Executions because it supports the request load on the Visibility store and helps optimize performance. +To enable advanced Visibility features, use any of the supported databases, such as MySQL, PostgreSQL, SQLite, or +Elasticsearch, as your Visibility store. We recommend using Elasticsearch for any Temporal Service setup that handles +more than a few Workflow Executions because it supports the request load on the Visibility store and helps optimize +performance. -To migrate from Cassandra to a supported SQL database, see [Migrating Visibility database](#migrating-visibility-database). +To migrate from Cassandra to a supported SQL database, see +[Migrating Visibility database](#migrating-visibility-database). -**Persistence configuration** +### Persistence configuration -Set your Cassandra Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. +Set your Cassandra Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then +define the Visibility store configuration under `datastores`. -The following example shows how to set a Visibility store `cass-visibility` and define the datastore configuration in your Temporal Service configuration YAML. +The following example shows how to set a Visibility store `cass-visibility` and define the datastore configuration in +your Temporal Service configuration YAML. ```yaml #... @@ -330,11 +372,14 @@ persistence: #... ``` -**Database schema and setup** +### Database schema and setup -Visibility data is stored in a database table called `executions_visibility` that must be set up according to the schemas defined (by supported versions) in https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility. +Visibility data is stored in a database table called `executions_visibility` that must be set up according to the +schemas defined (by supported versions) in https://github.com/temporalio/temporal/tree/main/schema/cassandra/visibility. -The following example shows how the [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script sets up your Visibility store. +The following example shows how to set up your Cassandra Visibility store using `temporal-cassandra-tool`. For more +examples with different databases, refer to the +[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts). ```bash #... @@ -370,16 +415,19 @@ setup_cassandra_schema() { ## How to integrate Elasticsearch into a Temporal Service {#elasticsearch} -You can integrate Elasticsearch with your Temporal Service as your Visibility store. -We recommend using Elasticsearch for large-scale operations on the Temporal Service. +You can integrate Elasticsearch with your Temporal Service as your Visibility store. We recommend using Elasticsearch +for large-scale operations on the Temporal Service. -To integrate Elasticsearch with your Temporal Service, edit the `persistence` section of your `development.yaml` configuration file to add Elasticsearch as the `visibilityStore`, and run the index schema setup commands. +To integrate Elasticsearch with your Temporal Service, edit the `persistence` section of your `development.yaml` +configuration file to add Elasticsearch as the `visibilityStore`, and run the index schema setup commands. -**Persistence configuration** +### Persistence configuration -Set your Elasticsearch Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and then define the Visibility store configuration under `datastores`. +Set your Elasticsearch Visibility store name in the `visibilityStore` parameter in your Persistence configuration, and +then define the Visibility store configuration under `datastores`. -The following example shows how to set a Visibility store named `es-visibility` and define the datastore configuration in your Temporal Service configuration YAML. +The following example shows how to set a Visibility store named `es-visibility` and define the datastore configuration +in your Temporal Service configuration YAML. ```yaml persistence: @@ -397,60 +445,102 @@ persistence: visibility: temporal_visibility_v1_dev ``` -**Index schema and index** - -The following example shows how the [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script sets up an Elasticsearch Visibility store. - -```bash -#... -# Elasticsearch -: "${ENABLE_ES:=false}" -: "${ES_SCHEME:=http}" -: "${ES_SEEDS:=}" -: "${ES_PORT:=9200}" -: "${ES_USER:=}" -: "${ES_PWD:=}" -: "${ES_VERSION:=v7}" -: "${ES_VIS_INDEX:=temporal_visibility_v1}" -: "${ES_SEC_VIS_INDEX:=}" -: "${ES_SCHEMA_SETUP_TIMEOUT_IN_SECONDS:=0}" -#... -# Validate your ES environment -#... -# Wait for ES to start -#... -# ES_SERVER is the URL of Elasticsearch server; for example, "http://localhost:9200". -SETTINGS_URL="${ES_SERVER}/_cluster/settings" -SETTINGS_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/cluster_settings_${ES_VERSION}.json -TEMPLATE_URL="${ES_SERVER}/_template/temporal_visibility_v1_template" -SCHEMA_FILE=${TEMPORAL_HOME}/schema/elasticsearch/visibility/index_template_${ES_VERSION}.json -INDEX_URL="${ES_SERVER}/${ES_VIS_INDEX}" -curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${SETTINGS_URL}" -H "Content-Type: application/json" --data-binary "@${SETTINGS_FILE}" --write-out "\n" -curl --fail --user "${ES_USER}":"${ES_PWD}" -X PUT "${TEMPLATE_URL}" -H 'Content-Type: application/json' --data-binary "@${SCHEMA_FILE}" --write-out "\n" -curl --user "${ES_USER}":"${ES_PWD}" -X PUT "${INDEX_URL}" --write-out "\n" +### Index schema and index + +To set up Elasticsearch as your Visibility store, use the `temporal-elasticsearch-tool` available in the +`temporalio/admin-tools` image. + +The following example shows how to set up an Elasticsearch Visibility store with a MySQL persistence store using +`temporal-elasticsearch-tool`. For more examples with different databases, refer to the +[samples-server repository](https://github.com/temporalio/samples-server/tree/main/compose/scripts). + +{/* SNIPSTART compose-mysql-es-setup */} +[compose/scripts/setup-mysql-es.sh](https://github.com/temporalio/samples-server/blob/docker-image-restructure-snipsync/compose/scripts/setup-mysql-es.sh) +```sh +set -eu + +# Validate required environment variables +: "${ES_SCHEME:?ERROR: ES_SCHEME environment variable is required}" +: "${ES_HOST:?ERROR: ES_HOST environment variable is required}" +: "${ES_PORT:?ERROR: ES_PORT environment variable is required}" +: "${ES_VISIBILITY_INDEX:?ERROR: ES_VISIBILITY_INDEX environment variable is required}" +: "${ES_VERSION:?ERROR: ES_VERSION environment variable is required}" + +echo 'Starting MySQL and Elasticsearch schema setup...' +echo 'Waiting for MySQL port to be available...' +nc -z -w 10 mysql 3306 +echo 'MySQL port is available' + +# Create and setup temporal database +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal create +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal setup-schema -v 0.0 +temporal-sql-tool --plugin mysql8 --ep mysql -u root -p 3306 --db temporal update-schema -d /etc/temporal/schema/mysql/v8/temporal/versioned + +# Setup Elasticsearch index +# temporal-elasticsearch-tool is available in v1.30+ server releases +if [ -x /usr/local/bin/temporal-elasticsearch-tool ]; then + echo 'Using temporal-elasticsearch-tool for Elasticsearch setup' + temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" setup-schema + temporal-elasticsearch-tool --ep "$ES_SCHEME://$ES_HOST:$ES_PORT" create-index --index $ES_VISIBILITY_INDEX +else + echo 'Using curl for Elasticsearch setup' + echo 'WARNING: curl will be removed from admin-tools in v1.30.' + echo 'Waiting for Elasticsearch to be ready...' + max_attempts=30 + attempt=0 + until curl -s -f "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s"; do + attempt=$((attempt + 1)) + if [ $attempt -ge $max_attempts ]; then + echo "ERROR: Elasticsearch did not become ready after $max_attempts attempts" + echo "Last error from curl:" + curl "$ES_SCHEME://$ES_HOST:$ES_PORT/_cluster/health?wait_for_status=yellow&timeout=1s" 2>&1 || true + exit 1 + fi + echo "Elasticsearch not ready yet, waiting... (attempt $attempt/$max_attempts)" + sleep 2 + done + echo '' + echo 'Elasticsearch is ready' + echo 'Creating index template...' + curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/_template/temporal_visibility_v1_template" -H 'Content-Type: application/json' --data-binary "@/etc/temporal/schema/elasticsearch/visibility/index_template_$ES_VERSION.json" + echo '' + echo 'Creating index...' + curl --head --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" 2>/dev/null || curl -X PUT --fail "$ES_SCHEME://$ES_HOST:$ES_PORT/$ES_VISIBILITY_INDEX" + echo '' +fi + +echo 'MySQL and Elasticsearch setup complete' ``` +{/* SNIPEND */} -**Elasticsearch privileges** +### Elasticsearch privileges Ensure that the following privileges are granted for the Elasticsearch Temporal index: - **Read** - - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): `create`, `index`, `delete`, `read` + - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): + `create`, `index`, `delete`, `read` - **Write** - - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): `write` + - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): + `write` - **Custom Search Attributes** - - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): `manage` - - [cluster privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster): `monitor` or `manage`. + - [index privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices): + `manage` + - [cluster privileges](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-cluster): + `monitor` or `manage`. ## How to set up Dual Visibility {#dual-visibility} -To enable [Dual Visibility](/dual-visibility), set up a secondary Visibility store with your primary Visibility store, and configure your Temporal Service to enable read and/or write operations on the secondary Visibility store. +To enable [Dual Visibility](/dual-visibility), set up a secondary Visibility store with your primary Visibility store, +and configure your Temporal Service to enable read and/or write operations on the secondary Visibility store. -With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to write to primary only, secondary only, or to both primary and secondary stores. +With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Service to +write to primary only, secondary only, or to both primary and secondary stores. #### Set up secondary Visibility store -Set the secondary store with the `secondaryVisibilityStore` configuration key in your Persistence configuration, and then define the secondary Visibility store configuration under `datastores`. +Set the secondary store with the `secondaryVisibilityStore` configuration key in your Persistence configuration, and +then define the secondary Visibility store configuration under `datastores`. You can configure any of the [supported databases](/self-hosted-guide/visibility) as your secondary store. @@ -477,7 +567,8 @@ persistence: password: 'temporal' ``` -To configure Elasticsearch as both your primary and secondary store, use the configuration key `elasticsearch.indices.secondary_visibility`, as shown in the following example. +To configure Elasticsearch as both your primary and secondary store, use the configuration key +`elasticsearch.indices.secondary_visibility`, as shown in the following example. ```yaml persistence: @@ -564,7 +655,8 @@ setup_mysql_schema() { } ``` -For Elasticsearch as both primary and secondary Visibility store configuration in the previous example, an example setup script would be as follows. +For Elasticsearch as both primary and secondary Visibility store configuration in the previous example, an example setup +script would be as follows. ```bash #... @@ -609,9 +701,12 @@ setup_es_index() { #### Update Temporal Service configuration -With the primary and secondary stores set, update the `system.secondaryVisibilityWritingMode` and `system.enableReadFromSecondaryVisibility` configuration keys in your self-hosted Temporal Service's dynamic configuration YAML file to enable read and/or write operations to the secondary Visibility store. +With the primary and secondary stores set, update the `system.secondaryVisibilityWritingMode` and +`system.enableReadFromSecondaryVisibility` configuration keys in your self-hosted Temporal Service's dynamic +configuration YAML file to enable read and/or write operations to the secondary Visibility store. -For example, to enable write operations to both primary and secondary stores, but disable reading from the secondary store, use the following. +For example, to enable write operations to both primary and secondary stores, but disable reading from the secondary +store, use the following. ```yaml system.secondaryVisibilityWritingMode: @@ -629,7 +724,9 @@ For details on the configuration options, see: ## How to migrate Visibility database {#migrating-visibility-database} -To migrate your Visibility database, [set up a secondary Visibility store](#dual-visibility) to enable [Dual Visibility](/dual-visibility), and update the dynamic configuration in your Temporal Service to update the read and write operations for the Visibility store. +To migrate your Visibility database, [set up a secondary Visibility store](#dual-visibility) to enable +[Dual Visibility](/dual-visibility), and update the dynamic configuration in your Temporal Service to update the read +and write operations for the Visibility store. Dual Visibility setup is optional but useful in gradually migrating your Visibility data to another database. @@ -637,14 +734,17 @@ Before you begin, verify [supported databases and versions](/self-hosted-guide/v The following steps describe how to migrate your Visibility database. -After you make any changes to your [Temporal Service configuration](/temporal-service/configuration), ensure that you restart your services. +After you make any changes to your [Temporal Service configuration](/temporal-service/configuration), ensure that you +restart your services. #### Set up secondary Visibility store -1. In your Temporal Service configuration, [add a secondary Visibility store](/references/configuration#secondaryvisibilitystore) to your Visibility setup under the Persistence configuration. +1. In your Temporal Service configuration, + [add a secondary Visibility store](/references/configuration#secondaryvisibilitystore) to your Visibility setup under + the Persistence configuration. - Example: To migrate from Cassandra to Elasticsearch, add Elasticsearch as your secondary database and set it up. - For details, see [secondary Visibility database schema and setup](#dual-visibility). + Example: To migrate from Cassandra to Elasticsearch, add Elasticsearch as your secondary database and set it up. For + details, see [secondary Visibility database schema and setup](#dual-visibility). ```yaml persistence: @@ -667,8 +767,8 @@ After you make any changes to your [Temporal Service configuration](/temporal-se closeIdleConnectionsInterval: 15s ``` -1. Update the [dynamic configuration](/temporal-service/configuration#dynamic-configuration) keys on your self-hosted Temporal Service to enable write operations to the secondary store and disable read operations. - Example: +1. Update the [dynamic configuration](/temporal-service/configuration#dynamic-configuration) keys on your self-hosted + Temporal Service to enable write operations to the secondary store and disable read operations. Example: ```yaml system.secondaryVisibilityWritingMode: @@ -679,35 +779,39 @@ After you make any changes to your [Temporal Service configuration](/temporal-se constraints: {} ``` -At this point, Visibility data is read from the primary store, and all Visibility data is written to both the primary and secondary store. -This setting applies only to new Visibility data generated after Dual Visibility is enabled. -It does not migrate any existing data in the primary store to the secondary store. +At this point, Visibility data is read from the primary store, and all Visibility data is written to both the primary +and secondary store. This setting applies only to new Visibility data generated after Dual Visibility is enabled. It +does not migrate any existing data in the primary store to the secondary store. -For details on write options to the secondary store, see [Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings). +For details on write options to the secondary store, see +[Secondary Visibility dynamic configuration reference](/references/dynamic-configuration#secondary-visibility-settings). #### Run in dual mode -When you enable a secondary store, only new Visibility data is written to both primary and secondary stores. -The primary store still holds the Workflow Execution data from before the secondary store was set up. +When you enable a secondary store, only new Visibility data is written to both primary and secondary stores. The primary +store still holds the Workflow Execution data from before the secondary store was set up. -Running in dual mode lets you plan for closed and open Workflow Executions data from before the secondary store was set up in your self-hosted Temporal Service. +Running in dual mode lets you plan for closed and open Workflow Executions data from before the secondary store was set +up in your self-hosted Temporal Service. Example: -- To manage closed Workflow Executions data, run in dual mode until the Namespace [Retention Period](/temporal-service/temporal-server#retention-period) is reached. - After the Retention Period, Workflow Execution data is removed from the Persistence and Visibility stores. - If you want to keep the closed Workflow Executions data after the set Retention Period, you must set up [Archival](/self-hosted-guide/archival). -- To manage data for all open Workflow Executions, run in dual mode until all the Workflow Executions started before enabling Dual Visibility mode are closed. - After the Workflow Executions are closed, verify the Retention Period and set up Archival if you need to keep the data beyond the Retention Period. +- To manage closed Workflow Executions data, run in dual mode until the Namespace + [Retention Period](/temporal-service/temporal-server#retention-period) is reached. After the Retention Period, + Workflow Execution data is removed from the Persistence and Visibility stores. If you want to keep the closed Workflow + Executions data after the set Retention Period, you must set up [Archival](/self-hosted-guide/archival). +- To manage data for all open Workflow Executions, run in dual mode until all the Workflow Executions started before + enabling Dual Visibility mode are closed. After the Workflow Executions are closed, verify the Retention Period and + set up Archival if you need to keep the data beyond the Retention Period. -You can run your Visibility setup in dual mode for an indefinite period, or until you are ready to deprecate the primary store and move completely to the secondary store without losing data. +You can run your Visibility setup in dual mode for an indefinite period, or until you are ready to deprecate the primary +store and move completely to the secondary store without losing data. #### Deprecate primary Visibility store When you are ready to deprecate your primary store, follow these steps. -1. Update the dynamic configuration YAML to enable read operations from the secondary store. - Example: +1. Update the dynamic configuration YAML to enable read operations from the secondary store. Example: ```yaml system.secondaryVisibilityWritingMode: @@ -718,11 +822,12 @@ When you are ready to deprecate your primary store, follow these steps. constraints: {} ``` - At this point, Visibility data is read from the secondary store only. - Verify whether data on the secondary store is correct. + At this point, Visibility data is read from the secondary store only. Verify whether data on the secondary store is + correct. -1. When the secondary store is vetted and ready to replace your current primary store, change your Temporal Service configuration to set the secondary store as your primary, and remove the dynamic configuration set in the previous steps. - Example: +1. When the secondary store is vetted and ready to replace your current primary store, change your Temporal Service + configuration to set the secondary store as your primary, and remove the dynamic configuration set in the previous + steps. Example: ```yaml persistence: @@ -742,33 +847,40 @@ When you are ready to deprecate your primary store, follow these steps. ## Managing custom Search Attributes {#custom-search-attributes} -To manage your custom Search Attributes on Temporal Cloud, use `tcld`. -With Temporal Cloud, you can create and rename custom Search Attributes. +To manage your custom Search Attributes on Temporal Cloud, use `tcld`. With Temporal Cloud, you can create and rename +custom Search Attributes. -To manage your custom Search Attributes on self-hosted Temporal Clusters, use Temporal CLI. With self-hosted Temporal Service, you can create and remove custom Search Attributes. -Note that if you use [SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, creating a custom Search Attribute creates a mapping with a database field name in the Visibility store `custom_search_attributes` table. -Removing a custom Search Attribute removes this mapping with the database field name but does not remove the data. -If you remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database field of the one that was recently removed. -This might cause unexpected results when you use the List API to retrieve results using the new custom Search Attribute. -These constraints do not apply if you use Elasticsearch. +To manage your custom Search Attributes on self-hosted Temporal Clusters, use Temporal CLI. With self-hosted Temporal +Service, you can create and remove custom Search Attributes. Note that if you use +[SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, creating a custom Search Attribute +creates a mapping with a database field name in the Visibility store `custom_search_attributes` table. Removing a custom +Search Attribute removes this mapping with the database field name but does not remove the data. If you remove a custom +Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database field of the one +that was recently removed. This might cause unexpected results when you use the List API to retrieve results using the +new custom Search Attribute. These constraints do not apply if you use Elasticsearch. ### How to create custom Search Attributes {#create-custom-search-attributes} -Add custom Search Attributes to your Visibility store using the Temporal CLI for a self-hosted Temporal Service and `tcld` for Temporal Cloud. +Add custom Search Attributes to your Visibility store using the Temporal CLI for a self-hosted Temporal Service and +`tcld` for Temporal Cloud. -Creating a custom Search Attribute in your Visibility store makes it available to use in your Workflow metadata and [List Filters](/list-filter). +Creating a custom Search Attribute in your Visibility store makes it available to use in your Workflow metadata and +[List Filters](/list-filter). **On Temporal Cloud** -To create custom Search Attributes on Temporal Cloud, use [`tcld namespace search-attributes add`](/cloud/tcld/namespace/#search-attributes). -For example, to add a custom Search Attributes "CustomSA" to your Temporal Cloud Namespace "YourNamespace", run the following command. +To create custom Search Attributes on Temporal Cloud, use +[`tcld namespace search-attributes add`](/cloud/tcld/namespace/#search-attributes). For example, to add a custom Search +Attributes "CustomSA" to your Temporal Cloud Namespace "YourNamespace", run the following command. `tcld namespace search-attributes add --namespace YourNamespace --search-attribute "CustomSA"` **On self-hosted Temporal Service** -If you're self-hosting your Temporal Service, verify whether your [Visibility database](/self-hosted-guide/visibility) version supports advanced Visibility features. +If you're self-hosting your Temporal Service, verify whether your [Visibility database](/self-hosted-guide/visibility) +version supports advanced Visibility features. -To create custom Search Attributes in your self-hosted Temporal Service Visibility store, use `temporal operator search-attribute create` with `--name` and `--type` command options. +To create custom Search Attributes in your self-hosted Temporal Service Visibility store, use +`temporal operator search-attribute create` with `--name` and `--type` command options. For example, to create a Search Attribute called `CustomSA` of type `Keyword`, run the following command: @@ -776,8 +888,8 @@ For example, to create a Search Attribute called `CustomSA` of type `Keyword`, r temporal operator search-attribute create --name="CustomSA" --type="Keyword" ``` -Note that if you use a SQL database with advanced Visibility capabilities, you are required to specify a Namespace when creating a custom Search Attribute. -For example: +Note that if you use a SQL database with advanced Visibility capabilities, you are required to specify a Namespace when +creating a custom Search Attribute. For example: ``` temporal operator search-attribute create --name="CustomSA" --type="Keyword" --namespace="yournamespace" @@ -785,7 +897,8 @@ temporal operator search-attribute create --name="CustomSA" --type="Keyword" --n You can also create multiple custom Search Attributes when you set up your Visibility store. -For example, the [auto-setup.sh](https://github.com/temporalio/docker-builds/blob/main/docker/auto-setup.sh) script that is used to set up your local [docker-compose Temporal Service](https://github.com/temporalio/docker-compose) creates custom Search Attributes in the Visibility store, as shown in the following code snippet from the script (for SQL databases). +The following example shows how custom Search Attributes can be created during Visibility store setup for SQL databases. +For setup examples, refer to the [samples-server repository](https://github.com/temporalio/samples-server) ```bash add_custom_search_attributes() { @@ -808,9 +921,8 @@ add_custom_search_attributes() { } ``` -Note that this script has been updated for Temporal Server v1.20, which requires associating every custom Search Attribute with a Namespace when using a SQL database. - -For Temporal Server v1.19 and earlier, or if using Elasticsearch for advanced Visibility, you can create custom Search Attributes without a Namespace association, as shown in the following example. +For Temporal Server v1.19 and earlier, or if using Elasticsearch for advanced Visibility, you can create custom Search +Attributes without a Namespace association, as shown in the following example. {/* CHECK FOR ACCURACY */} @@ -828,21 +940,24 @@ add_custom_search_attributes() { } ``` -When your Visibility store is set up and running, these custom Search Attributes are available to use in your Workflow code. +When your Visibility store is set up and running, these custom Search Attributes are available to use in your Workflow +code. ### How to remove custom Search Attributes {#remove-custom-search-attributes} -To remove a Search Attribute key from your self-hosted Temporal Service Visibility store, use the command `temporal operator search-attribute remove`. -Removing Search Attributes is not supported on Temporal Cloud. +To remove a Search Attribute key from your self-hosted Temporal Service Visibility store, use the command +`temporal operator search-attribute remove`. Removing Search Attributes is not supported on Temporal Cloud. -For example, if using Elasticsearch for advanced Visibility, to remove a custom Search Attribute called `CustomSA` of type Keyword use the following command: +For example, if using Elasticsearch for advanced Visibility, to remove a custom Search Attribute called `CustomSA` of +type Keyword use the following command: ``` temporal operator search-attribute remove \ --name="your_custom_attribute" ``` -With Temporal Server v1.20, if using a SQL database for advanced Visibility, you need to specify the Namespace in your command, as shown in the following command: +With Temporal Server v1.20, if using a SQL database for advanced Visibility, you need to specify the Namespace in your +command, as shown in the following command: ``` temporal operator search-attribute remove \ @@ -858,15 +973,16 @@ temporal operator search-attribute list and check the list. -If you're on Temporal Server v1.20 and later, specify the Namespace from which you removed the Search Attribute. -For example, +If you're on Temporal Server v1.20 and later, specify the Namespace from which you removed the Search Attribute. For +example, ``` temporal search-attribute list --namespace="yournamespace" ``` -Note that if you use [SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, a new custom Search Attribute is mapped to a database field name in the Visibility store `custom_search_attributes` table. -Removing this custom Search Attribute removes the mapping with the database field name but does not remove the data. -If you remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database field of the one that was recently removed. -This might cause unexpected results when you use the List API to retrieve results using the new custom Search Attribute. -These constraints do not apply if you use Elasticsearch. +Note that if you use [SQL databases](/self-hosted-guide/visibility) with Temporal Server v1.20 and later, a new custom +Search Attribute is mapped to a database field name in the Visibility store `custom_search_attributes` table. Removing +this custom Search Attribute removes the mapping with the database field name but does not remove the data. If you +remove a custom Search Attribute and add a new one, the new custom Search Attribute might be mapped to the database +field of the one that was recently removed. This might cause unexpected results when you use the List API to retrieve +results using the new custom Search Attribute. These constraints do not apply if you use Elasticsearch. diff --git a/docs/references/dynamic-configuration.mdx b/docs/references/dynamic-configuration.mdx index 9b4fb21e69..2675439e64 100644 --- a/docs/references/dynamic-configuration.mdx +++ b/docs/references/dynamic-configuration.mdx @@ -125,10 +125,7 @@ Not defining constraints on a dynamic configuration key sets the values across t {/* Note that the values set with most constraints take priority over values that are set with fewer constraints, regardless of the order in which they are set in the dynamic configuration key. */} -For more examples on how dynamic configuration is set, see: - -- [docker-compose](https://github.com/temporalio/docker-compose/tree/main/dynamicconfig) -- [samples-server](https://github.com/temporalio/samples-server/blob/main/tls/config/dynamicconfig/development.yaml) +For more examples on how dynamic configuration is set, see [samples-server](https://github.com/temporalio/samples-server/blob/main/tls/config/dynamicconfig/development.yaml). ## Commonly used dynamic configuration keys diff --git a/docs/web-ui.mdx b/docs/web-ui.mdx index aee2192560..0a9d2967fb 100644 --- a/docs/web-ui.mdx +++ b/docs/web-ui.mdx @@ -13,8 +13,7 @@ tags: import { RelatedReadContainer, RelatedReadItem } from '@site/src/components'; The Temporal Web UI provides users with Workflow Execution state and metadata for debugging purposes. It ships with -every [Temporal CLI](/cli) release and [Docker Compose](https://github.com/temporalio/docker-compose) update and is -available with [Temporal Cloud](/cloud). +every [Temporal CLI](/cli) release and is available with [Temporal Cloud](/cloud). You can configure the Temporal Web UI to work in your own environment. See the [UI configuration reference](/references/web-ui-configuration). diff --git a/snipsync.config.yaml b/snipsync.config.yaml index cffe0de3fb..dd00e8aca2 100644 --- a/snipsync.config.yaml +++ b/snipsync.config.yaml @@ -35,6 +35,9 @@ origins: owner: 'temporalio' repo: 'documentation' ref: 'main' + - owner: temporalio + repo: samples-server + ref: 'main' targets: - docs