Data conversion

This guide provides an overview of data handling using a Data Converter on the Temporal Platform.

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster. It is used by the Temporal SDK framework to serialize/deserialize data such as input and output of Activities and Workflows that need to be sent over the wire to the Temporal Cluster.

Data Converter encodes and decodes data

The Data Converter encodes data from your application to a PayloadWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more before it is sent to the Temporal Cluster in the Client call. When the Temporal Server sends the encoded data back to the Worker, the Data Converter decodes it for processing within your application. This technique ensures that all your sensitive data exists in its original format only on hosts that you control.

The main pieces of data that run through the Data Converter are arguments and return values:

• The Client:
  • Encodes Workflow, Signal, and Query arguments.
  • Decodes Workflow and Query return values.
• The Worker:
  • Decodes Workflow, Signal, and Query arguments.
  • Encodes Workflow and Query return values.
  • Decodes and encodes Activity arguments and return values.

Each piece of data (like a single argument or return value) is encoded as a PayloadWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more Protobuf message, which consists of binary data and key-value metadata.

For details, see the API references:

• Go
• Java
• Python
• TypeScript

Payload

A Payload represents binary data such as input and output from Activities and Workflows. Payloads contain metadata that describe the binary data, such as its data type or other arbitrary values for use by custom encoders/converters.

When processed through the SDK, the default Data ConverterWhat is a default Data Converter?

The default Data Converter is used by the Temporal SDK to convert objects into bytes using a series of Payload Converters.

Learn more serializes your data/value to a Payload before sending it to the Temporal Server. The default Data Converter processes supported type values to Payloads, and you can create a custom Payload ConverterWhat is a Payload Converter?

A Payload Converter serializes data, converting objects or values to bytes and back.

Learn more to convert your custom object types.

You can additionally apply custom codecsWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more (such as for encryption or compression) on your Payloads to wrap them into new encoded Payloads.

Default Data Converter

Each Temporal SDK includes and uses a default Data Converter. The default Data Converter converts objects to bytes using a series of Payload Converters and supports binary, Protobufs, and JSON formats. It encodes values in the following order:

• Null
• Byte array
• Protobuf JSON
• JSON

For example:

• If a value is an instance of a Protobuf message, it is encoded with proto3 JSON.
• If a value isn't null, binary, or a Protobuf, it is encoded as JSON. If any part of it is not serializable as JSON, an error is thrown.

Custom Data Converter

A custom Data Converter extends the default Data ConverterWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more with custom logic for PayloadWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more conversion or encoding.

You can create a custom Data Converter to alter formats (for example, using MessagePack instead of JSON) or add compression and encryption.

You can customize the default Data Converter behavior in two ways:

• A Payload Converter serializes data, converting objects to bytes and back. To convert custom objects or data types to PayloadsWhat is a Payload?
  
  A Payload represents binary data such as input and output from Activities and Workflows.
  
  Learn more and back, use a custom Payload Converter and set it on a Data Converter.
• A Payload Codec encodes and decodes PayloadsWhat is a Payload?
  
  A Payload represents binary data such as input and output from Activities and Workflows.
  
  Learn more, with bytes-to-bytes conversion. To use custom encryption and/or compression logic, create a custom Payload Codec with your encryption/compression logic in the encode function and your decryption/decompression logic in the decode function.

Custom Data Converters are not applied to all data; for example, Search AttributesWhat is a Search Attribute?

A Search Attribute is an indexed name used in List Filters to filter a list of Workflow Executions that have the Search Attribute in their metadata.

Learn more are simple values and persisted unencoded so they can be indexed for searching.

A customized Data Converter can have the following three components:

• Payload ConverterWhat is a Payload Converter?
  
  A Payload Converter serializes data, converting objects or values to bytes and back.
  
  Learn more
• Failure ConverterWhat is a Failure Converter?
  
  A Failure Converter converts error objects to proto Failures and back.
  
  Learn more
• Payload CodecWhat is a Payload Codec?
  
  A Payload Codec transforms an array of Payloads into another array of Payloads.
  
  Learn more

For details on how to implement custom Payload Converters in your SDK, see Custom Payload conversionHow to use custom payload conversion

Create your custom PayloadConverter and set it on a DataConverter in your Client options.

Learn more.

For details on how to implement custom encryption and compression in your SDK, see Data Encryption.

Payload Converter

A Payload Converter serializes data, converting values to bytes and back.

When you request a Workflow Execution through your Client and pass a data input, the input is serialized using a Data Converter that runs it through a set of Payload Converters. When your Workflow Execution starts, this data input is deserialized and passed as input to your Workflow.

For more information, see the API references.

• Go
• Java
• Python
• TypeScript

For supported values, see default Data Converter.

Custom payload conversion

If you use custom objects or types that are not supported by the Payload Converters provided in the SDKs, you can create a custom Payload Converter and configure the Data Converter with it to run the specific conversions.

You can set multiple encoding Payload Converters to run your conversions. When the Data Converter receives a value for conversion, it passes through each Payload Converter in sequence until the converter that handles the data type does the conversion.

For details on how to use the Payload Converter for custom data types, see Custom Payload ConversionHow to use custom payload conversion

Create your custom PayloadConverter and set it on a DataConverter in your Client options.

Learn more.

Failure Converter

A Failure Converter converts error objects to proto Failures and back. The default Failure Converter copies error messages and stack traces as plain text.

For details, see the API references.

• Go
• Java
• Python
• TypeScript

You can make a custom Failure Converter, but if you use multiple SDKs, you must implement the same logic in each. Creating a custom Failure Converter is not yet supported in Java.

Failure messages and stack traces are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures. If your errors might contain sensitive information, you can encrypt the message and stack trace by configuring the default Failure Converter to use your encoded attributes, in which case it moves your message and stack_trace fields to a Payload that's run through your codecWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more.

Payload Codec

A Payload Codec transforms an array of PayloadsWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more (for example, a list of Workflow arguments) into another array of Payloads.

The Payload Codec is an optional step that happens between the wire and the Payload ConverterWhat is a Payload Converter?

A Payload Converter serializes data, converting objects or values to bytes and back.

Learn more:

User code <--> Payload Converter <--> Payload Codec <--> Wire <--> Temporal Server

When serializing to Payloads, the Payload Converter is applied first to convert your objects to bytes, followed by codecs that convert bytes to bytes. When deserializing from Payloads, codecs are applied first to last to reverse the effect, followed by the Payload Converter.

For details, see the API references.

• Go
• Java
• Python
• TypeScript

Use a custom Payload Codec to transform your Payloads; for example, implementing compression and/or encryption on your Workflow Execution data.

Encryption

Using end-to-end encryption in your custom Data Converter ensures that sensitive application data is secure when handled by the Temporal Server.

Apply your encryption logic in a custom Payload Codec and use it locally to encrypt data. You maintain all the encryption keys, and the Temporal Server sees only encrypted data. Your data exists unencrypted only on the Client and the Worker process that is executing the Workflows and Activities, on hosts that you control.

For details, see Data encryption.

The following samples use encryption (AES GCM with 256-bit key) in a custom Data Converter:

• Go sample
• Java sample
• Python sample
• TypeScript sample

Remote data encoding

Remote data encoding is exposing your Payload Codec via HTTP endpoints to support remote encoding and decoding.

Running your encoding remotely allows you to use it with tctl to encode/decode data for several commands including tctl workflow start and with Temporal Web UI to encode and decode data in your Workflow Execution details view.

To run data encoding/decoding remotely, use a Codec ServerWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more. A Codec Server is an HTTP server that is configured to use your custom Payload Codec.

Before you use a remote data encoder to encode/decode your data, ensure that you consider all the security implications of running codecs remotely. For example, codecs that perform encryption may need to be secured to prevent decryption by untrusted callers.

Encoding data on the Web UI and tctl

You can perform some operations on your Workflow Execution using tctl and the Web UI, such as starting or sending a Signal to an active Workflow Execution using tctl or canceling a Workflow Execution from the Web UI, which might require inputs that contain sensitive data.

To encode this data, specify your Codec Server endpoints with the tctl command and configure your Web UI to use the Codec Server endpoints.

Decoding data on the Web UI and tctl

If you use custom encoding in your custom Data Converter, Payload data handled by the Temporal Cluster is encoded. Since the Web UI uses the VisibilityWhat is Visibility?

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.

Learn more database to show events and data stored on the Temporal Server, all data in the Workflow Execution History in your Web UI or tctl shows in the encoded format.

To see the original format of data in your Web UI and tctl, create a Codec ServerWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more with a remote data encoder and use the Payload Codec to decode your data locally.

Note that a remote data encoder is a separate system with access to your encryption keys and exposes APIs to encode and decode any data with the Payload Codec used. Evaluate and ensure that your remote data encoder endpoints are secured and only authorized users have access to them.

Samples:

• Go
• Java
• Python
• TypeScript

Codec Server

A Codec Server is an HTTP server that uses your custom Payload CodecWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more to encode and decode your data remotely through endpoints.

You can create a custom Payload Codec with your encoding logic (such as encryption and/or compression), and apply it to the data processed in your Workflows.

Using a custom Payload Codec in your Codec Server enables encoding and decoding data remotely through the endpoints that you expose on the Codec Server.

A Codec Server follows the Temporal Codec Server Protocol. It implements two endpoints:

• /encode
• /decode

Each endpoint receives and responds with a JSON body that has a payloads property with an array of PayloadsWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more. The endpoints run the Payloads through a Payload CodecWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more before returning them.

Most SDKs provide example Codec Server implementation samples, listed here:

• Go
• Java
• Python
• TypeScript

Usage

When using tctl or the Web UI to perform some operations on a Workflow Execution, you can configure the exposed Codec Server endpoints to remotely encode data sent to the Temporal Server and decode data received from the Temporal Server.

When you apply custom encoding with encryption or compression on your Workflow data, it is stored in the encrypted/compressed format on the Temporal Server. For details on what data is encoded, see Data encryption.

Before you use a Codec Server to encode your data, ensure that you consider all the security implications of running codecs remotely. For example, codecs that perform encryption might need to be secured to prevent decryption by untrusted callers.

Configure your Codec Server endpointsSetting Codec Server endpoints

Run a Codec Server with your Payload Codec and then configure tctl and the Web UI to use the server.

Learn more to decode the encoded data to its original format when viewed from the Web UI or tctl.

The following samples provide implementation examples for applying authentication on your Codec Server using the Go SDK.

• Codec Server
• GRPC proxy server

Setting Codec Server endpoints

To use a Codec ServerWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more, first run it with your Payload Codec and then configure tctl and the Web UI to use it.

tctl

After the Codec Server is started, provide the exposed endpoint to tctl using the --codec_endpoint global option.

For example, if you are running your Codec Server locally and expose port 8888 as your endpoint, you can run the following command to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace".

tctl --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow"  --run-id "<yourRunId>" --output "table"

Web UI

You can set the codec endpoints either in the Web UI or in the UI server configuration file before starting the UI server.

In the Web UI

Data Encoder icon

In the lower-left corner, select Data Encoder. In the codec endpoint dialog, enter the URL and port number for your codec endpoint. Refresh your Workflow Execution page to see encoded/decoded data.

In the Web UI server configuration file

Specify the codec endpoint in the Web UI server configuration file:

codec:
    endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}}

Start the UI server to use this endpoint for decoding data in Workflow Executions in the specified Namespace.