Self-hosted Temporal Workflows made easy with Northflank

In distributed systems, failures are not exceptional, they're expected. Services crash mid-transaction, deployments cause service restarts and networks can be temperamental. Building fault-resistant software to reliably handle business-critical processes, like order fulfilment or processing payments, is difficult and time-consuming. Temporal offers a robust, open-source solution, providing a fault-tolerant orchestration engine. It ensures your workflows complete by automatically handling infrastructure failures and providing tools to manage application errors.

Whilst Temporal offers a managed cloud service, self-hosting gives you full ownership over your data and configuration. This can be crucial for organisations with strict security, compliance, or regulatory needs. Northflank makes self-hosting Temporal easy by providing a ready-to-deploy IaC stack template, which encompasses all the services required to run your own Temporal cluster. You can even deploy these services into your existing cloud accounts by using Northflank's Bring Your Own Cloud (BYOC) feature.

In this article, we guide you through using the Northflank Temporal stack template to deploy a Temporal cluster.

1. Database (PostgreSQL)

   • PostgreSQL serves as Temporal's persistence layer. It stores workflow event histories, state information, task queues, and namespace configurations. It also enables Basic Visibility features (listing and basic filtering of workflows). Temporal supports multiple databases (Cassandra, MySQL), so if you'd rather use an alternative, feel free to use this stack template as a base and swap in your preferred option.

2. Temporal Server

   • The heart of the cluster.

3. Temporal UI

   • A web interface to observe active and completed workflows

4. Temporal Admin Tools

   • A set of CLI utilities for administration tasks such as managing namespaces or adjusting retention policies.

The stack template deploys the Temporal platform. To actually execute your workflows, you'll need to run your own worker processes. These are components of your application, built using one of the many Temporal SDKs, which connect to the cluster defined in the stack template. This guide will show you how to run a sample worker later on.

Before deploying Temporal on Northflank, make sure you have completed the following:

• Sign up or log in to your Northflank account and create or select a team
• Optional: If you want to deploy Temporal into your existing AWS/GCP/Azure account, configure Northflank’s BYOC.
• Optional: Install Node.js, npm/Yarn, Git and the Northflank CLI, if you'd like to execute the sample TypeScript worker towards the end of this article.

To deploy Temporal using the stack template, please complete the following:

1. Sign up or log in to your Northflank account
2. Navigate to our Temporal stack template
3. Click Deploy Temporal Now
4. Feel free to customise your project name, colour and region
5. Click Deploy stack to save and run the Temporal template
6. Once the template has finished, select the temporal-ui service
7. Open the code.run domain under Ports & DNS to access the web UI for Temporal

You now have a working Temporal cluster! Below are some ideas on how to start using it, as well as some Northflank and Temporal features to explore:

As mentioned previously, your cluster needs application workers to execute code. Below we'll detail how to run the TypeScript “Hello World” sample provided by Temporal:

1. Use the Northflank CLI to forward the temporal service to your local machine, so you're able to access its TCP ports (use the correct project name/id for your Temporal project):

   sudo northflank forward service --projectId temporal --serviceId temporal

2. Locally, clone the sample repository:

   git clone https://github.com/temporalio/samples-typescript.git
   cd samples-typescript/hello-world

3. Update the address in both src/worker.ts and src/client.ts from localhost:7233 to temporal:7233.

4. Install the dependencies and run the worker:

   npm install
   npm run start.watch

5. In another terminal, navigate to the same directory and run Workflow Client:

   npm run workflow

6. The workflow should print Hello, Temporal! and you'll be able to see the workflow logged in your web UI.

Currently your Temporal web UI is accessible via a code.run domain. Northflank allows you to configure custom domains for your services, to make your deployments accessible on your domains and subdomains.

Northflank provides easy access to container logs and metrics via both the web platform and the CLI tool. You're able to access both real-time and historic data for all services, jobs and addons, enabling you to easy monitor services as well as debug issues.

Namespaces in Temporal are a unit of isolation within Temporal. Workflows are scoped to specific namespaces and workers and clients must specify which namespace they are connecting to. The Northflank stack template sets up the default namespace, however you able able to define extra using the tctl tool. You can access this tool via the shell of the temporal-admin-tools container.

The retention period of a namespace dictates how long event history (detailed logs) for completed workflows is retained for. The default retention period is 1 day. You can modify this using the tctl tool, via the temporal-admin-tools container shell. For example:

tctl namespace update \\
  --namespace default \\
  --retention 72h

By default, Temporal UI doesn't come with built-in authentication, but you can secure it using one (or multiple) of Northflank's features:

• IP Policies enables you to limit access to specific IP addresses and ranges
• Basic Auth requires users to enter a username and password
• SSO Provider require users to authenticate via your SSO provider (enterprise only)
• Northflank CLI using the northflank forward command to forward access to the service locally