The what and why of ephemeral preview environments on Kubernetes

This post is an overview of preview environments. It covers why they’re crucial for modern development teams, what makes them challenging to implement (particularly for backend and full-stack apps), and how Northflank simplifies the entire process. This guide focuses on preview environments for apps running in Kubernetes clusters and tools within the cloud native ecosystem. That said, most of this should still be broadly applicable to any other type of deployment target.

Note that preview environments are also sometimes colloquially known as ephemeral or sandbox environments. All of these terms are interchangeable.

What is a preview environment?

Preview environments are temporary, full-stack environments spun up to test and validate new features before they merge into the main codebase.

They give developers, QA teams, product managers, and other stakeholders a fully functional environment. This is often an entire microservices-based stack plus supporting databases. Stakeholders can then inspect how an upcoming feature behaves in real-world scenarios.

These environments are automatically created, usually triggered by a Git pull request or merge request. Once the code is validated or a feature is merged, the environment is typically torn down, freeing up resources and efficiently managing costs.

How are preview envs different from other envs?

Preview envs are like staging since both environments run on non-local environments. They differ from staging and other pre-prod environments, because the code you would run on a preview environment is usually unmerged and on its own branch.

Preview environments (PEs) might also seem similar to how you share code running on your local machine. Preview envs differ from your local development environment since sharing your local just points to what you’re working on and only works when your computer is on. In contrast, multiple preview environments can co-exist for multiple branches, and they’re accessible even when devs are offline.

One of the major challenges with setting up any pre-production environment is matching production. Preview environments may require more effort than prod. This is because deployments to prod are usually sent to existing, already made environments. In contrast, deploying to a PE means you have to create the cluster and associated state from scratch for each new environment.

A diagram comparing ephemeral preview environments and staging environments. Both environments are important for testing, but slight differences in their implementation shifts where they find their best use.

Benefits of preview environments

Challenges aside, the benefits of ephemeral preview envs are massive. Preview environments are the perfect way to share work-in-progress with business stakeholders.

Being private and isolated to a specific change is critical. The trend in DevOps is for work to shift left. In other words, business stakeholders are involved earlier in the process. Preview environments have become a standard practice, because they are an ideal way to get stakeholders involved earlier in the development process.

• Faster feedback loops
  When developers can spin up a complete environment on every feature branch or pull request, it’s easier to identify bugs, performance issues, and integration gaps early in the development process. When a PR is opened as sometimes you push to a branch and it’s in no state to deploy - when a PR is made it’s more in a ready state.
• Consistent testing
  “It works on my machine” is no longer a problem. With preview environments replicating production settings (frameworks, databases, services, and more), teams can reliably test new features without environment discrepancies.
• Improved collaboration
  Stakeholders from QA, product, marketing, and sales can visit a live URL to see a feature in action. QA in particular can be idle while waiting for features to be ready for testing. Getting access to PEs earlier empowers other roles to offer feedback early, leading to more refined products.
• Reduced bottlenecks
  Large teams no longer have to queue for a shared dev or staging environment. Parallel preview environments let multiple feature branches proceed without waiting for a “free” test environment.

Ultimately, the biggest advantage is fast, continuous feedback. Agile methodologies and Extreme Programming emphasize rapid feedback loops for good reason—it improves both code quality and developer productivity.

The faster feedback loops that preview environments fit into is normally a deadzone in most developer’s feedback cycles. When code is local, it can be shared with Ngrok or Tilt in conjunction with tools like Docker Compose or Minikube. Once code is committed, there’s no real way to see what it’s like before you merge and promote to staging.

Of course, you could ask developers to keep a local version running. But, what happens when a stakeholder is asked to review a change and other work in local development introduces a breaking change? What if the developer has already clocked out for the day? Only a solution like preview environments keeps feedback flowing while code lives in limbo after commit but before integration testing on staging.

Key features of a preview environment

• Made per branch or PR
  Every branch or pull request (PR) can have its own environment. This allows developers and stakeholders to preview changes without impacting other deployments.

• Isolated
  Changes in a preview environment are self-contained. In addition to keeping shared envs like staging and QA clean, isolation also means PEs are often on a private branch that’s isolated from other developer’s work-in-progress.

• Production-like testing
  Preview environments are typically configured to mimic the production environment as closely as possible. This means there are similar dependencies, configuration, and underlying infrastructure. That matching is crucial for accurate testing.

• Ephemeral
  These environments are temporary and automatically created and destroyed as needed (for example, upon PR creation and merge). The ephemeral approach means unused environments will scale down so that you can avoid unnecessary costs.

• Automated
  Like almost everything in the cloud, preview environments have to be automated. If you’re manually managing PEs, you're likely losing track of them or wasting time waiting on manual provisioning.

  

  An example of creating a new preview environment template, using Northflank's visual editor. Everything seen on this screen may also be done by using the CLI or API calls.

Typical workflow and challenges

When code is local, developers can share work-in-progress with dedicated tools. However, once changes are pushed to a repository, there’s a “dead zone” before code merges into staging. PEs fill this gap. This list is a simplified workflow.

1. Developer commits code and opens a PR.
2. CI/CD detects the commit, checks for keywords in the commit message, and automatically spins up a preview environment.
3. The new code (along with test data) is deployed.
4. The feature is reviewed and tested by stakeholders.
5. When the branch is merged or closed, the preview environment is scaled down or destroyed.

This workflow keeps feedback flowing during a critical phase—after code is committed but before it merges with the main branch.

Preview environments usually exist in large numbers because there might be one per PR or even one per feature branch. Unsurprisingly, that makes scalability and automation absolutely crucial. Implementing preview environments manually results in human error, delays, and higher cloud bills.

A comparison of ephemeral preview environments and staging. This diagram shows how ephemeral previews shift the usual development cycle left. QA, PMs, and security are all able to get involved much earlier in the process.

There are a number of other challenges worth highlighting, as well.

Complexity of full-stack workloads

Preview environments are less complex for simple static websites (like using Next.js on Vercel or Jamstack on Netlify). However, full-stack or backend workloads involve containerized microservices, databases, job schedulers, and more -- meaning you need to implement Kubernetes, increasing the DevOps complexity and overhead. At Northflank, we’ve seen some organisations run up to 30 microservices in a single PE. Setting up ephemeral Kubernetes environments for each pull request balloons your environment complexity without a proper platform.

Infrastructure and cost

Spinning up dozens (or hundreds) of environments in a large team could skyrocket infrastructure bills. If not managed well, ephemeral environments can eat up compute, storage, and other resources. Intelligent cost controls are essential and may include strategies like leveraging spot/preemptible instances and time-based auto-shutdown.

Data handling and migrations

Preview environments, like most test envs, need realistic datasets. Manually cloning databases, managing migrations, and ensuring data security can be cumbersome. A single mistake can break a feature branch or corrupt shared data, delaying progress.

Tooling fragmentation

Many teams have multiple repositories, microservices written in different languages, and a variety of pipelines like Argo and Tekton. Stitching these together into a cohesive, automated workflow often turns into “YAML whack-a-mole,” where developers spend more time on DevOps plumbing than coding.

Tracking cloud costs

Tracking infrastructure sprawl is trivial with staging. The staging cluster is tagged, and there is just a single version of each service to run. It’s no big deal to track that manually. Tracking preview environments, on the other hand, is a lot more involved. Each version for every commit gets its own deployment and potentially its own cluster.

A manual process is chaos. Human error makes tagging and tracking resources inconsistent. Since there’s no automated way to scale down, dangling resources are a threat.

Access control

At some level, security is just who has access to what. With ephemeral preview environments, you have customer data and topology info about your production environment. As a result, ephemeral preview environments need security that is at least on-par with staging. Ideally, you’re providing access through SSO.

Secrets and security

Keep it secret, keep it safe. The secrets you use in your environment need to be securely stored and you need a system for organising and injecting different secret groups based on the environment. Connection details for staging are different from production. Likewise, connection details for a database you spin up in a preview environment are unique to that environment. You need a way to create and securely share that connection info with the right environment.

Scaling down

It’s rare to scale down a staging environment. It’s exceptionally common to scale down preview environments. For effective ephemeral environments, you must have a solution for identifying which images are in active use, and which can be snapshotted and spun down.

Commit message trigger words

You might not always want to create an ephemeral preview for a commit. That’s why it’s essential to hook your PE tools in with Git. That way, commit messages can be parsed for keywords and trigger CI/CD actions.

Promoting workloads

It’s great having access to all of these different environments, but how do you promote workloads between envs? There are a glut of options for CI/CD and many have their own approach. Preview environments are dynamically created, and thus quite different from something like staging. Ideally, this process should be highly automated so that developers are able to self-serve as idealized by the DevOps philosophy.

An example of a release flow on Northflank. This is Northflank's visual editor for making re-usable release flow templates.

DIY versus buy

There are a number of tools for creating ephemeral preview environments on the cloud native landscape². A DIY approach might start with an automated provisioning tool like Karpenter or Istio. Argo CD and other projects can be tied together to create a pipeline.

Building it yourself with open source is free as in beer, but it’s also free as in puppies. In other words, you have to continue to take care of it. You might find you also need to fill in the gaps if you’re unable to find solutions for tracking costs, detecting idle environments, and scaling up and down.

This shortened checklist of requirements summarizes and hopefully offers a bit of guidance as you consider what features should be on your preview environments (even if they’re DIY).

• Provisioning and release flows: Select containerization, orchestration (Kubernetes/PaaS), and CI/CD options.
• Configuration: Define Infrastructure-as-Code, environment variables, and secrets management.
• Secure access: Enforce SSO, securely generate preview URLs, and determine how you want to surface those links to stakeholders.
• Data cloning and migrations: Programmatically create and seed your environment with test data.
• Build caching for warm starts: Minimize waiting time and maximize cloud savings by snapshotting, and restoring PEs that have already spun down due to inactivity.
• Tear down: Systematize how you want to monitor, track, and destroy unused resources.

I'm biased (this is the Northflank blog after all), but Northflank's features are a good example of what to look for when considering how to approach ephemeral preview environments.

Northflank resolves the complexity mentioned above with an out-of-the-box platform for preview environments. No matter the workload complexity, if you can dream (build) it, we can support it. Northflank offers granular billing to control costs, automates spinning up and down, and simplifies promoting workloads. The result? Well, you should already know the benefits if you're still reading.

• User-friendly interfaces and IaC means devs are equipped to manage their own pipelines and release flows for preview environments. No need for scattered YAML files and partial solutions. Drag and drop components, sync changes to git, and create workflows for promoting workloads between environments. Configure and reference your version control metadata in one place.
• Automatically create PEs on pull requests or merge requests. Native GitHub, GitLab, and Bitbucket integrations trigger ephemeral environments automatically upon pull requests or feature branches.
• Security and secrets management encrypts data at rest and inflight, keeping sensitive PII³ safe. Northflank’s secret groups inject environment variables at runtime without exposing them in your repositories.
• Track costs and locate dangling resources with the per account, per project, and per resource billing data. Scale down environments by setting how long you want to wait before tearing down, and set working hours so PEs automatically shut down on nights and weekends. Deploy on spot or preemptible servers to cut infrastructure costs by up to 90%. When PRs are closed or merged the related infra is deleted. Previews can be paused manually or automatically by time based logic. For example, set your preview envs to scale down during the night when developers are out of the office.
• Spin up databases, so that they are ready for preview using cloned data from staging in seconds. Reset environments if a migration goes wrong and eliminate complex rollback procedures. Set up a job to regularly obfuscate¹ your prod data, and clone that output as seed data for any PE.
• Build caching and container start times are improved by Northflank’s caching mechanisms, ensuring previews are ready in minutes rather than hours. The average PE creation time is under a minute.
• Observability and logging on every PE. Gain full visibility with real-time logs, metrics, and monitoring dashboards. Quickly troubleshoot issues and provide feedback to developers.

If you’re interested in reading a full guide on how to set up preview environments with Northflank, check the preview environment page on our docs.

An example of using Northflank's pipelines and templates to re-use code for preview environments alongside dev, staging, and prod setups.

In conclusion

The typical challenges revolve around cost control, configuration complexity, and external integrations. With the right approach, the benefit of preview environments is they vastly improve the speed and quality of your software development lifecycle. Many of those benefits can be broken down by role.

• Quality assurance (QA): Test new features earlier in the process and more thoroughly, ensuring changes don’t break existing workflows before reaching staging or production.
• Product management (PM): Get a live preview of how new features integrate with the rest of the product in real-time, enabling faster iteration and more informed roadmapping.
• Engineering: Parallelize development with multiple ephemeral environments. Reduce dependency on busy dev/staging environments and accelerate your developer feedback loop.
• Marketing, sales, and leadership: Demonstrate new features to clients or internal stakeholders before they go live. Start refining marketing materials and sales pitches earlier in the process.

If you’re working on projects without ephemeral previews, it’s time to give them a shot. They can be set up in minutes to hours with the right provider (like, Northflank) and they’re sure to have a big impact.

Foot notes are listed below this point.