WARNING: Kubernetes Undefined as Platform

Kubernetes is a platform for building platforms. Kubernetes is for operators, not developers. Grabbing a big Cloud-hosted flavor of Kubernetes is sure to delight your ops team, but it's just as likely to leave your dev team grumbling. The reason? Kubernetes is not the platform developers need; It's a complex set of primitives misaligned with their primary focus—building applications.

Platforms are defined by your ability to build on them. If you’re a platform engineer, Kubernetes really is a platform. You can build what you need on top of it. If you’re an app developer, Kubernetes is overwhelming. And if you’re a platform engineer, it’s overwhelming for you, too! That might be why so many IDP projects built on Kubernetes go sideways and get re-platformed. Despite all of the good that Kubernetes does, we still lack a post-commit platform that developers love.

Getting to a place where both operations and development are happy is a problem that other platforms struggled with in the past. KubeCon even hosted keynotes that compared Kubernetes to one platform that really broke through. As we head into KubeCon Salt Lake City 2024, It’s worth revisiting that, and some of the other platforms that led to Kubernetes.

In 2019 Bryan Liles keynoted KubeCon with his talk, In Search of the Kubernetes "Rails" Moment. He made a bold point by saying that YAML kind of sucks. In the world of Kubernetes, YAML manifests mean screens full of undefined fields and a dizzying array of tasks. That's a far cry from an experience like rails new blog. In other words, YAML is the wrong abstraction for app developers.

Ruby on Rails was a platform built in an era where LAMP (Linux, Apache, MySQL, and PHP) was a dominant stack. Like Kubernetes, the problem with LAMP was figuring out how to make it usable for software engineers.

Today, Kubernetes feels akin to the L in LAMP. Both Linux and Kubernetes are platforms that other components build on. Linux is definitively an OS, and Kubernetes is the OS for the Cloud. It’s wild to think of an app developer wrangling kernel-level Linux APIs. But, with Kubernetes, wrangling is the status quo.

As platform engineers, we need a platform that not only abstracts away the complexities but also frees developers to focus on writing the code they get paid for.

Pivotal’s Cloud Foundry was an early attempt at providing a sophisticated platform as a service. They nailed the vision of simplifying application deployment and enabling the “you build it, you run it” ethos. PCF had an easy onboarding like Rails; instead of rails blog new there was cf push. The experience felt similar, but the big leap made by Cloud Foundry was supporting nearly every language and framework (not just Ruby). Developers just needed to commit their code. PCF is what drove everything post-commit.

Yet, the platform still required large teams to maintain and operationalize alongside a hefty hardware investment that took months to provision. Because the effort required to adopt PCF, it didn’t quite live up to its full potential, nor did it adapt fast enough to a cloud-native era. Remember how the missing piece of Kubernetes was good developer experience? The missing piece of Cloud Foundry was an adaptable and pleasant operations experience.

The cloud-native ecosystem is much more robust, as is the size of the problem, considering how many more software engineers are shipping workloads – with considerable effort and sometimes unsuccessfully – compared to a decade ago.

I should mention that Cloud Foundry rose to prominence in the early 2010s, around the same time as Apache Mesos. Mesos was on the other end of the spectrum from PCF. It focused heavily on operational experience, but never quite found its footing. Heroku was from a similar era, but focused on developer experience while hiding the operational aspects.

When Kubernetes rose to prominence, its success was partly fueled by its flexibility. There are many reasons why Kubernetes succeeded all of these platforms. K8s gave the Cloud a standard API, it was declarative, and its focus on containers abstracted nicely over VMs. Another reason Kubernetes succeeded is because ingredients can be swapped in and out. As an example, K3s swaps out etcd for a more traditional relational database.

The emergence of EKS, GKS, and AKS cemented Kubernetes as the definitive OS for the Cloud – each with their own flavors and challenges. It’s worth remembering that the application abstraction is still a task left to platform builders. It’s easy to see why. How do you want to get your code from Dev to Prod? Every team and organization is going to do that a little bit differently. That’s an important detail to keep in mind when recalling the “Kubernetes is a platform for building platforms” mantra. Finding the right DX is quite a challenging task.

So, what should a platform actually look like? I can share some of the philosophies that have guided me as I’ve built the Northflank platform. Most platform engineers share an overarching vision, already. That vision is that everything post-commit is abstracted by the platform. That kind of abstraction frees developers to ship their workloads in a self-service way. They should be able to build, deploy, and scale their workloads without being infrastructure experts. As long as the controls and levers are still available for tuning APIs below the surface of the platform, we’ve got a winning solution.

That’s the broad vision that I think most of us share. That broad vision translates into design philosophies, and ultimately requirements. I’d love to hear about the philosophies and requirements that guide other platform builders. In that spirit, here are a few of mine.

• IaC is a starting point. Infra-as-code is essential, but it's too static, yet the release process is inherently dynamic. It leaves open questions like “How do I get code from Dev to Staging to Prod?” and “How do I restore production in another region or cloud?” The platform should provide a golden path that answers that question.
• Automate CI/CD pipelines. CI/CD is where the post-commit journey begins. Minimize manual intervention, and live the GitOps dream.
• You build it, you run it. Developers must be able to deploy and scale their applications with a few clicks or commands.
• Polyglot is standard. Most businesses making software are too big to not build with multiple languages and frameworks. The platform must be polyglot — not just for ephemeral, but also stateful and scheduled.
• All workloads. A platform should be agnostic about how complex the workload type and be able to support all containerized frameworks.
• Make Troubleshooting Easy. One of the largest headaches when running software is troubleshooting. All of the APIs hidden from app developers need to still be accessible to SREs.
• Bi-directional, real-time intefaces. If I update the workloads in Git, the UI should reflect those changes and vice versa. And don’t make your teams guess where info about their workloads lives. We shouldn’t accept stale information in cloud UIs.

In essence, the future platform should empower teams to "make workloads, not infrastructure."

By embracing a platform that prioritizes developer experience without compromising operational flexibility, organizations can accelerate their delivery cycles, reduce overhead, and stay competitive. A good platform frees developers to do what they do best—write code—while operations ensures that the supporting infrastructure continues to run smoothly.

DevOps is about uniting developers and operations. Platforms aren’t really platforms if they cater to one over the other. That’s something I’ll keep in mind as we head into KubeCon 2024 in Salt Lake City. There are over a dozen talks about platforms at the main event, and a whole co-located platform engineering day, as well.

What I’ve shared here flows from my experience building platforms on Kubernetes at Northflank. If you spot me wandering around KubeCon, I would love to hear what you think. Is it possible to make a successful platform that de-prioritizes either half of DevOps? What philosophies guide you as you build your IDP? What do you see as the major challenges while platform engineering?