From Panic to Peace: Making Kubernetes Controller Observability Suck Less

This is a summary of a talk about something most teams only notice when it goes wrong: debugging Kubernetes controllers. Cat Morris and Derik Evangelista frame the session as a story in four chapters, but the core problem is straightforward. Their product, Kratix, is a Kubernetes controller, and they realised that when users hit problems, the debugging experience was far more painful than it should have been.

The talk walks through what they learned from watching real users struggle and the changes they made to reduce controller observability frustration. You can find the recording below, or read on for a summary!

https://www.youtube.com/watch?v=1P76Roya9oQ

The wake-up call: debugging was harder than it should be

The turning point came during a customer community practice session in late 2024. One of their users, Sid, was demoing how he debugged Kratix in practice.

What should have been a simple troubleshooting flow became a long sequence of aliases, log watches, pod lookups, greps, label tweaks, Kubernetes events, repeated tab-switching, and finally a troubleshooting guide written by the product team themselves. It took around a dozen steps to find and fix a single issue.

For Cat, that was the moment the problem became obvious. The issue was not just that debugging was hard. The problem was that the product required users to know too much about its internals to understand what had gone wrong.

The team narrowed the pain down to two big problems:

• Interacting with the platform was hard

• The logs were hard to understand

Everything else flowed from there.

Better observability starts with making information easier to find

Derik explains that the first question they asked was not “how do we add more logs?” but “how do we make the right information easier to reach?”

kubectl should be enough in theory, but in practice users were relying on long, ‘memorable-only-if-you’ve-run-them-50-times’ commands to find what they needed. That suggested the problem was not missing data. It was the interface to the data.

The obvious options were familiar ones:

• Improve the command-line experience

• Add a CLI

• Add a GUI

They recognised that a CLI can simplify frequent operations, much like tools such as Tekton. But a CLI is still another thing to install and learn. So they also leaned into a graphical interface using Headlamp.

That gave users something much more immediate. Instead of hunting through logs and pod names, they could see red and green states at a glance, click around, and start discovering what was wrong without first reading a wall of documentation. It also acknowledged something many engineering teams eventually learn: users do not always read the docs, so the product has to teach through the interface.

Logging less was part of the answer

The second big issue was log quality. Kubernetes controllers reconcile often, and when you log naively inside that loop, you can generate huge amounts of noise very quickly. In Kratix’s case, that meant thousands of lines in minutes, even when nothing was actually broken.

The team stepped back and asked two useful questions:

• Who are the logs for?

• What are they for?

Sometimes logs are for the operator. Sometimes they are for the developer working on the controller. Sometimes they are there to show the system is alive. Sometimes they are there to help debug a real issue.

The problem was that one stream of logs was trying to do all of those jobs at once.

Their answer was more structure and more intentionality:

• introducing clearer log levels

• reducing unnecessary reconciliations

• adding structured logging

• deciding which errors were really worth logging in an eventually consistent system

One example Derik called out was using predicates to stop controllers from reconciling on changes that did not actually matter. Another was recognising that not every apparent error needs to be treated as an error, especially in Kubernetes, where retries and conflicts are often normal.

Taken together, those changes reduced their log volume significantly and made the remaining logs more useful.

Good Kubernetes UX means surfacing the right status in the right place

The third part of the talk came from another internal trigger. After watching a video review of a similar Kubernetes product, the team started asking themselves a useful question: if someone reviewed Kratix the same way, what would they criticise?

That led them to a hard truth. Their resources did not expose enough useful state through native Kubernetes interfaces like kubectl get and kubectl describe.

This mattered because users should not have to understand the entire object tree of a controller just to work out why one child resource had failed. The product should help them follow that chain.

So the team defined a set of principles for the data they wanted to expose:

• Users should be able to move from a parent resource to a failing child resource

• The experience should feel natural to kubectl users

• Each resource should show a clear, consistent state

• Changes in state should emit events

In practice, that meant doing a lot of relatively unglamorous but important work:

• Adding common labels so related resources could be found together

• Improving printer columns so kubectl get surfaced useful states

• Introducing more consistent status fields and conditions

• Emitting events whenever significant state changed

The result was simple but important: users no longer had to dig blindly. More of the right information was available in the places Kubernetes users already expect to look.

The “observability onion”

Cat closes with the model that ties the whole talk together: the observability onion.

Cat and Derik's platform “observability onion”

Her point is that observability has layers, and solving only one layer rarely solves the real problem.

At the centre are the logs. They contain the deepest detail, but they are only useful if you already know where to look.

Around that is status. Status tells you if something is healthy or not.

Then come events, which tell you that something changed and might need attention.

Then metrics, which let you understand whether you are meeting your operational objectives.

Then traces, which help tell the story of what happened across the system over time.

And finally, there are interfaces and dashboards, which turn all of that into something different personas can actually use, including stakeholders who are not going anywhere near kubectl.

That was the bigger lesson from the talk. Better controller observability is not just about adding more telemetry. It is about exposing the right kind of information, at the right layer, for the right person.

Final thoughts

What makes this talk useful is that it starts from a very familiar engineering mistake: assuming the product is observable because the data technically exists somewhere. Cat and Derik show that this is not enough. If users need aliases, grep, multiple tabs, and a troubleshooting guide just to understand a common failure mode, the observability model is not doing its job.

Their improvements were not especially exotic. They were mostly about discipline:

• Clearer interfaces

• Fewer, better logs

• Better use of native Kubernetes status and events

• More consistent resource design

• And a better understanding of who actually needs which information

The result is not “perfect observability.” It is something more useful than that: a system that makes the path from panic to diagnosis much shorter.

If you’re building controllers, that is probably the standard worth aiming for.

You can watch the talk here: https://www.youtube.com/watch?v=1P76Roya9oQ