top of page

Day 2000 Doesn't Care How Fast You Shipped

36 minutes ago
5 min read

I spent a good few hours of this week scanning our own images for a certificate authority that had just been declared untrustworthy. eMudhra's intermediate keys turned out to be recoverable in milliseconds, so anything still trusting them can be talked into trusting a forged certificate. 


In the same week, PHP had its own moment with PHP2Shell. PHP runs something like 70% of the web, and the early counts reckon the better part of a million hosted versions will never get a fix! So plenty of people spent the week rebuilding and redeploying a lot of things (or not), and plenty more are exposed right now and don't know it, which is just a touch terrifying.


Two very different problems. A textbook CVE on one side, a broken chain of trust on the other. But they ask the same question, and it's the one we care about more than almost any other in Syntasso: when a fix exists, how does it reach what you've already built, and how long does that take?


Day 2000 Doesn't Care How Fast You Shipped
Fast provisioning isn't enough. Discover why platform engineering must address fleet-wide upgrades, security and infrastructure lifecycle management.

Shipping is the easy part

We talk a lot about the “maintain” story, because it's a big part of the Platform Engineering game.


We've become pretty good at creating infrastructure. A portal and a pipeline will stamp out a new environment before you’ve finished your morning coffee, but it's day 2, and day 2000, where the money and the risk actually live: patching it, upgrading it, proving it still does what security needs, killing it off cleanly at the end. Every template you run is one more thing someone now owns forever, usually with no safe way to change it once, and for a lot of our customers, there are thousands upon thousands of those things! 


A trust event is the cleanest test of all this, precisely because it's so rare. A CA breaks maybe once every few years. Sounds like good news, until you realise it means nobody's built the muscle to deal with it. You don't get fit for a fire drill you run once a decade.


So, two worlds.


A fleet, or a thousand individual problems?

In the first world, your infrastructure is a fleet. Every capability is a versioned Promise with its own lifecycle, and "fix eMudhra everywhere" is a base-image bump and a staged rollout, a canary group, maintenance window, the lot. Ten copies or ten thousand, it’s the same execution, because create, change and upgrade are all handled by your SKE platform.


In the second world, you've got a garden full of pets wearing cattle costumes. Thousands of apps, each spun up by a workflow that finished months ago and left the desired state nowhere near where it should be. Now you're doing archaeology with a clock running and the CISO suite breathing down your neck: which of these trust this CA? Who owns this one? Will editing the template help the copies already out there? (No. It won't.) The gap between "a fix exists" and "everything has it" is your exposure, and this week it went on public display.


None of this means throwing away your existing automation. Terraform, GitOps, CI/CD and configuration management tools all have a part to play. The challenge is coordinating them across every instance of a platform capability, long after it was first provisioned. That's what platform orchestration adds: a way to manage the lifecycle of what you've already created, not just automate the creation of new things.


The cost of fixing everything

What does that difference look like when something breaks? Let's put some rough numbers on it. 


Say you're like one of our bigger customers with 3,000-odd services carrying this trust store. Even at a generous twenty minutes each to find it, fix it, test it and ship it, and on a bad day it's never twenty minutes, that's a thousand engineer-hours. Call it twenty-five engineer-weeks, burned on a single cracked CA, under a clock, while the rest of the roadmap waits and the entire dark web tries to hack your estate. In a fleet-managed platform, you're coordinating one change across thousands of instances rather than managing thousands of independent fixes. That gap is the whole value of the maintain story, and it only shows up on the day you need it.


In the fleet-managed world, the platform team updates the affected capability once, then uses SKE to coordinate the change across existing instances. The underlying tools still rebuild, test, and deploy the workloads, but the platform provides a consistent way to manage the rollout, track progress, and handle failures. Ten instances or ten thousand, you're not starting from scratch every time.


What our own experience taught us

I'll be honest about our own week, because this affects us too. eMudhra was mostly a shrug for us because rebuild-on-a-clean-base-and-redeploy is just how we default. That's the point of working this way. But I'll be just as honest about what it didn't save us from: part of this mess rides a cross-signed intermediate that chains up to a root that half the internet still trusts. 


You can't rebuild that away. It needs revocation and expiry to run its course. A platform doesn't make the hard problems vanish. It makes the boring ones one click, so you've got attention left for the ones that genuinely need you.


A clean vulnerability scan isn't a clean bill of health


One more thing, because people keep getting caught by it (yes, us too!). Your scanners are built for CVEs, and for most of the stack, eMudhra never became one. It's a valid certificate sitting in a trust store; Snyk and Trivy will cheerfully hand that image a clean bill of health. "No known vulnerabilities" and "trusts a CA that's been cracked" are not the same thing.


The honourable exception, and hats off to them, is the Go community. Go didn't sit around waiting for the world's trust stores to catch up. They shipped it as an actual security release — CVE-2026-111943 — and hard-coded a block on the eight compromised intermediates inside crypto/x509, so a forged chain fails even if your app brings its own roots. (Node did much the same.)


That's the difference between removing a root and refusing the attack, and it's the one move that also shuts the cross-signed path I was moaning about a minute ago. It turns out it's not flagged as a CVE unless you're Go. Everyone else, you're on your own.


Your platform has to maintain what it creates

And to the people who still swear by their portals and pipelines: I get it, they're genuinely useful. They've made it much easier to create infrastructure and get software into production. But creating something is only the beginning of the responsibility.


The real test of a platform comes when something changes. A compromised certificate, a critical vulnerability, a new compliance requirement or an urgent upgrade. Can you find everything affected? Can you change it consistently? Can you prove the change reached everything it needed to?


That's what separates a collection of provisioning workflows from a platform you can operate and maintain.


Creating is easy. The next two thousand days are the real work.




Your platform can create 3,000 services. Can it safely update all 3,000?

Syntasso Kratix Enterprise (SKE) helps you move beyond provisioning to manage, upgrade and govern your entire platform fleet. See what SKE can do for your platform

 
 
 

Comments


bottom of page