Do not get me mistaken; we’re sturdy supporters of Kubernetes. It’s a vital piece of our structure and offers large worth when wielded appropriately. However, Kubernetes was initially supposed to behave as a container orchestration platform for stateless workloads, not stateful functions.
Over the previous few years, the Kubernetes group has achieved a fantastic job evolving the challenge to help stateful workloads by creating StatefulSets, which is Kubernetes’ reply to storage-centric workloads.
StatefulSets run the gamut from databases, queues, and object retailer to janky outdated net functions that want to switch a neighborhood filesystem for no matter motive. They supply builders with a set of fairly highly effective ensures:
- Constant community id for every pod: This lets you simply configure the DNS handle to the pod in your software. It really works nice for database connection strings or configuring difficult Kafka purchasers. We additionally use it for organising erlang’s mesh community at occasions too.
- Persistent quantity automation: Each time a pod is restarted, even whether it is rescheduled onto a special node, the persistent quantity is reattached to the node it’s positioned on. That is considerably restricted by the capabilities of the CSI (Container Storage Interface) you’re utilizing. As an example on AWS this solely works inside the identical regional AZ since EBS volumes are AZ-linked.
- Sequential Rolling Updates: StatefulSet updates are designed to be rolling and constant. It can all the time replace in the identical order which might help protect methods which have delicate coordination protocols.
These ensures cowl a ton of the operations wanted to run a stateful workload. Particularly, it nearly utterly handles the supply portion. Provided that EBS uptime and redundancy ensures are extraordinarily sturdy, the StatefulSet’s rescheduling automation nearly trivially ensures you a excessive availability service. Nonetheless, some caveats do apply (e.g., that you’ve room in your cluster and don’t botch the AZ setup.)
Kubernetes has a ton of promise on this space, and in principle, might actually evolve right into a platform to simply run stateful workloads alongside the stateless ones most builders use it for.
What’s Lacking From the Kubernetes StatefulSet?
So why do we expect StatefulSets are damaged? Nicely, in the event you run via the operational wants of a stateful workload in your head, there’s one key part that you simply may discover is lacking:
What do you do when you could resize the underlying disk?
The dataset is a typical database retailer that usually grows at a reasonably fixed optimistic charge. Except you help horizontal scaling and partitioning, you’ll want so as to add headroom within the disk as that dataset grows. That is the place Kubernetes falls flat on its face.
Presently, the StatefulSet controller has no built-in help for quantity resizing. That is even if nearly all CSI implementations have native help for quantity resizing the controller might hook into. There’s a workaround, nevertheless it’s nearly ludicrously roundabout:
- Delete the StatefulSet whereas orphaning pods to keep away from downtime with: kubectl delete sts <identify> –cascade=orphan
- Manually edit the persistent quantity for every pod to the brand new storage dimension
- Manually edit the StatefulSet quantity declare with the brand new storage dimension and add a dummy pod annotation to power a rolling replace
- Recreate the StatefulSet with that new spec which permits the controller to reclaim management of the orphaned pods and start the rolling replace which is able to set off the CSI to use the quantity resize
💡
Okay, so there’s a reasonably noteworthy flaw in Kubernetes StatefulSets, however there is a workaround even when it’s considerably janky.
That shouldn’t be too unhealthy, proper?
But it surely will get worse!
The scenario will get downright painful if you notice the influence of this limitation and that numerous the Kubernetes operators have been constructed to handle stateful workloads.
A reasonably good instance is the Prometheus operator, which is a superb challenge for each provisioning Prometheus databases and permitting a CRD-based workflow for configuring metrics, scrapers, and alerts.
The issue arises as a result of the built-in controller for the operator has no logic to handle StatefulSet resize, nevertheless it does have the logic to recreate its underlying StatefulSet if it sees an occasion that triggered its deletion. Because of this you successfully haven’t any method to make use of the above workaround, for the reason that second you do a cascade orphan delete, the operator will recreate the StatefulSet towards the outdated spec and stop correct resize. The one resolution is to delete all the CRD or discover a tweak that may idiot the operator into not reconciling the article (typically scale to zero will do that).
Regardless, because of this flaw, there’s successfully no option to resize a Prometheus occasion with the operator with out both vital downtime or information loss. Contemplating how sturdy the automation in StatefulSets is in all different circumstances, it’s fairly stunning that that is nonetheless a possible failure mode.
Our Head of Neighborhood, Abhi, truly hit this subject with interaction between operators and StatefulSet quantity resizes as properly whereas implementing it within the open-source Vitess operator.
“Contemplating the pure complexity of a Vitess deployment, you may infer that disk resizing is proportionally difficult. Vitess is a database sharing system that sits on prime of MySQL, that means that quantity resizing needed to be each partitioning-aware and shard-aware. We needed to manually write our personal shard-safe rolling restarts, create a cascade situation that labored with the parent-child construction of Vitess customized sources, and handle each conceivable failure situation to forestall downtime. Shoutout to notable Kubernetes contributor enisoc for designing this function.”
Different broadly used and notable database operators, like Zalando’s Postgres operator, successfully reimplement the identical process we carried out within the Plural operator in their very own codebase. This causes a ton of wasted developer cycles on an issue that ought to solely must be fastened as soon as.
The Potential of Kubernetes
Generally, we’re extraordinarily bullish on the potential for Kubernetes to make the operations of nearly any workload nearly trivial, and an enormous a part of our mission at Plural is to make {that a} risk.
That mentioned, we additionally should be clear-eyed about gaps that also stay within the Kubernetes ecosystem, so we are able to both work round them or shut them upstream. I believe it’s fairly clear it is a vital hole, and if prioritized, this may very well be fastened fairly simply in a future launch of Kubernetes.
In the event you thought this was attention-grabbing, take a look at what we’re constructing on Kubernetes right here.
