Kubernetes is a container orchestrator, not a VM runtime.

Projects like KubeVirt and Virtlet represent some full-circle irony. It gets to a) the actual value proposition of something like Kubernetes and also b) the ultimate futility of everything in this field.

Kubernetes, properly conceived, would just be a description language for compute assets and their lifecycles. The problem, of course, is that we already had that several times over, and Google couldn’t really gain any ground v. Amazon in the “cloud wars” with something agnostic and abstract like that. So they built an elaborate set of daemons and unnecessary indirections on top of already-functional system layers, called it Kubernetes, and unleashed it on everyone else, with the promise that if you want to be Googly, you want to use Kubernetes, which works pretty well on Google Cloud by the way, wink wink. It’s what Google uses to make some Googly magic happen, so how can you (or your boss) say no?!

Noobs ate this up because they apparently didn’t know about things like health checks before Kubernetes (see https://news.ycombinator.com/item?id=16972873 ), and besides, how can you argue with *Google*? If Google does it, or Google uses it, all the hard stuff must already be figured out. I have heard this sentiment expressed nearly verbatim dozens of times as I’ve discussed k8s with people.

Meanwhile, Solaris/illumos devs and FreeBSD devs have been touting the virtues of OS-provided sandboxes that *are* well-integrated with existing system management layers for 15+ years. But because they don’t have the sheen of a Fortune 100 (Google/k8s) or a massive pile of VC money to set alight (Docker), they’ve been, essentially, ignored, until Google saw an opportunity for a power grab against Amazon.

Technically speaking, Kubernetes is a container orchestrator with zero respect for state. Running actual VMs in it is an even worse idea than running a database in it, which is already a pretty bad idea. People want to do it because they get comfortable with kubectl and want to use it for everything, showing that we did need a unified control plane, something Kubernetes seem to address inadvertently.

Please know that you don’t have to, and honestly probably don’t want to, use Kubernetes or Docker for general OS-based virtualization intended to supplant e.g. an EC2 instance. There’s LXC, OpenVZ, and others on Linux, jails on BSD, zones on Solaris/illumos. Those *are* designed for this type of use case, they respect state, you can mostly use normal utilities to manage them, and you will surely have a better experience with them than you would trying to stick long-running processes on k8s.

(Note also that cgroups, the kernel mechanisms that enable containerization on Linux, are emphatically *not* security-oriented, whereas zones and jails are.)

Modern containerization on Linux is so ill-conceived, and yet there’s a cult around it big enough to make me worried that my 10 year old account on HN will get nuked for being the lone voice willing to discuss it. This shows the futility of everything.

%d bloggers like this: