Tech

Deep Fried Kubernetes

Alex Popell

• 4 min read
Deep Fried Kubernetes

Kubernetes, or k8s if you're cool, is an open-source container-orchestration system originally designed by Google. It was released 6 years ago, uses a nautical naming theme (means 'helmsman' or 'pilot' in Greek. so cute.), and most jobs in DevOps (systems admins that know how to code, or vice versa) require about 10 years of experience with the technology.

The future, until they pick a new future.

Containers are systems that run on top of the operating system installed on your server. They can run on any system, and can be moved around (this website is running inside a container).  They need to be orchestrated (sexier way of saying "managed") because to make things in IT "just work" they need to be set up like a complicated cascade of dominoes, and often require an offering of part of the technician's eternal soul.

My newsgroup is not miserable, but yes.

This tech is cool because if I have a k8s-based infrastructure, I can throw my servers in the lake, buy new ones, and be up and running with an identical setup within a few minutes. Technology leaders like Google, Slack, me, Shopify, IBM, and Spotify know this, which is why we're collectively investing in this system for managing our networks.

If you're not interested in the future of your fast food experience, and you may not be, you can stop reading now, since this post is not going to get more interesting. Feel free to write me a complaint letter (here's a link to generate one).

Chick-fil-A

One interesting example comes from one of my favourite networking teams on the internet: the folks at Chick-fil-A, who run a blog cataloging various IT endeavours they undertake to more-efficiently clog up all our arteries. I've never actually visited one of their restaurants, but I'd like to go, just to check out how a bleeding-edge network behaves in the field, and get some chicken.

Chick-fil-A randomly has an incredible IT department

Unlike the companies I listed at the beginning, these guys have to overcome three major issues:

  1. Fast food restaurants don't have on-site IT staff/technicians. Your first responder's highest credential might be an elementary school diploma from 2 years ago.
  2. The physical environment is a fast food restaurant, not a data center.
  3. Massive horizontal scale (they have over 6000 locations, and it has to work everywhere)

They came up with a good, practical solution. The kind you can tell was made by real people, thinking about how to solve real problems:

Cheap and easy. ← Not even gonna make a joke.

The three guys on the left (some Intel NUCs, probably $500 CAD) go into the guy on the right. At the initial setup of the store, a networking team puts in a relatively simple network switch setup, like any other fast food place's IT protocol, except this "server" is hosted on-site. And it's running on Kubernetes. On these three little NUCs. And you can throw one in the ocean, and put a new one in.

I bet their meetings are brief and only scheduled when necessary

Specifically, it's running on something called Rancher Kubernetes Engine (RKE):

  • RKE runs in clusters made up of nodes
  • It requires one of the nodes to act as a leader node.
  • When a new NUC joins the cluster, it'll be configured over the network automagically and ask the others if one of them is the leader.
  • If it gets a reply (and it probably will, as this uses UDP) it'll accept its leader.
  • If not, it'll tell the others that it wants to be the leader. They'll have an election, and it'll probably end up being the new node.
  • The nodes all talk to each other sometimes with random sleeps and leader declarations to make sure they still have a leader, and can elect a new leader if theirs is missing.
  • A lot of normally really difficult network operations are handled automatically and at scale, with no downtime, and no resources are wasted
Various tools they use (Highlander is internally designed by them) and the accompanying processes.

That last point is really what it's all about. Think of all of the processes required to run the network of a fast food restaurant: mobile orders, inventory, sales, receipts, refunds, shift management, drive-thru, a million random things. Think of what happens if any core process fails. Have you ever seen someone flip out by waiting for a burger longer than 5 minutes? Ever seen them demand a refund and the computer system not work?

Creating the infrastructure is difficult, but creating it in a way that augments the work humans do is the very spirit of automation. These folks built a super-efficient system using cutting-edge technology, and they communicate it all in a clear way that fosters growth in the community. They make the fast food workers' jobs easier (and don't worry, not obsolete either. Yet.).

I find that really cool and really inspiring.

The blog posts:
https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541
https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2