Kubernetes at LIFX

Presented to REA DevOps guild

Caveats

Will probably not scale to larger teams
Practices developed for a small team
High adjacency to domain knowledge
Mostly nice services that follow most of the 12 factor pattern
Ignoring stateful applications

Speaker notes are incomplete they are coming soon

Where we were

Mesos/Marathon cluster
Self managed systems
Internally developed tools to expose services
Old versions of all the things
Super small team

Why Kubernetes?

Good primitives to build on
Good momentum
More integrated solutions
Does out of the box what we needed custom code to do

Why GKE?

limited ops time
better to spend time on more valuable tasks
smaller systems to manage
no Kubernetes controllers to manage

Migration time

First thing to do is make “all” the tools

for a very small value of all

Really just a small Go app with text/template, crypto/aes piping templates to kubectl and bash to glue it all together.

“Normal” migration process

Create a build pipeline
Create walls of yaml
Test deploy
Change dns
Profit?
Realise you missed several blobs of yaml or broken lib
Revert dns change
Update yaml, Fix libraries
Deploy
Change dns

Holiday

Abnormal migrations

The LIFX Broker

One tcp connection from every device
Super long tcp connection lifetimes
Many Many Many connections
Perfect thundering herd

Mesos deploys

Not great

Home baked deploy system

Shiny,

Kubenetes built in deployments

dope

New connections

Super Dope

Lessons

Services and Deployments just worked

maxSurge 100%
terminationGracePeriod 3600

11 Months from initial commit to almost finished

it is always dns

Here be dragons conjecture

Opinions here are my own

Kubernetes Suitability & Capability

Self managed Kubernetes clusters and small teams

Where does Kubernetes fit

Number of services ->	Few	Many	Many Many
1 Team	bash for loops	Probably*	Yes*
Several Teams	Perhaps Nomad or mesos?	Yes	Very yes

* Managed Kubernetes only

Blobs of yaml and bash

What could go wrong

Config Drift

So much config drift

Tooling

Communication between teams is going to be an issue
“Best practice” will evolve over time
Tooling needs to support changing deployment patterns and config patterns over time

Best Kubernetes features (IMNSHO)

Nodes are just nodes
Sidecars
Webhooks make process automation easier.

* Don't use much of Kubernetes * Economy of scale. Gets better with size. * Large clusters are better than small and per team clusters are :| We don't use much of Kubernetes. The more significant capabilities of Kubernetes require larger scales than we really use. Nodes in Kubernetes are just empty boxes with docker (for now) and a kubelet binary. This simplifies your ami builds and management requirements. You don't even need to grant access into the nodes for "local" debugging as kubelet gives you strong rbac and remote attach capabilities. Sidecars let you have composable services. This works well when you have specialised processes consumed across teams or for use with Service mesh solutions or other tooling. Webhooks are basically compliance and integration hooks. Admission controllers let you proactively check the state for all the things as they are being created or changed rather than reactively on an event. Mutation webhooks can add premade objects to resources to reduce the amount that individual teams have to manage.

Best Kubernetes features (continued)

Operators and custom resources
Abstracts infrastructure to allow common deployment systems and patterns
Tooling being built for multiple clusters

Operators == Codified Operational Knowledge. Automation to provide or manage collections of services Custom resources allow description of custom services. Like the deployment -> pod mapping etcd of 5 nodes with version x or handle upgrading your ES cluster Most services aren't special. once you satisfy their package dependencies they can be abstracted away Tools like spinnaker allow you to build templated deployment pipelines. Letting teams worry about less This common pipeline allows you to update it overtime as practices change Operators are "codified Operational knowledge" You see these as ways to manage sets of pods for various services. the elasticsearch operator or kubedb or etcd operators. These build and maintain various cluster resources in repeatable ways. This is easy to build for internal services as well. the metacontroller or coreos controller framework generate or have systems for you to use in your own operators for your own internal practices A lot of what Kubernetes provides are smoother abstractions of the underlying infrastructure. But most applications don't really need the flexibility of Kubernetes Building common tooling ontop of Kubernetes to construct in (to borrow a term from the spinnaker project) a paved road. This means that teams don't need to redevelop deployment processes or have a large amount of boilerplate Tools like spinnaker with it's templated pipelines that give you pipelines that can be used multiple times combined with the smaller contract of Kubernetes give allow you to reuse large amounts of work developed by others (either inside or outside the company)

TL;DR

Do I think you should strongly consider Kubernetes

Do I think it’s going to be easy

This migration would not have succeeded if it wasn’t for the other team members. They built a lot of the tools, developed many of the processes and did most of the migrations

Daniel
Stephen
Nick
Carl

Questions?

References coming soon

now that I’m updating these slides it’s really not that soon at all sorry :|