# Kubernetes at LIFX ### Kevin Collas-Arundell #### [@kcollasarundell](https://twitter.com/) #### 09 Aug 2018
Presented to REA devops guild

Caveats

  • Will probably not scale to larger teams
  • Practices developed for a small team
  • High adjacency to domain knowledge
  • Mostly nice services that follow most of the 12 factor pattern
  • Ignoring stateful applications
Speaker notes are incomplete they are coming soon

Where we were

  • Mesos/Marathon cluster
  • Self managed systems
  • Internally developed tools to expose services
  • Old versions of all the things
  • Super small team

Why Kubernetes?

  • Good primatives to build on
  • Good momentum
  • More integrated solutions
  • Does out of the box what we needed custom code to do

Why GKE?

  • limited ops time
  • better to spend time on more valuable tasks
  • smaller systems to manage
  • no kubernetes controllers to manage

Migration time

First thing to do is make “all” the tools

for a very small value of all

Really just a small Go app with text/template, crypto/aes piping templates to kubectl and bash to glue it all together.

“Normal” migration process

  • Create a build pipeline
  • Create walls of yaml
  • Test deploy
  • Change dns
  • Profit?
  • Realise you missed several blobs of yaml or broken lib
  • Revert dns change
  • Update yaml, Fix libraries
  • Deploy
  • Change dns

Holiday

Abnormal migrations

The LIFX Broker
  • One tcp connection from every device
  • Super long tcp connection lifetimes
  • Many Many Many connections
  • Perfect thundering herd

Mesos deploys

Not great

Home baked deploy system

Shiny,

Kubenetes built in deployments

dope

New connections

Super Dope

Lessons

  • Services and Deployments just worked
maxSurge 100%
terminationGracePeriod 3600

11 Months from initial commit to almost finished

it is always dns

Here be dragons conjecture

Opinions here are my own

Kubernetes Suitability & Capability

  • Self managed Kubernetes clusters and small teams

Nope

Where does Kubernetes fit

Number of services -> Few Many Many Many
1 Team bash for loops Probably* Yes*
Several Teams Perhaps Nomad or mesos? Yes Very yes

* Managed kubernetes only

Blobs of yaml and bash

What could go wrong

Config Drift

So much config drift

Tooling

  • Communication between teams is going to be an issue
  • “Best practice” will evolve over time
  • Tooling needs to support changing deployment patterns and config patterns over time

Best kubernetes features (IMNSHO)

  • Nodes are just nodes
  • Sidecars
  • Webhooks make process automation easier.

Best kubernetes features (continued)

  • Operators and custom resources
  • Abstracts infrastructure to allow common deployment systems and patterns
  • Tooling being built for multiple clusters

TL;DR

Do I think you should strongly consider Kubernetes

Do I think it’s going to be easy

Nope

This migration would not have succeeded if it wasn’t for the other team members. They built a lot of the tools, developed many of the processes and did most of the migrations

  • Daniel
  • Stephen
  • Nick
  • Carl

Questions?

References coming soon