Presented to REA DevOps guild


  • Will probably not scale to larger teams
  • Practices developed for a small team
  • High adjacency to domain knowledge
  • Mostly nice services that follow most of the 12 factor pattern
  • Ignoring stateful applications

Speaker notes are incomplete they are coming soon

Where we were

  • Mesos/Marathon cluster
  • Self managed systems
  • Internally developed tools to expose services
  • Old versions of all the things
  • Super small team

Why Kubernetes?

  • Good primitives to build on
  • Good momentum
  • More integrated solutions
  • Does out of the box what we needed custom code to do

Why GKE?

  • limited ops time
  • better to spend time on more valuable tasks
  • smaller systems to manage
  • no Kubernetes controllers to manage

Migration time

First thing to do is make “all” the tools

for a very small value of all

Really just a small Go app with text/template, crypto/aes piping templates to kubectl and bash to glue it all together.

“Normal” migration process

  • Create a build pipeline
  • Create walls of yaml
  • Test deploy
  • Change dns
  • Profit?
  • Realise you missed several blobs of yaml or broken lib
  • Revert dns change
  • Update yaml, Fix libraries
  • Deploy
  • Change dns


Abnormal migrations

The LIFX Broker
  • One tcp connection from every device
  • Super long tcp connection lifetimes
  • Many Many Many connections
  • Perfect thundering herd

Mesos deploys

Not great

Home baked deploy system


Kubenetes built in deployments


New connections

Super Dope


  • Services and Deployments just worked
maxSurge 100%
terminationGracePeriod 3600

11 Months from initial commit to almost finished

it is always dns

Here be dragons conjecture

Opinions here are my own

Kubernetes Suitability & Capability

  • Self managed Kubernetes clusters and small teams
Team fortress 2 engineer saying nope

Where does Kubernetes fit

Number of services ->FewManyMany Many
1 Teambash for loopsProbably*Yes*
Several TeamsPerhaps Nomad or mesos?YesVery yes

* Managed Kubernetes only

Blobs of yaml and bash

What could go wrong

Config Drift

So much config drift


  • Communication between teams is going to be an issue
  • “Best practice” will evolve over time
  • Tooling needs to support changing deployment patterns and config patterns over time

Best Kubernetes features (IMNSHO)

  • Nodes are just nodes
  • Sidecars
  • Webhooks make process automation easier.

Best Kubernetes features (continued)

  • Operators and custom resources
  • Abstracts infrastructure to allow common deployment systems and patterns
  • Tooling being built for multiple clusters


Do I think you should strongly consider Kubernetes

Do I think it’s going to be easy

Team fortress 2 engineer saying nope

This migration would not have succeeded if it wasn’t for the other team members. They built a lot of the tools, developed many of the processes and did most of the migrations

  • Daniel
  • Stephen
  • Nick
  • Carl


References coming soon

now that I’m updating these slides it’s really not that soon at all sorry :|