This is a story about Derps and Kubernetes

kca.id.au/t/o/

The Derps in this talk are fictional Derps.

They are the product of the authors mind.

Any resemblance to actual Derps is coincidental.

There are 2 types of people:

Those who have broken Production

and those who haven’t

Yet

Kubernetes logo

Devops borat explaining that to error is human but devops repeats it 1000 times

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: real-ingress
    namespace: prod
spec:
  rules:
  - host: foo.a-bad.dev
    http: {}

A few months later

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: dev-ingress
  namespace: dev
spec:
  rules:
  - host: foo.a-bad.dev
    http: {}

Um Kevin, Why did the API service stop working

A gif from the Simpsons of homer mixing cereal and milk, It then combusts

git revert -m 'Kevin derped' && git push origin master -f

Let’s not do that again

  1. Run a PIR
  2. Write up your PIR
  3. Action items for everyone!
  4. Hopefully fix them
the logo for excel

Write a runbook and get people to not do the thing

All runbooks are incorrect. Some are useful

How to stop your derps breaking production

Again

The logo for the open policy agent
A dolphin dressed as an admiral

Gatekeeper

You Shall Not pass

apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: duplicateingress
spec:
  crd:
    spec:
      names:
        kind: duplicateIngress
        listKind: duplicateIngressList
        plural: duplicateIngresses
        singular: duplicateIngress
  targets:
- target: admission.k8s.gatekeeper.sh
  rego: |
    package k8suniqueingresshost

    violation[{"msg": msg}] {
      input.review.kind.kind == "Ingress"
      host := input.review.object.spec.rules[_].host
      other := data.inventory.namespace[_][_]["Ingress"][_]
      other.spec.rules[_].host == host
      msg := sprintf(
        "https/kca.id.au/talks/opa-devfest/images/no.gif"
        )
    }
apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: "opa-operator"
spec:
  sync:
    syncOnly:
      - group: "networking.k8s.io"
        version: "v1beta1"
        kind: "Ingress"
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: duplicateIngress
metadata:
  name: kevin-is-terrible-at-names
spec:
  match:
    kinds:
      - apiGroups: ["networking.k8s.io"]
        version: ["v1beta1"]
        kinds: ["Ingress"]
Error from server ([denied by kevin-is-terrible-at-names],
    "https/kca.id.au/talks/opa-devfest/images/no.gif")

a gif of Tracy Morgan shaking their head saying no

Let’s find some nails

A Screenshot of policies developed by the gatekeeper team

Conftest

  • Part of your CI build
  • Yaml, Dockerfile, experimental HCL2
  • deny[] instead of violation[]

But wait. there’s tests

test_collision {
    input := {"review": review(ingress(*stuff*)}
    inv := inventory_data([ingress( *otherstuff* )])
    results := violation
      with input as input
      with data.inventory as inv
    count(results) == 1
}

Serious time

What’s the normal case?

invalid name

invalid name

invalid name

Getting Better

  • Challenge your assumptions and designs
  • Review experiences of people from different backgrounds.
  • Look for experts in the field and engage them.
  • Diverse teams deliver better.

kca.id.au/t/o/

References:

violation[{"msg": msg, "details": {}}] {
  c := input_containers[_]
  c.securityContext.privileged
  msg := sprintf(
  "But no %v, securityContext: %v",
    [c.name, c.securityContext]
  )
}
input_containers[c] {
    c := input.review.object.spec.containers[_]
}
input_containers[c] {
    c := input.review.object.spec.initContainers[_]
}
apiVersion: templates.gatekeeper.sh/v1beta1
kind: ConstraintTemplate
metadata:
  name: k8srequiredlabels
spec:
  crd:
    spec:
      names: {}
      validation:
        openAPIV3Schema: {}
openAPIV3Schema:
  properties:
    message:
      type: string
    labels:
      type: array
      items:
        type: object
        properties:
          key:
            type: string
          allowedRegex:
            type: string
violation[{}] {
  provided := {label |
    input.review.object.metadata.labels[label]}
  required := {label |
    label := input.parameters.labels[_].key}
  missing := required - provided
  count(missing) > 0
}
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredLabels
metadata:
  name: owners
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Namespace"]
  parameters:
    message: "terrible error message"
    labels:
      - key: owner
        allowedRegex: "^[a-zA-Z]+.a-bad.dev$"
@kcollasarundell