Perpetual list of stuff to procrastinate about

This will hopefully be a somewhat up to date dump of what my brain is working through over time.

Operational and Development Stuff

Developer Responsibility

• Pushing responsibility left
  • SLOs as part of the developer KPIs
  • Keeping the product inside the creating team
  • Long term ownership
  • The job is not over until it’s deleted
• Confidence
  • Context sharing difficulties
  • Compatibility guarantees
  • Beyonce rule
  • Infrastructure upgrade responsibility
• Business Goals
  • QP$
  • new functionality vs reliability vs priorities
• SLOs as conversation tools
  • Using the SLO to decide what to work on
  • Feature vs reliability

Services as a graph

• Dependency modeling
  • I Depend on these services which depend on those
  • These services depend on me
  • Who just changed?
• Oops I forgot that service
• Who owns this?
• Where are the Docs!
• What’s in the box?

Thoughts about On call

Combining markdown from multiple repositories into a single docsy site.

ADR tools

Honeycomb.io for mystery investigations

• The dangers of nanoservices\microservices
• Cardinality
• Long term vs short term investigation\planning
• Most queries are boring
  • Discarding boring stuff
  • Higher rates for errors
  • Higher rates for “that’s weird”
• Comparison
• The importance of speed

IAP tool roundup

• pomerium
• oauth2-proxy
• nginx auth lookup

Deploying

Pipeline 1 to 10

• Code delivery concept
• Benefits for single path
• Optimising for frequency
• Dynamic pipelines
• Fan out and in

DEPLOY DEPLOY DEPLOY

• Why deployment velocity matters
  • Value only comes when people use it
  • Quicker delivery == smaller experiments
• Frequent deployments
  • Build process understanding
  • Standardise deployment process with incident process
  • Consistent expectations
  • Run book memorisation or removal
• Validation
  • Constant validation of deployment tooling
  • Validation of function and performance as part of your deploy automation
• Org wide goals
  • Standardise on deployment expectations
  • Maintain ability to experiment
• Focus on path to prod

CD Round up

Coordination is important as stuff runs in multiple environments\countries\regions. Big bang releases are generally the worst possible outcome due to the substantial risk involved

• Coordination vs Independence
• The mistakes of manual pipelines
• Gitops
  • Argo
  • Flux
• CI Repurposed
  • Github Actions
  • Gitlab CI
  • Jenkins
  • Buildkite
• SAAS solutions
  • Circle CI

Faster builds by being lazy

• Monorepos in 5 minutes
• Bazel
• Speed
  • Why action Cache
  • Reuse tests from others
  • Slow tests are unused
  • Parallelised builds
  • Remote Builds
    • Cross compile
• Explicit dependency trees
  • Enables build reuse
  • action graph
  • Clear dependency context
  • Cross Service testing
• MultiLanguage support
  • Bundling to provide a smoother onboarding
  • Only testing what you care about
    • Test me when what I depend on changes
    • Test those who depend on me when I change
• Downstream Triggers

GKE thread

• Many Clusters vs Multi Cluster Vs BIG Clusters
• Cluster maintenance requirements
• Benefits of Spinnaker (feed off CD roundup?)
• Dynamic deployment pipelines
• Pulling to a new cluster
• Blueprinted deployment patterns
• Service mesh or mess

Multi Cluster Operations

• Submariner to connect across clusters
• State options
• Admirality for deploy federation

Cluster-API is a magic wtf.

Canarying your traffic

• The test environment is prod? Always has been
  • Differences
    • Scale
    • Traffic load and patterns of load
    • Do you know how long your service has been running?
    • Concurrent load and sustained load
    • Spikiness
  • Hyrums law
  • Longer term testing
• Canary tools
  • Flagger
  • Spinnaker
  • ??
• Pre Merge Canarying

kNative and Gloo

• Minimising the width of the interface contract
• API gateway vs mesh
• 80/20 implementations

Krustlet investigation

• What is Krustlet
• Wasm vs wasi
• Rust for great justice

API Interfaces

RPC vs Queue solutions

• Conclusion
• Queuing solutions
  • What you get
    • Buffering
    • Asynchronous
  • What you need to build
    • Error mechanism
    • Response channel
    • Back pressure
  • What do you still need to configure
    • topic information
    • Queue location
    • message specification
  • What you need to worry about
    • Scaling
    • Single distributed point of failure
    • Merging of control plane and data plane
    • Idempotency
• RPC solutions
  • What you get
    • Response channel
    • Error mechanism
  • What you need to build
    • Asynchronous patterns
    • Back Pressure
    • Idempotency
  • What do you still need to configure
    • Endpoint discovery
    • I specification
  • What you need to worry about
    • Scaling
    • Control plane solutions
    • Idempotency

gRPC for programatic conversion

• One IDL
• IDL First API creation
• Forward and backward versioning
• Multiple External Protocols
  • Soap Proxy
  • Rest
  • Graphql

Book thingies to write

• Non fiction
  • SWE at google
  • gopl.io
  • Seeking SRE
  • SRE and SRW books
  • Philosophy of software design
• Fiction
  • Pratchett

Overly complex and not sensible in any way webcam solution

• Route audio and camera feed to multiple computers in a local network
• Restrictions:
  • Due to driver signing on MacOS video source MUST appear as a UVC webcam to support slack and etc.
  • to reduce duplicate speakers audio out from client computers should rout e to source box for mixing
  • Software installation is not possible on some devices
• Raspberry Pis 4 running as a usb gadget
  • Composite USB device configured to be:
    • a sound card providing:
      • microphone in
      • speaker out
    • UVC webcam
  • Speaker out routed to source\speaker box with pulse audio
  • Video streamed to pi with RTMP
• One overly complex solution