# Exploring gRPC balancing ### Kevin Collas-Arundell #### [@kcollasarundell](https://twitter.com/) #### 06 Feb 2018
Not finished. Sorry !section # Caveats * Examples are in go for reasons * Probably implementable in other languages\protocols * Everything runs on Kubernetes * Code is incomplete * Bugs are included free of charge Note: reason = I'm Lazy !section ## TL;DR Frontend thing depends on a background process. ![One Frontend gopher talks to one backend gopher](/talks/balancer/oneToOne.png)<!-- .element: height="50%" width="80%" --> Note: This is how the world used to be. One front end thing with one backend bit. Nice, simple and "wonderful" !section But there aren't usually 1 to 1 relationships ![Many front end gophers to many backend gophers](/talks/balancer/manyToMany.png) <!-- .element height="60%" width="70%" --> !section ![Kubernetes logo](/talks/balancer/Kubernetes_Gophers.png) <!-- .element height="60%" width="50%" --> !section ### Kubernetes Service ```yaml kind: Service apiVersion: v1 metadata: name: rng-clusterip spec: selector: provides: rng ports: - protocol: TCP name: grpclb port: 8081 targetPort: 8081 ``` ```go conn, err := grpc.Dial( "rng-clusterip", grpc.WithInsecure() ) ``` Note: Kubernetes comes out of the box with effective loadbalancing that works in most cases. It uses the kube-dns to create a DNS name pointing at a virtual IP that only exists for kube-proxy to help route using iptables on the node. !section * Pros:<!-- .element: class="fragment fade-in" data-fragment-index="1" --> * Built in <!-- .element: class="fragment fade-in" data-fragment-index="1" --> * Simple and effective <!-- .element: class="fragment fade-in" data-fragment-index="1" --> * Automatic DNS <!-- .element: class="fragment fade-in" data-fragment-index="1" --> * Virtual IP for acccess <!-- .element: class="fragment fade-in" data-fragment-index="1" --> * TCP or UDP stream based,</br> Works for any application<!-- .element: class="fragment fade-in" data-fragment-index="1" --> * Cons: <!-- .element: class="fragment fade-in" data-fragment-index="2" --> * Deterministicish round robin <!-- .element: class="fragment fade-in" data-fragment-index="2" --> * TCP or UDP stream based,</br> Only works on the TCP protocol <!-- .element: class="fragment fade-in" data-fragment-index="2" --> Note: If this works for your use case ++. By working at the protocol layer rather than application. you don't have the ability to balance long lived connections. The deterministic routing of connections appears to be on a per node basis so small clusters could see issues with this routing !section # Bugger !section ## gRPC Client Side Balancing !section ## 3\2.5 chunks in the gRPC balancer code * Resolver: Finds * Balancer: Maintains connections * Picker: Picks which connection to send the request over Note: Pickers are kinda standalone and kinda related to the balancer. Easiest parts to swap out (spoilers) are the resolver and picker. !section ## The first thing we do, </p> Find all the pods !section ```yaml kind: Service apiVersion: v1 metadata: name: rng-headless spec: clusterIP: None selector: provides: rng ports: - protocol: TCP name: grpclb port: 8081 targetPort: 8081 ``` Note: This is the same as the first service in much the same way that bricks are just like a house. !section ```bash dig -tSRV _grpclb._tcp.rng-headless _grpclb._tcp.rng-headless.BLAH. 30 IN SRV 10 25 8081 6563663761663230.rng-headless.BLAH. _grpclb._tcp.rng-headless.BLAH. 30 IN SRV 10 25 8081 3633363663623365.rng-headless.BLAH. ... ``` <span class="fragment highlight-current-red" data-fragment-index="1">10 </span> <span class="fragment highlight-current-red" data-fragment-index="2">25 </span> <span class="fragment highlight-current-red" data-fragment-index="3">8081 </span> <span class="fragment highlight-current-red" data-fragment-index="4">36...5.rng-headless.BLAH.</span> Note: Priority, Weighting, Port, cname\alias record Swap blah with the full domain name !section ```go conn, err := grpc.Dial("dns:///rng-headless", grpc.WithInsecure(), grpc.WithBalancerName(roundrobin.Name), ) ``` <span class="fragment fade-up" data-fragment-index="1"> "<span class="fragment highlight-red" data-fragment-index="1">dns://</span ><span class="fragment highlight-blue" data-fragment-index="2" >/</span ><span class="fragment highlight-green" data-fragment-index="3">rng-headless</span >" </span> </p> <span class="fragment fade-up" data-fragment-index="1"> "<span class="fragment highlight-red" data-fragment-index="1">scheme://</span ><span class="fragment highlight-blue" data-fragment-index="2">authority/</span ><span class="fragment highlight-green" data-fragment-index="3">endpoint</span >" </span> </p> <span class="fragment fade-up" data-fragment-index="4"> grpc.WithBalancerName(<span class="fragment highlight-red" data-fragment-index="5">roundrobin.Name</span>) </span> Note: type Target struct { Scheme string Authority string Endpoint string } scheme == how to look, authority == where to look, endpoint == what to look for. Scheme specifies which resolver to use to handle lookups roundrobin.Name is how we specify which picker to use !section ## And that's it !section ## Or Not * Downsides * DNS resolution only happens on connection change or every 30 minutes * Large environments will spend more time on connection management than using them. Note: Dns resolution takes a looooong time if you are just scaling up. Connection growth rate gets silly very quickly as environments grow !section ## 3 Components in the grpc balancer stuff * Resolver: Finds * Balancer: Maintains connections * Picker: Picks which connection to send the request over !section ## Round robin doesn't work for everything !section ```go type PickerBuilder interface { Build( readySCs map[resolver.Address]balancer.SubConn ) balancer.Picker } type Picker interface { Pick( ctx context.Context, opts PickOptions ) (conn SubConn, done func(DoneInfo), err error) } ``` Note: Pickers are the most frequently touched part of the balancer as every single call against the gRPC connection will use the picker. Pickers decide which sub Connection to use for that particular call. !section ```go const baseName = "consistentHash" func NewBuilder(key interface{}) balancer.Builder { name := baseName + fmt.Sprintf("%v", key) return base.NewBalancerBuilder( name, &hashPicker{ Name: name, key: key }) } ``` !section ```go type contextValue string ``` ```go key = contextValue("userid") b := consistentHashBalancer.NewBuilder(key) balancer.Register(b) conn, err := grpc.Dial( "dns:///rng-headless", grpc.WithBalancerName(b.Name()) ) ``` !section ```go func (h *hashPicker) Build( readySCs map[resolver.Address]balancer.SubConn ) balancer.Picker { r := newRing(100) for address, sc := range readySCs { r.addNode(address, sc) } return &hashPicker{ key: h.key, subConns: r, } } ``` !section ```go func (p *hashPicker) Pick( ctx context.Context, opts balancer.PickOptions, ) (balancer.SubConn, func(balancer.DoneInfo), error) { if p.subConns.size <= 0 { return... } value := ctx.Value(p.key) s, ok := value.(string) if !ok { return nil, nil, "derp"} sc := p.subConns.get(s) return *sc.subConn, nil, nil } ``` !section ## Replacing the resolver !section ### Need something to build the resolver ```go type Builder interface { Build( target Target, cc ClientConn, opts BuildOption, ) (Resolver, error) Scheme() string } ``` ### and interact with it once it's running ```go type Resolver interface { ResolveNow(struct{}) Close() } ``` Note: the target mentioned above type Target struct { Scheme string Authority string Endpoint string } !section ```go type Resolver struct { scheme string cc resolver.ClientConn source resolve.ResolveClient } ``` !section ```go type Resolver struct { scheme string cc resolver.ClientConn source resolve.ResolveClient } ``` !section ```go func (r *Resolver) Build(stuff) (resolver.Resolver, error) { r.cc = cc conn, _ := grpc.Dial(target.Authority) r.source = resolve.NewResolveClient(conn) c, _ := r.source.ResolveStream( context.Background(), &resolve.Source{Name: target.Endpoint}, ) go r.stream(c) return r, nil } ``` Note: // NewAddress is called by resolver to notify ClientConn a new list // of resolved addresses. // The address list should be the complete list of resolved addresses. // NewServiceConfig is something we're going to ignore for now. !section ```go for { rawAddresses, _ := stream.Recv() var backends []resolver.Address for _, rawAddress := range rawAddresses.Name { backends = append(backends, resolver.Address{ Addr: rawAddress, }) } r.cc.NewAddress(backends) } ``` !section # Here be unfinished dragons !section ```go func subset( backends []resolver.Address, clientID, subsetSize int, ) []resolver.Address { subsetCount := len(backends) / subsetSize round := int64(clientID / subsetCount) subsetID := clientID % subsetCount ``` Note: This is mainly setup for the second slide. this is all reducing the number of backends you connect to. subsetSize == number of connections you want to establish The clientID needs to be an idenity that is an evenly distributed. Rounds are roughly speaking groups of nodes that base off the same shuffled list. subsetID is where in the shuffled slice you should look !section ```go // shuffle the slice of addresses r := rand.New(rand.NewSource(round)) shuffledBackends := make([]resolver.Address, len(backends)) for i, o := range r.Perm(len(backends)) { shuffledBackends[i] = backends[o] } start := subsetID * subsetSize return shuffledBackends[start : start+subsetSize] ``` Note: each round shuffles the slice with a known seed to keep it the same throughout a round. each we then pick which part of the slice to return to the caller. == a very even distribution of connections !section ![Connection distribution](/talks/balancer/sresubset.jpg) !section # Questions? <video data-autoplay loop src="/talks/balancer/cat.mp4"></video> Note: Cat is there to distract you from asking questions !section ## Acknowledgements * [GopherCon 2017: Alan Shreve - gRPC: From Tutorial to Production](https://www.youtube.com/watch?v=7FZ6ZyzGex0) * [SRE your gRPC talk](https://www.youtube.com/watch?v=eoy9z0UlaII) * [SRE your gRPC](https://www.usenix.org/sites/default/files/conference/protected-files/srecon17asia_slides_sheerin.pdf) * [SRE Book](https://landing.google.com/sre/book/chapters/load-balancing-datacenter.html) * made with [Reveal.js](reveal.js) * [Laurent Aphecetche](https://github.com/aphecetche) for the hugo\reveal integration * [@ashleymcnamara](https://twitter.com/ashleymcnamara) for the art of awesome * [Reimplementing Maglev in rust](https://www.youtube.com/watch?v=_GRM1Ij_3t0)