Advanced Scheduling in Kubernetes with Dynamic Node Pools — Part II

In our previous article, we had a glimpse of how to use node Affinity to schedule a pod on a given dynamic node pool. Here, we will continue further on pod affinity and anti-affinity.

So pod affinity, in essence, pod affinity and anti-affinity allow us to schedule pods relative to one another. This feature provides more agility in scheduling pods.

As an example, we can use pod affinity to co-locate certain types of pods or services together while anti-affinity allows us to spread pods across nodes or availability zones to avoid service interference or ensure resource availability.

As it was discussed with node affinity earlier, pod affinity and anti-affinity also have the ability to define the soft and hard requirements through,

requiredDuringSchedulingIgnoredDuringExecution
preferredDuringSchedulingIgnoredDuringExecution

There’s more to pod affinity than node affinity

When it comes to pod affinity and anti-affinity, there are some things we need to be mindful of.

• Scheduling can be notably slow on large clusters as pod affinity and anti-affinity requires a substantial amount of processing
• Cannot have an empty topologyKey for pod affinity or anti-affinity which emphasizes that the nodes in the cluster should be consistently labeled with matching the topologyKey. Some references are added latter in this article if you are unfamiliar with the term topologyKey.

Let’s have a look at the technical aspects of pod affinity and anti-affinity with a more practical scenario. Lets assume that we have two applications namely A and B with an in-memory cache application running alongside. Simply we have,

• Application A directly interact with in-memory Cache for its workloads
• Application B should spread across to avoid co-location on the same node as Application A as much as possible

Following snippet shows the manifest file for the Application-A which we discussed earlier. This needs to be co-located with In memory cache application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-type-A-co-locate-with-cache
spec:
  selector:
    matchLabels:
      app: app-A
  replicas: 3
  template:
    metadata:
      labels:
        app: app-A
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - cache
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: web-app-A
        image: k8s.gcr.io/web-app-image

Here also, we can see our manifest fields starting from apiVersion and continue. In here, the podAffinity has been declared with the scheduling option requiredDuringSchedulingIgnoredDuringExecution, which in return always co-locate the Application type A pods with pods which are having the label app:cache, assuming that the in memory cache application has been labeled with the app:cache key and value.

Lets see the manifest file for podAntiAffinity also. Here, we will try to spread the pods containing Application-B away from pods with Application-A.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-type-B-locate-away-from-A
spec:
  selector:
    matchLabels:
      app: app-B
  replicas: 2
  template:
    metadata:
      labels:
        app: app-B
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - app-A
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: web-app-B
        image: k8s.gcr.io/web-app-backend-image

As we can see in the above manifest file, we have added a soft requirement using preferredDuringSchedulingIgnoredDuringExecution to locate pods created as spread across and away from pods with Application-A. Here, the pods will be scheduled away from any pods with the label app: app-A which we have added in the manifest of our Application-A. As this is defined as a soft requirement, pods will still schedule even though the scheduling criteria does not satisfied.

This is a very simple example on how we can use pod affinity and anti-affinity on a practical scenario.

You can further refer to the official Kubernetes documentation for more information and there are lots of resources that can be found on advance scheduling as well. Here, I have added a very high level scenario for ease of explaining but it is possible to add podAffinity and podAntiAffinity in the same manifest file enhancing the scheduling capability more.

References

[1] https://kubernetes.io/blog/2017/03/advanced-scheduling-in-kubernetes/

[2] https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

[3] https://thenewstack.io/implement-node-and-pod-affinity-anti-affinity-in-kubernetes-a-practical-example/

[4] https://github.com/kubernetes/community/blob/master/contributors/design-proposals/scheduling/podaffinity.md

[5] https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#:~:text=topologyKey%20is%20the%20key%20of,Pods%20into%20each%20topology%20domain.