Notifying Kubernetes events to Slack using Kubewatch

Tue, Jul 24, 2018 2-minute read

Kubernetes clusters are constantly generating events, but none of that activity reaches you unless you go looking for it. In this post we’ll forward those events straight into a Slack channel using bitnami-labs/kubewatch.

The kubewatch repo can be accessed using this link.

The first piece we need is a Slack bot, so I created a new one named kubewatch-webischia.

With the bot in place, I copied its token, which is what lets kubewatch post into the channel.

Now we can install kubewatch itself. It can be installed with Helm or using YAML files, and for this example I’m going with YAML files.

kubewatch-configmap.yaml

The handler scope is where we put our Slack configuration, such as the bot token, while the resource scope is where we declare which Kubernetes events we want to be notified about.

apiVersion: v1
kind: ConfigMap
metadata:
  name: kubewatch
data:
  .kubewatch.yaml: |
    namespace: "default"
    handler:
      slack:
        token: xoxb-OUR-BOT-TOKEN
        channel: kubernetes-events
    resource:
      deployment: true
      replicationcontroller: false
      replicaset: false
      daemonset: false
      services: true
      pod: true
      secret: true
      configmap: false

kubewatch-service-account.yaml

For kubewatch to see those events in the first place, it needs the right permissions, so we grant them through RBAC to let it list events.

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubewatch
rules:
- apiGroups: [""]
  resources: ["pods", "replicationcontrollers"]
  verbs: ["get", "watch", "list"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: kubewatch
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubewatch
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubewatch
subjects:
  - kind: ServiceAccount
    name: kubewatch
    namespace: default

kubewatch.yaml

Finally, here is the pod YAML, which defines the image from bitnami/tuna.

One thing we shouldn’t forget is serviceAccountName: kubewatch in the spec scope, since that ties the pod back to the permissions we just created.

apiVersion: v1
kind: Pod
metadata:
  name: kubewatch
  namespace: default
spec:
  serviceAccountName: kubewatch
  containers:
  - image: tuna/kubewatch:v0.0.1
    imagePullPolicy: Always
    name: kubewatch
    volumeMounts:
    - name: config-volume
      mountPath: /root
  - image: gcr.io/skippbox/kubectl:v1.3.0
    args:
      - proxy
      - "-p"
      - "8080"
    name: proxy
    imagePullPolicy: Always
  restartPolicy: Always
  volumes:
  - name: config-volume
    configMap:
      name: kubewatch

With everything applied, it’s time to put kubewatch to the test. I created a new service, and sure enough kubewatch notified us on our Slack channel.

Creating a pod

Deleting a pod