Catching something interesting in your logs is only half the battle; you also want to hear about it where you’re already paying attention. In this article we set up a watcher and configure it to send alerts straight to our Slack channels through a webhook.

The starting point is Kibana with Sentinl already in place. If you don’t know how to set this up, you can check our post here.

From there, the first thing we create is a watcher.

With the watcher in place, we give it an input that tells it what to look for. Our input searches the index for entries where the container name is jboss-wildfly, with the type kube-logs. In this example I’m running on the Kubernetes platform, and Kibana Sentinl manages the Kubernetes logs.

{
  "search": {
    "request": {
      "index": [
        "filebeat-*"
      ],
      "types": [
        "kube-logs"
      ],
      "body": {
        "query": {
          "match": {
            "container_name": {
              "query": "jboss-wildfly",
              "type": "phrase"
            }  } } } }
  }
}

Once it’s running, the watcher starts raising alarms, which show up on the Sentinl panel like this.

And the same alert lands in our Slack channel.