In OpenShift 4 it’s pretty easy to configure sending alerts to a variety of destinations including Slack. While my work tends to be more focused on the development side of the house then operations, my goal for my homelab cluster is to be as production-like as possible hence the need to configure receivers.

To send messages to Slack the first thing you will need to do is setup a slack organization if you do not already have one and then setup channels to receive the alerts. In my case I opted to create three channels: alerts-critical, alerts-default and alerts-watchdog. These channels mirror the default filtering used in OpenShift 4, of course you may want to adjust as necessary based on whatever filtering routes and rules you have in place.

Once you have your channels in place for receiving alerts, you can add incoming webhooks by following the documentation here. Each webhook you create in Slack will have a URL in the following format:

https://hooks.slack.com/services/XXXXX/XXXXX/XXXXX

The XXXXX will of course be replaced by something specific to your channel. Please note that Slack webhooks do not require authentication so you should not expose these URLs in public git repositories as it may lead to your channels getting spammed.

Once you have that, you need to configure the default alertmanager-main secret. The OpenShift documentation does a good job of explaining the process both from a GUI and a yaml perspective, in my case I prefer using yaml since I am using a GitOps approach to manage it.

An example of my Slack receivers configuration is below with the complete example on github:

  - name: Critical
    slack_configs:
    - send_resolved: false
      api_url: https://hooks.slack.com/services/XXXXXX/XXXXXX/XXXXXX
      channel: alerts-critical
      username: '{{ template "slack.default.username" . }}'
      color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'
      title: '{{ template "slack.default.title" . }}'
      title_link: '{{ template "slack.default.titlelink" . }}'
      pretext: '{{ .CommonAnnotations.summary }}'
      text: |-
        {{ range .Alerts }}
          *Alert:* {{ .Labels.alertname }} - `{{ .Labels.severity }}`
          *Description:* {{ .Annotations.message }}
          *Started:* {{ .StartsAt }}
          *Details:*
          {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
          {{ end }}
        {{ end }}
      fallback: '{{ template "slack.default.fallback" . }}'
      icon_emoji: '{{ template "slack.default.iconemoji" . }}'
      icon_url: '{{ template "slack.default.iconurl" . }}'

This configuration is largely adapted from this excellent blog post by Hart Hoover called Pretty AlertManager Alerts in Slack. With this configuration, the alerts appear as follows in Slack:

As mentioned previously, the slack webhook should be treated as sensitive and as a result I’m using Sealed Secrets to encrypt the secret in my git repo which is then applied by ArgoCD as part of my GitOps process to configure the cluster. As a security measure, in order for Sealed Secrets to overwrite the existing alertmanager-main secret you need to prove you own the secret by placing an annotation on it. I’m using a pre-sync hook to do that in ArgoCD via a job:

apiVersion: batch/v1
kind: Job
metadata:
  name: annotate-secret-job
  namespace: openshift-monitoring
  annotations:
    argocd.argoproj.io/hook: PreSync
spec:
  template:
    spec:
      containers:
        - image: registry.redhat.io/openshift4/ose-cli:v4.6
          command:
            - /bin/bash
            - -c
            - |
              oc annotate secret alertmanager-main sealedsecrets.bitnami.com/managed="true" --overwrite=true
          imagePullPolicy: Always
          name: annotate-secret
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      serviceAccount: annotate-secret-job
      serviceAccountName: annotate-secret-job
      terminationGracePeriodSeconds: 30

The complete alertmanager implementation is part of my cluster-config repo which I will be covering more in a subsequent blog post.