One of the underappreciated benefits of OpenShift is the included and supported SSO product called, originally enough, Red Hat Single Sign-On (RH-SSO). This is the productized version of the very popular upstream Keycloak community project which has seen widespread adoption amongst many different organizations.

While deploying RH-SSO (or Keycloak) from a GitOps perspective is super easy, managing the configuration of the product using GitOps is decidedly not. In fact I’ve been wanting to deploy and use RH-SSO in my demo clusters for quite awhile but balked at manually managing the configuration or resorting to the import/export capabilities. Also, while the Keycloak Operator provides some capabilities in this area, it is limited in the number of objects it supports (Realms, Clients and Users)and is still maturing so it wasn’t an option either.

An alternative tool that I stumbled upon is Keycloakmigration which enables you to configure your keycloak instance using yaml. It was designed to support pipelines where updates need to constantly flow into keycloak, as a result it follows a changelog model rather then a purely declarative form which I would prefer for GitOps. Having said that, in basic testing it works well in the GitOps context but my testing to date, as mentioned, has been basic.

Let’s look at how the changelog works, here is an example changelog file:

includes:
  - path: 01-realms.yml
  - path: 02-clients-private.yml
  - path: 03-openshift-users-private.yml
  - path: 04-google-idp-private.yml

Notice that it is simply specifying a set of files with each file in the changelog represents a set of changes to make to Keycloak, for example the 01-realms.yml adds two realms called openshift and 3scale:

id: add-realms
author: gnunn
changes:
  - addRealm:
      name: openshift
  - addRealm:
      name: 3scale

The file to add new clients to the openshift realm, 02-clients-private.yml, appears as follows:

id: add-openshift-client
author: gnunn
realm: openshift
changes:
# OpenShift client
- addSimpleClient:
    clientId: openshift
    secret: xxxxxxxxxxxxxxxxxxxxxxxx
    redirectUris:
      - "https://oauth-openshift.apps.home.ocplab.com/oauth2callback/rhsso"
- updateClient:
    clientId: openshift
    standardFlowEnabled: true
    implicitFlowEnabled: false
    directAccessGrantEnabled: true
# Stackrox client
- addSimpleClient:
    clientId: stackrox
    secret: xxxxxxxxxxxxxxxxxxxxx
    redirectUris:
      - "https://central-stackrox.apps.home.ocplab.com/sso/providers/oidc/callback"
      - "https://central-stackrox.apps.home.ocplab.com/auth/response/oidc"
- updateClient:
    clientId: stackrox
    standardFlowEnabled: true
    implicitFlowEnabled: false
    directAccessGrantEnabled: true

To create this changelog in kustomize, we can simply use the secret generator:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: sso

generatorOptions:
  disableNameSuffixHash: true

secretGenerator:
- name: keycloak-migration
  files:
  - secrets/keycloak-changelog.yml
  - secrets/01-realms.yml
  - secrets/02-clients-private.yml
  - secrets/03-openshift-users-private.yml
  - secrets/04-google-idp-private.yml

Now it should be noted that many of the files potentially contain sensitive information including client secrets and user passwords, as a result I would strongly recommend encrypting the secret before storing it in git using something like Sealed Secrets. I personally keep the generated commented out and only enable it when I need to generate the secret before sealing it. All of the files with the -private suffix are not stored in git.

Once you have the secret generated with the changelog and associated files, a Post-Sync job in ArgoCD can be used to execute the Keycloakmigration tool to perform the updates in Keycloak. Here is the job I am using:

apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-migration
  namespace: sso
  annotations:
    argocd.argoproj.io/hook: PostSync
    argocd.argoproj.io/hook-delete-policy: HookSucceeded
spec:
  template:
    spec:
      containers:
      - image: klg71/keycloakmigration
        env:
        - name: BASEURL
          value: "https://sso-sso.apps.home.ocplab.com/auth"
        - name: CORRECT_HASHES
          value: "true"
        - name: ADMIN_USERNAME
          valueFrom:
            secretKeyRef:
              name: sso-admin-credential
              key: ADMIN_USERNAME
        - name: ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: sso-admin-credential
              key: ADMIN_PASSWORD
        imagePullPolicy: Always
        name: keycloak-migration
        volumeMounts:
        - name: keycloak-migration
          mountPath: "/migration"
          readOnly: true
        - name: logs
          mountPath: "/logs"
      dnsPolicy: ClusterFirst
      restartPolicy: OnFailure
      terminationGracePeriodSeconds: 30
      volumes:
      - name: keycloak-migration
        secret:
          secretName: keycloak-migration
      - name: logs
        emptyDir: {}

In this job the various parameters are passed as environment variables. We directly mount the SSO admin secret into the container so that the tool can interact with Keycloak. The other interesting parameter to note is the CORRECT_HASHES parameter. I had some issues where if I manually changed an object the migration would refuse to run since it no longer followed the changelog. Since my environment is ephemeral and subject to troubleshooting, I opted to add this parameter to force the process to continue. I do need to test this out further before deciding whether I should leave it or remove it.

In summary, this shows one possible approach to configuring Keycloak using a GitOps approach. While my testing to this point has been very basic, I’m optimistic about the possibilities and look forward to trying it out more.