In OpenShift, Operators are typically installed through the Operator Lifecycle Manager (OLM) which provides a great user interface and experience. Unfortunately OLM was really designed around a UI experience and as a result when moving to a GitOps approach there are a few things to be aware of in order to get the best outcomes. The purpose of this blog is to outline a handful of best practices that we’ve found after doing this for awhile, so without further ado here is the list:
1. Omit startingCSV in Subscriptions
When bringing an operator into GitOps, it’s pretty common to install an operator manually and then extract the yaml for the subscription and push it into a git repo. This will often appear as per this example:
apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: labels: operators.coreos.com/amq7-cert-manager-operator.openshift-operators: "" name: amq7-cert-manager-operator namespace: openshift-operators spec: channel: 1.x installPlanApproval: Automatic name: amq7-cert-manager-operator source: redhat-operators sourceNamespace: openshift-marketplace startingCSV: amq7-cert-manager.v1.0.1
OLM will automatically populate the startingCSV for you which represents the specific version of the operator that you want to install. The problem with this is that operator versions will change regularly with updates meaning that everytime it changes you will need to update the version in the git repo. The majority of the time we simply want to consume the latest and greatest operator, omitting the startingCSV accomplishes that goal and greatly reduces the maintenance required for the subscription yaml.
Of course if you have a requirement to install a very specific version of the operator by all means include it, however in my experience this requirement tends to be rare.
2. Create OperatorGroup with namespaces
An OperatorGroup to quote the documentation “provides multitenant configuration to OLM-installed Operators”. Everytime you install an operator there must be one and only one OperatorGroup in the namespace. Some default namespaces, like openshift-operators, will have an OperatorGroup out of the box and you do not need to create a new one from GitOps. However if you want to install operators into your own namespaces you will need to have an OperatorGroup.
When using kustomize there is a temptation to bundle the OperatorGroup with a Subscription. This should be avoided because if you want to install multiple operators, say Prometheus and Grafana, in the same namespace they will create multiple OperatorGroups and prevent the operators from installing.
As a result if I need to install operators in GitOps I much prefer creating the OperatorGroup as part of the same kustomize folder where I’m creating the namespace. This allows me to aggregate multiple operators across different bases without getting into OperatorGroup confusion.
3. Omit the olm.providedAPIs annotation in OperatorGroup
Similar to startingCSV, when manually installing operators you will notice that OLM populates an annotation called olm.providedAPIs. Since OLM will populate it automatically there is no need to include this in the yaml in git as it becomes one more element that you will need to maintain.
4. Prefer manual installation mode
When installing an operator via OLM you can choose to install it in manual or automatic mode. For production clusters you should prefer the manual installation mode in order to control when operator upgrades happen. Unfortunately when using manual mode OLM requires you to approve the initial installation of the operator. While this is easy to do in the console UI it’s a little more challenging with a GitOps tool.
Fortunately my colleague Andrew Pitt has you covered and wrote an excellent tool to handle this, installplan-approver. This is a kubernetes job that you can deploy alongside the operator Subscription that watches for the installplan that OLM creates and automatically approves it. This gives you the desired workflow of automatic installation but manual approvals of upgrades.
Since this is run as a kubernetes job it only runs once and will not accidentally approve upgrades. In other words, subsequent synchronizations from a GitOps tool like Argo CD will not cause the job to run again since from the GitOps tool perspective the job aleeady exists and is synchronized.
5. Checkout the operators already available in Red Hat COP gitops-catalog
Instead of re-inventing a wheel, check out the operators that have already been made available for GitOps in the Red Hat Community of Practice (COP) gitops-catalog. This catalog has a number of commonly used operators already available for use with OpenShift Gitops (OpenShift Pipelines, OpenShift GitOps, Service Mesh, Logging and more). While this catalog is not officially supported by Red Hat, it provides a starting point for you to create your own in-house catalog and benefiting from the work of others.
Well that’s it for now, if you have more best practices feel free to add them in the comments.