Writing a Kubernetes Operator in Golang: Keeping Your Helm Charts Always Up to Date

As a developer, ensuring that your users always have access to the latest version of your product is a top priority. Whether it's new features, bug fixes, or enhancements, you want to deliver the best experience seamlessly. As we navigate the ever-evolving landscape of Kubernetes and containerization, it’s apparent that distributing applications via Helm has become not just a preference but a necessity, creating a unique set of challenges when ensuring our customers are up to date.

Traditionally, Helm chart updates require users to initiate the update process by running commands like `helm update` or `helm upgrade`. This manual intervention can lead to delays in adopting new features, security patches, or bug fixes. Users may not always be aware of the availability of updates or may forget to execute these commands. Moreover, Helm charts often have dependencies on other charts or external services, which adds complexity to the update process.

Instead, imagine a scenario where your product could automatically update itself on your customers' clusters whenever a new version becomes available. That's the power of Kubernetes operators.

In this blog post, we'll guide you through creating a Kubernetes operator in Golang to solve this unique use case, and show a simplified version of the operator we ourselves use for own product. We'll use the `operator-sdk` framework to create the boilerplate needed for the operator and streamline the development process. By the end of this post, you'll have a working operator that keeps your Helm charts up to date on your customers' clusters.

# Understanding Kubernetes Operators

Kubernetes operators are a powerful abstraction for managing complex, stateful applications on Kubernetes. They extend the Kubernetes API, enabling you to define, deploy, and manage custom resources that represent your applications. Operators automate common operational tasks, such as provisioning, scaling, updating, and monitoring, making it easier to run and maintain complex applications in a Kubernetes-native way.

At the heart of Kubernetes operators are Custom Resource Definitions (CRDs). CRDs are extensions of the Kubernetes API that allow you to define your custom resources. These custom resources represent your applications, services, or any other entities you want to manage with an operator.

When developing an operator, you create one or more CRDs to define the structure and schema of your custom resources. In our case, we will define a `ChartRelease` resource with fields like `name`, `repo`, and `version`. Operators are designed to watch for changes to CR’s and take actions based on those changes. They follow a reconciliation loop pattern to maintain the desired state of custom resources, where the operator compares the current state of a resource with its desired state, as defined in the CR spec.

# Initializing a Kubernetes Operator Project

## Prerequisites

Before we dive into coding, make sure you have the following tools installed:

- Go programming language
- `kubectl`
- [`operator-sdk`](https://sdk.operatorframework.io/docs/installation/)

To create a new operator project, run:

```bash
operator-sdk init my-chart-operator \
--domain=my-chart.io \
--repo github.com/example/chart-operator
```

Your project directory should now look like this:

```
.
├── Dockerfile
├── Makefile
├── PROJECT
├── README.md
├── config/
│   └── ...
├── go.mod
├── go.sum
├── hack
└── main.go
```

The `init` command generated a Go module with a main program (`main.go`) which we will use to run the Manager - a component responsible for registering our API definitions and running the Controllers (the operator’s smallest unit). But more on that later.

## Creating Your First API

As mentioned earlier, Kubernetes operators are worthless without CRDs, so let’s create our first one:

```bash
operator-sdk create api \
--group mychart \
--version v1 \
--kind ChartRelease \
--resource --controller
```

You should notice two new directories added to your project:

```
.
├── api/
│   └── v1/
│       └── chartrelease_types.go
└── controllers/
   └── chartrelease_controller.go
```

### Defining the CRD

In the newly-generated `api` directory you will find `chartrelease_types.go`. This is where we can modify the CRD so that it meets our needs. In our case, we want the CRD to look somewhat as follows:

```go
type ChartReleaseSpec struct {
Repo string
Name string
}

type ChartReleaseStatus struct {
Version string
}
```

We’ve just defined a ****spec**** and a ******status****** for our CRD:

- A ****spec**** defines the schema of the CRD. In our case, we want to include the Helm repo and the name of the Helm chart.
- A ******status****** tracks the current state of the CRD. We will use it to tell which version of the Helm chart is currently installed in the cluster.

‍

### Implementing the Controller

Another directory generated by the `create api` command is `controllers`, and inside it a ready-to-use `chartrelease_controller.go` file. Each Controller must implement the `Reconcile` function, which is the controller’s main logic:

```go
func (r *ChartReleaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
// Get the CRD that triggered reconciliation
chartRelease := &mychartv1.ChartRelease{}
if err := r.Get(ctx, req.NamespacedName, chartRelease); err != nil {
return ctrl.Result{}, err
}

// Do some logic...

// Report back to Manager
return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil
}
```

As we can see, the reconciliation loop consists of three steps:

1. Retrieve the CRD relevant for this reconciliation
2. Perform necessary changes in the cluster as defined in the CRD
3. Report back to the Manager - did the reconciliation succeed? And, optionally, do we want to schedule another reconciliation for this CRD? (In our case - YES! We want the controller to check for Helm chart updates every 5 minutes)

It’s worth mentioning that by running `operator-sdk create api` we automatically registered the new Controller in `main.go`:

```go
if err = (&controllers.ChartReleaseReconciler{
Client: mgr.GetClient(),
Scheme: mgr.GetScheme(),
}).SetupWithManager(mgr); err != nil {
setupLog.Error(err, "unable to create controller", "controller", "ChartRelease")
os.Exit(1)
}
```

Good for us!

# Polling Helm Chart Updates

We’ve shown the basic implementation of Kubernetes controller, how about getting to work? We’re looking to auto-update our Helm chart whenever a new version is available, which can be broken down into mini-tasks:

1. Fetch the list of versions available for the Helm chart
2. Compare the latest version with the one installed in the cluster
3. If a newer version is available →
   1. Install the newer version in the cluster
   2. Update the `ChartRelease` CRD status to show the latest installed version

## Fetching Helm Chart Versions

Retrieving the list of versions for a Helm chart is pretty simple. The following code snippet does exactly this for [OCI-based registries](https://helm.sh/docs/topics/registries/), however it’s easy to achieve the same result for other registries, with some minor tweaks.

```go
import (
"net/url"
"path"
"helm.sh/helm/v3/pkg/registry"
)

func getLatestVersion(chartRelease *mychartv1.ChartRelease) (string, error) {
// Get the chart's info as defined in the CRD
repoURL := chartRelease.Spec.Repo
chartName := chartRelease.Spec.Name
chartURL := path.Join(repoURL, chartName)

tags, err := listVersions(resgitryClient, chartURL)
if err != nil {
return "", err
}
return tags[0], nil
}

func listVersions(chartURL string) ([]string, err) {
registryClient, err := registry.NewClient(registry.ClientOptWriter(io.Discard))
if err != nil {
return nil, err
}

// returns a sorted list of tags
return registryClient.Tags(chartURL)
}
```

## Comparing with Installed Chart

Remember the CRD status we’ve defined earlier? Now is a good time to use it! We’ll compare the latest version that we’ve found with the one installed in the cluster:

```go
func shouldUpgrade(chartRelease *mychartv1.ChartRelease, latestVersion string) bool {
installedVersion := chartRelease.Status.Version
return latestVersion != installedVersion
}
```

Note that the `latestVersion` in this code example isn’t necessarily newest, or most up-to-date. To address this, we can introduce a more complex and realistic comparison here, for example by requiring `latestVersion > installedVersion` (semantically), or satisfying a version constraint (say `vX.0.0` or a pre-release tag `vX.Y.Z-alpha`).

## Helming Around 🪖

Finally, if we realize the chart must be updated, we can use the Helm Go SDK to do so:

```go
import (
"context"
"os"
"helm.sh/helm/v3/pkg/action"
"helm.sh/helm/v3/pkg/chart"
"helm.sh/helm/v3/pkg/chart/loader"
"helm.sh/helm/v3/pkg/cli"
"helm.sh/helm/v3/pkg/registry"
)

func (r *ChartReleaseReconciler) update(ctx context.Context, chartRelease *mychartv1.ChartRelease, version string) error {
actionConfiguration, err := createHelmAction(chartRelease.Namespace)
if err != nil {
return err
}

// Specify the `helm upgrade` options
upgrade := &action.NewUpgrade(actionConfiguration)
upgrade.Version = version
upgrade.Namespace = chartRelease.Namespace

// Pull the requested chart
chartURL := path.Join(repoURL, chartName)
chart, err := getChart(upgrade, chartURL)
if err != nil {
return err
}

// Run the `helm upgrade` command
// We're using `nil` for our chart values, but you may use any `map[string]interface{}`
release, err := upgrade.Run(chartRelease.Spec.Name, chart, nil)
if err != nil {
return err
}

// Update the CRD status
chartRelease.Status.Version = release.Chart.Metadata.Version
return r.Status().Update(ctx, chartRelease)
}

func getChart(upgrade *action.Upgrade, url string) (*chart.Chart, error) {
fp, err := upgrade.LocateChart(url, cli.New())
if err != nil {
return nil, err
}
return loader.Load(fp)
}

func createHelmAction(namespace string) (*action.Configuration, error) {
settings := cli.New()
settings.SetNamespace(namespace)
actionConfig := new(action.Configuration)
if err := actionConfig.Init(settings.RESTClientGetter(), namespace, os.Getenv("HELM_DRIVER"), klog.Infof); err != nil {
return nil, erro
}

registryClient, err := registry.NewClient(registry.ClientOptWriter(io.Discard))
if err != nil {
return nil, err
}

actionConfig.RegistryClient = registryClient
return actionConfig, nil
}
```

# Let’s Play 🎮

To see our controller in action, let’s first set up a CRD to play with. Luckily, we can run `make manifests` and have `operator-sdk` create the CRD files for us (`config/samples/mychart_v1_chartrelease.yaml`).

Let’s add some spice to it:

```yaml
apiVersion: mychart.my-chart.io/v1
kind: ChartRelease
metadata:
 labels:
   app.kubernetes.io/name: chartrelease
   app.kubernetes.io/instance: chartrelease-sample
   app.kubernetes.io/part-of: my-operator
   app.kubernetes.io/managed-by: kustomize
   app.kubernetes.io/created-by: my-operator
 name: chartrelease-sample
spec:
 Repo: oci://ghcr.io/example
Name: my-chart
```

We’re pretty much done, all that’s left to do is let the controller run!

To do so, let’s run `make install run`:

```bash
$ make install run

kustomize installed to /Users/assafavital/Projects/k8sop/bin/kustomize
customresourcedefinition.apiextensions.k8s.io/chartreleases.mychart.my-chart.io created
go fmt ./...
go vet ./...
go run ./main.go
2023-09-07T13:28:16+03:00       INFO    controller-runtime.metrics      Metrics server is starting to listen    {"addr": ":8080"}
2023-09-07T13:28:16+03:00       INFO    setup   starting manager
2023-09-07T13:28:16+03:00       INFO    Starting server {"kind": "health probe", "addr": "[::]:8081"}
2023-09-07T13:28:16+03:00       INFO    Starting server {"path": "/metrics", "kind": "metrics", "addr": "[::]:8080"}
2023-09-07T13:28:16+03:00       INFO    Starting EventSource    {"controller": "chartrelease", "controllerGroup": "mychart.my-chart.io", "controllerKind": "ChartRelease", "source": "kind source: *v1.ChartRelease"}
2023-09-07T13:28:16+03:00       INFO    Starting Controller     {"controller": "chartrelease", "controllerGroup": "mychart.my-chart.io", "controllerKind": "ChartRelease"}
2023-09-07T13:28:16+03:00       INFO    Starting workers        {"controller": "chartrelease", "controllerGroup": "mychart.my-chart.io", "controllerKind": "ChartRelease", "worker count": 1}
```

The command created the CRD in our cluster, and started running the controller locally (outside the cluster). However, nothing’s happening yet. Why? Because we must first add our CRD instance to the cluster: `kubectl apply -f config/samples/mychart_v1_chartrelease.yaml`.

Et voilà - our controller picked up on it:

```yaml
2023-09-07T13:43:17+03:00       INFO    Reconciling ChartRelease        {"controller": "chartrelease", "controllerGroup": "mychart.my-chart.io", "controllerKind": "ChartRelease", "ChartRelease": {"name":"chartrelease-sample","namespace":"default"}, "namespace": "default", "name": "chartrelease-sample", "reconcileID": "55f56b69-939c-407f-829a-128ca570c879"}
```

Dying to know if it worked? Simply look at the CRD’s status (`kubectl describe chartrelease chartrelease-sample`) and look for the “Version” field.

## Going Remote

Running the operator locally is fun and all, but obviously we are looking to deploy it to the cluster. Luckily for us (again 🌟) `operator-sdk` makes it so easy for us:

```bash
make deploy IMG="<your docker image>"
# e.g. make deploy IMG="example.com/chart-operator:v1"
```

This command compiles the operator, builds it as a Docker image, pushes it to the specified registry, and finally deploys the operator to your cluster:

```bash
$ kubectl get pods

NAME                                  READY   STATUS    RESTARTS   AGE
controller-manager-5758b9846c-lrqpk   1/1     Running   0          3m
```

# Conclusion

In this comprehensive guide, we've explored the world of Kubernetes Operators and how they can revolutionize the way you manage and automate your applications on Kubernetes. As a developer, you've learned how to keep your users up to date with the latest chart versions effortlessly.

By harnessing the power of Kubernetes Operators, you can achieve the following goals:

1. **Automated Updates**: Kubernetes Operators enable you to automate the process of updating your Helm charts on your customers' clusters whenever new chart versions become available. This eliminates the need for users to manually run `helm update`, ensuring they always have access to the latest features and bug fixes.
2. **Understanding Operators**: We've delved into the core concepts of Kubernetes Operators, including their reliance on Custom Resource Definitions (CRDs) to represent and manage your applications. CRDs serve as the foundation for defining the desired state of your resources and driving the reconciliation process.
3. **Initializing Your Operator Project**: You've learned how to kickstart your operator project using the `operator-sdk`, ensuring that it's ready to manage your Helm charts effectively. This includes setting up the necessary tools and dependencies.
4. **Local Testing**: We've covered the importance of testing your operator locally to verify that it functions correctly before deploying it to a live Kubernetes cluster. This step ensures that your operator behaves as expected and prevents issues from arising in a production environment.
5. **Preparing for Publishing**: Once you've thoroughly tested your operator, we've outlined the steps to prepare it for publication. You've seen how to build container images, create Kubernetes manifests for custom resources and RBAC rules, and deploy your operator to a cluster.

By following the guidelines presented in this blog post, you're well-equipped to create a Kubernetes Operator in Golang that keeps your Helm charts automatically up to date on your customers' clusters. This automation not only enhances the user experience but also streamlines the management of your applications, ensuring they remain current, reliable, and hassle-free.

# What’s Next?

We have a minimal operator up and running, however there’s a lot we can do to further enhance it:

1. Filter events that trigger reconciliation using [predicate.Funcs](https://pkg.go.dev/sigs.k8s.io/controller-runtime/pkg/predicate@v0.14.6#Funcs).
2. Support definition of Helm values in CRDs.
3. Add logging and analytics to monitor Helm installation in your customers’ clusters.
4. Create self-healing mechanism to make sure your Helm chart is running and healthy.

And of course, the sky’s the limit! ☁️

‍