Kubernetes Operators: Custom Kubernetes Controller Guide

Kubernetes has emerged as the de facto standard for container orchestration, enabling developers to deploy, scale, and manage containerized applications seamlessly. However, managing complex applications in Kubernetes can still be a challenging task. Enter Kubernetes Operators, a powerful framework that extends Kubernetes’ capabilities by allowing you to automate the management of custom resources and complex applications.

In this blog post, we’ll explore the concept of Kubernetes Operators, focusing on developing and deploying custom controllers. We’ll walk through the process step by step, providing practical examples and code snippets to help you understand how to harness the power of Kubernetes Operators.

Understanding Kubernetes Operators

Kubernetes Operators are a set of custom controllers that extend the Kubernetes API to manage applications and their components. These operators leverage the declarative nature of Kubernetes manifests to automate tasks such as deployment, scaling, and maintenance of complex applications. Operators can be used to manage a wide range of applications, from databases to monitoring solutions.

The key components of a Kubernetes Operator include:

1. Custom Resource Definitions (CRDs): Defines custom resources and their specifications.
2. Custom Controllers: Watches for changes in CRDs and takes actions to reconcile the actual state with the desired state.

Helm installs and upgrades manifests. An Operator keeps watching and fixes drift: failover, scale, backups. Write an Operator when the app has a lifecycle Kubernetes does not know. If you only template YAML once, Helm or Kustomize is enough. Operators are code you will own at 2 a.m.

Use Operator SDK or Kubebuilder unless you have a reason not to. You get CRDs and a reconcile loop. From scratch you rebuild leader election and RBAC. You still owe tests and a release process.

Go is the default. APIs and most examples are Go-first. Other SDKs exist. Hiring is thinner. If the team cannot debug a reconcile loop, an Operator is a bigger bet than the YAML suggests.

Test on minikube or kind, as in the setup section. Install the CRD, apply a sample, check the child objects. Change or delete the CR and watch it reconcile. Never develop against production CRDs.

The first Operator usually does too much per event, or never requeues a failed API call. Start with one custom resource and one child object. If Helm and the operator both own the same Deployment, pick one owner.

Setting Up Your Development Environment

Before we dive into developing a custom controller, let’s set up our development environment. Ensure you have the following tools installed:

1. kubectl: Kubernetes command-line tool.
2. minikube: Lightweight Kubernetes cluster for local development.
3. Operator SDK: A toolkit for building Kubernetes Operators.

# Install kubectl
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

# Install minikube
curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Install Operator SDK
curl -LO https://github.com/operator-framework/operator-sdk/releases/latest/download/operator-sdk_linux_amd64
sudo install operator-sdk_linux_amd64 /usr/local/bin/operator-sdk

The MyApp example in the post is a teaching app. A real database operator is a different class of work (backups, users, version upgrades). Do not promise “we will operator-ize Postgres this sprint” after this tutorial.

Pin Operator SDK and kubectl versions. “latest” curl lines in a blog go stale. Check the release page the day you install.

RBAC for the operator’s ServiceAccount should be the verbs it needs on the CRs and child objects. Cluster-admin for the operator is how a bug deletes more than MyApp.

When the operator is wrong, users will kubectl apply harder. That fight is a sign the spec is unclear or two controllers are reconciling the same fields.

Status vs spec: users edit spec. The controller writes status. If you let users edit status, or you overwrite spec, you will confuse everyone including yourself.

Finalizers keep a CR around until cleanup finishes. Forget to remove the finalizer and the object sits in Terminating forever. That is the first Slack message on a bad day.

Developing a Simple Kubernetes Operator

For this example, let’s create a basic Kubernetes Operator that manages a custom resource named MyApp. This application will be a simple web service that exposes a welcome message.

Step 1: Initialize Operator Project

operator-sdk init --domain=mycompany.com --repo=github.com/mycompany/myapp-operator
cd myapp-operator

Step 2: Create a Custom Resource Definition (CRD)

Edit api/v1/myapp_types.go to define the MyApp custom resource:

// api/v1/myapp_types.go

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

// MyAppSpec defines the desired state of MyApp
type MyAppSpec struct {
  // Add fields as needed
}

// MyAppStatus defines the observed state of MyApp
type MyAppStatus struct {
  // Add fields as needed
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status

// MyApp is the Schema for the myapps API
type MyApp struct {
  metav1.TypeMeta   `json:",inline"`
  metav1.ObjectMeta `json:"metadata,omitempty"`

  Spec   MyAppSpec   `json:"spec,omitempty"`
  Status MyAppStatus `json:"status,omitempty"`
}

// +kubebuilder:object:root=true

// MyAppList contains a list of MyApp
type MyAppList struct {
  metav1.TypeMeta `json:",inline"`
  metav1.ListMeta `json:"metadata,omitempty"`
  Items           []MyApp `json:"items"`
}

func init() {
  SchemeBuilder.Register(&MyApp{}, &MyAppList{})
}

Step 3: Generate Kubernetes Controller Code

operator-sdk create api --group=myapp --version=v1 --kind=MyApp

Step 4: Implement the Kubernetes Controller

Edit controllers/myapp_controller.go to add your controller logic:

// controllers/myapp_controller.go

package controllers

import (
	"context"
	"reflect"

	myappv1 "github.com/mycompany/myapp-operator/api/v1"
	ctrl "sigs.k8s.io/controller-runtime"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

// MyAppReconciler reconciles a MyApp object
type MyAppReconciler struct {
	client.Client
	Log    logr.Logger
	Scheme *runtime.Scheme
}

// +kubebuilder:rbac:groups=myapp.mycompany.com,resources=myapps,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=myapp.mycompany.com,resources=myapps/status,verbs=get;update;patch

func (r *MyAppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	log := r.Log.WithValues("myapp", req.NamespacedName)

	// Fetch the MyApp instance
	myapp := &myappv1.MyApp{}
	err := r.Get(ctx, req.NamespacedName, myapp)
	if err != nil {
		log.Error(err, "unable to fetch MyApp")
		return ctrl.Result{}, client.IgnoreNotFound(err)
	}

	// Reconciliation logic here

	return ctrl.Result{}, nil
}

func (r *MyAppReconciler) SetupWithManager(mgr ctrl.Manager) error {
	return ctrl.NewControllerManagedBy(mgr).
		For(&myappv1.MyApp{}).
		Complete(r)
}

Step 5: Build and Deploy the Operator

operator-sdk build myapp-operator
docker push myapp-operator
operator-sdk run local --watch-namespace=default

Deploying Your Custom Resource

Now that we have our operator running, let’s create an instance of our custom resource:

# myapp-instance.yaml

apiVersion: myapp.mycompany.com/v1
kind: MyApp
metadata:
  name: example-myapp
spec:
  # Add custom spec fields as needed

Apply the resource to your cluster:

kubectl apply -f myapp-instance.yaml

Watch the logs of your operator to see the reconciliation process:

kubectl logs deployment/myapp-operator-controller-manager -n default -c manager

Congratulations! You’ve successfully developed and deployed a simple Kubernetes Operator. This example is just the tip of the iceberg; Kubernetes Operators can be extended to manage more complex applications, databases, and services.

Metrics on the reconcile loop (duration, errors, requeue count) tell you the operator is wedged before users do. Add them before you add a second CRD.

OLM (Operator Lifecycle Manager) is how some clusters install operators. It is extra machinery. For a single in-house operator, a Deployment and a YAML install is enough until you are publishing to a catalog.

Do not run the operator as root in the container unless you must. Same rules as any other pod.

Watch cache: the controller should use informers, not list the API every second. A naive loop will rate-limit you out of your own cluster. The SDK’s generated layout does this if you do not throw it away.

CRD versioning (v1alpha1 to v1) needs a conversion story. Shipping alpha to production customers is how you get stuck. Say alpha in the name until the spec is boring.

If MyApp only creates a Deployment, a Helm chart plus a CronJob may be enough. The tutorial is still worth doing so you know when you are overbuilding.

Conclusion

Kubernetes Operators empower developers to automate the management of complex applications within Kubernetes. By developing custom controllers and leveraging the power of CRDs, you can create Operators tailored to your specific needs. This blog post has provided a practical guide, complete with code examples, to help you get started on your journey to mastering Kubernetes Operators.

Remember, the true power of Operators lies in their ability to adapt and scale with your applications. As you explore this realm further, you’ll discover endless possibilities for automating and streamlining your Kubernetes workflows.

References:

• Kubernetes Docs