Building a k8s CRD Go (golang) Client With kubebuilder

This is a follow up to my article about generating a CRD and Golang client.

In my previous article, I generated a CRD and Go client for that CRD with code-generator. My requirements changed slightly for the project I was working on and now the CRD objects I’m going to be creating are NOT going to be namespaced (they will have “scope: Cluster” in the CRD definition), so I needed to do some regeneration. I decided I wanted to see if kubebuilder was a better experience.

Let’s get started….

mkdir examplecrd2
cd examplecrd2
go mod init github.com/jasonhancock/examplecrd2

kubebuilder init --domain jasonhancock.com --repo github.com/jasonhancock/exammplecrd2
kubebuilder create api --group examplecrd --version v1 --kind ExampleResource --namespaced=false

Customize the type in api/v1/exampleresource_types.go by removing the Foo field in the ExampleResourceSpec and add our color and size fields:

// ExampleResourceSpec defines the desired state of ExampleResource
type ExampleResourceSpec struct {
    Color string `json:"color"`
    Size  string `json:"size"`
}

In my case I didn’t need a status, so I removed the ExampleResourceStatus type.

Then run “make manifests”

Open the CRD (config/crd/bases/examplecrd.jasonhancock.com_exampleresources.yaml) and remove the subresources.

This generates (spews) a bunch of shit all over your project directory. I didn’t really want all that stuff (I’m sure it’s great if you need it, but I didn’t). So I created a new project directory and copied over just the bits and pieces I needed:

mkdir /tmp/examplecrd3
cd /tmp/examplecrd3
go mod init github.com/jasonhancock/examplecrd2
mkdir crds


cp ../examplecrd2/config/crd/bases/examplecrd.jasonhancock.com_exampleresources.yaml crds/
rsync -av ../examplecrd2/api/ api

You’ll notice I didn’t copy any clientSet code over. This is because kubebuilder doesn’t want you to use a generated clientSet, instead using the controller-runtime dynamic client.

Now let’s set up a main.go:

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	examplecrdv1 "github.com/jasonhancock/examplecrd2/api/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/types"
	"k8s.io/client-go/rest"
	"k8s.io/client-go/tools/clientcmd"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func main() {
	scheme := runtime.NewScheme()
	examplecrdv1.AddToScheme(scheme)

	var config *rest.Config
	var err error
	if kubeconfig := os.Getenv("KUBECONFIG"); kubeconfig != "" {
		config, err = clientcmd.BuildConfigFromFlags("", kubeconfig)
	} else {
		// we're running in-cluster. Try using the service account
		config, err = rest.InClusterConfig()
	}

	if err != nil {
		log.Fatal(fmt.Errorf("getting kubeconfig: %w", err))
	}

	controllerClient, err := client.New(config, client.Options{Scheme: scheme})
	if err != nil {
		log.Fatal(err)
	}

	cl := NewClient(controllerClient)
	ctx := context.Background()

	resp, err := cl.List(ctx, "")
	if err != nil {
		log.Fatal(err)
	}

	for _, v := range resp.Items {
		fmt.Printf("%s %s %s\n", v.ObjectMeta.Name, v.Spec.Color, v.Spec.Size)
	}

	getResp, err := cl.Get(ctx, "", "testing")
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%s %s %s\n", getResp.ObjectMeta.Name, getResp.Spec.Color, getResp.Spec.Size)
}

type Client struct {
	client client.Client
}

func NewClient(client client.Client) *Client {
	return &Client{client: client}
}

func (c *Client) List(ctx context.Context, namespace string) (*examplecrdv1.ExampleResourceList, error) {
	var list examplecrdv1.ExampleResourceList
	err := c.client.List(ctx, &list, &client.ListOptions{Namespace: namespace})
	return &list, err
}

func (c *Client) Get(ctx context.Context, namespace, name string) (*examplecrdv1.ExampleResource, error) {
	var obj examplecrdv1.ExampleResource
	err := c.client.Get(ctx, types.NamespacedName{Namespace: namespace, Name: name}, &obj)
	return &obj, err
}

You’ll notice I created a Client type. I may take a stab at making this a generic type, but that’s a subject for another article.

And let’s deploy a resource (example.yaml):

apiVersion: "examplecrd.jasonhancock.com/v1"
kind: "ExampleResource"
metadata:
  name: "testing"
spec:
  color: red
  size: large

kubectl apply -f example.yaml

Query the ExampleResources with kubectl:

$ kubectl get exampleresources
NAME      AGE
testing   17m

And let’s run our main.go:

$ export KUBECONFIG=~/.kube/config
$ go run main.go
testing red large
testing red large

Using kubebuilder was much better than using the kubernetes/code-generator method as discussed in the original article, however it still generated a lot of cruft and bootstrapping a project assumes you’re doing so from a greenfield, which in my case, I was not…the field was very brown and I had to take extra care not to stomp on existing files which is why I did the code generation in an isolated project directory and then copied over only the pieces I wanted/needed.

Overall I think kubebuilder is a better/smoother expierience, but I really wish there were more customization options to only generate specific things.

You can find the code for this example on Github in the jasonhancock/examplecrd2 repository.