Jason Hancock
Software Engineer
github facebook twitter

Building a k8s CRD Go (golang) Client

· by jsnby · Read in about 5 min · (859 Words)
kubernetes golang

UPDATE: I’ve redone this work with kubebuilder and documented that here. That process is much smoother.

This document is mainly intended for myself to reference in the future the next time I have to build a CRD. The code can be found in the jasonhancock/examplecrd repo on Github.

I was recently building something and decided that I was going to store some data as a Custom Resource inside the k8s cluster. It’s 2023 and it seems like this should be an easy task. My application that will be creating the custom resources is written in Go (golang), so I’ll need to be able to generate a Go client for my custom resource type. This turned out to be harder than it really should be. There were some other articles covering things, but they haven’t aged well:

The tooling also leaves a lot to be desired like not supporting Go modules/having to have your code inside the $GOPATH is just one example.

Let’s start by working through the CustomResourceDefinition. First, I needed to pick some unique name for the API group. This should be unique but informative. I’m going to select examplecrd.jasonhancock.com. Now we should decide on a Kind. I’m going to call it ExampleResource, which translates into a name of exampleresource.examplecrd.jasonhancock.com. You can see those in the CRD below.

We then need to decide how to scope our resources…should they be namespaced, or cluster-wide? I’m going to go with namespaced.

Now we decide what data we want on the resource. Keeping this example simple and generic, I’m going to say each resource has a color and a size, both of which are strings and both of which are required. Our CRD looks like this:

apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "exampleresources.examplecrd.jasonhancock.com"
spec:
  group: "examplecrd.jasonhancock.com"
  scope: "Namespaced"
  names:
    plural: "exampleresources"
    singular: "exampleresource"
    kind: "ExampleResource"
  versions:
  - name: v1
    served: true
    storage: true
    schema:
      openAPIV3Schema:
        type: object
        required:
        - spec
        properties:
          spec:
            type: object
            required:
            - color
            - size
            properties:
              color:
                type: string
              size:
                type: string

I’m going to set up a workspace to contain this work:

cd ~/development
mkdir examplecrd
cd examplecrd
go mod init github.com/jasonhancock/examplecrd

I’m going to write the CRD from above into a file crd.yaml, then apply it with kubectl:

kubectl apply -f crd.yaml

Let’s manually create an ExampleResource with kubectl. example.yaml:

apiVersion: "examplecrd.jasonhancock.com/v1"
kind: "ExampleResource"
metadata:
  name: "testing"
spec:
  color: red
  size: large

kubectl apply -f example.yaml

And we can list our resource:

$ kubectl get exampleresources
NAME      AGE
testing   39s

$ kubectl get exampleresource testing -o yaml
apiVersion: examplecrd.jasonhancock.com/v1
kind: ExampleResource
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"examplecrd.jasonhancock.com/v1","kind":"ExampleResource","metadata":{"annotations":{},"name":"testing","namespace":"default"},"spec":{"color":"red","size":"large"}}
  creationTimestamp: "2023-07-26T22:11:39Z"
  generation: 1
  name: testing
  namespace: default
  resourceVersion: "6950"
  uid: be066c5d-7cd2-4a2e-916e-3b6f17f37879
spec:
  color: red
  size: large

So that’s great. But now I want to do the same thing from some Go code, so let’s get started generating our client. Let’s create a types.go file in the path from the root of our repo of pkg/apis/examplecrd.jasonhancock.com/v1/types.go:

package v1

import metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"

//go:generate controller-gen object paths=$GOFILE

// ExampleResourceSpec is the spec of the ExampleResource object.
// +kubebuilder:object:generate=true
type ExampleResourceSpec struct {
	Color string `json:"color"`
	Size  string `json:"size"`
}

// +genclient

// ExampleResource is our example resource.
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ExampleResource struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec ExampleResourceSpec `json:"spec"`
}

// ExampleResourceList is a list of ExampleResource objects.
// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
type ExampleResourceList struct {
	metav1.TypeMeta `json:",inline"`
	metav1.ListMeta `json:"metadata,omitempty"`

	Items []ExampleResource `json:"items"`
}

Create a doc.go in that same directory:

// +k8s:deepcopy-gen=package
// +k8s:defaulter-gen=TypeMeta
// +groupName=examplecrd.jasonhancock.com

package v1

And a register.go:

package v1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
	"k8s.io/apimachinery/pkg/runtime"
	"k8s.io/apimachinery/pkg/runtime/schema"
)

// SchemeGroupVersion defines the group info of the API.
var SchemeGroupVersion = schema.GroupVersion{
	Group:   "examplecrd.jasonhancock.com",
	Version: "v1",
}

var (
	schemeBuilder      runtime.SchemeBuilder
	localSchemeBuilder = &schemeBuilder

	// AddToScheme adds to the scheme.
	AddToScheme = localSchemeBuilder.AddToScheme
)

func init() {
	// We only register manually written functions here. The registration of the
	// generated functions takes place in the generated files. The separation
	// makes the code compile even when the generated files are missing.
	localSchemeBuilder.Register(addKnownTypes)
}

// Resource takes an unqualified resource and returns a Group qualified GroupResource
func Resource(resource string) schema.GroupResource {
	return SchemeGroupVersion.WithResource(resource).GroupResource()
}

// Adds the list of known types to the given scheme.
func addKnownTypes(scheme *runtime.Scheme) error {
	scheme.AddKnownTypes(
		SchemeGroupVersion,
		&ExampleResource{},
		&ExampleResourceList{},
	)

	scheme.AddKnownTypes(
		SchemeGroupVersion,
		&metav1.Status{},
	)

	metav1.AddToGroupVersion(
		scheme,
		SchemeGroupVersion,
	)

	return nil
}

Let’s update go.mod by running a tidy:

go mod tidy

Let’s set up some tooling….

cd /tmp
git clone git@github.com:kubernetes/code-generator.git

Configure our shell:

export CODEGEN_PKG=/tmp/code-generator

Fake the tooling to think we’re in a $GOPATH:

mkdir /tmp/go/src/github.com/jasonhancock
cd /tmp/go/src/github.com/jasonhancock
ln -s ~/development/examplecrd
cd /tmp/go/src/github.com/jasonhancock/examplecrd
export GOPATH=/tmp/go
./hack/update_codegen.sh

The hack/update_codegen.sh can be found in the examplecrd repo. I had some trouble getting the binaries from the code-generator to build.

Generate the deepcopy stuff:

cd ~/development/examplecrd/pkg/apis/examplecrd.jasonhancock.com/v1
go generate

unset GOPATH
go mod tidy

You should then be able to build/run the main.go:

export KUBECONFIG=~/.kube/config
cd ~/development/examplecrd
go run main.go
testing red large

This entire process seems like it’s harder than it needs to be. It would be great if the k8s maintainers would at least drop the $GOPATH requirement and work with Go modules as that would likely make things a lot simpler/user friendly.