Kubernetes lets you teach the API server about brand new object types and then write the logic that brings them to life. In this post, I’ll walk through creating a CRD (Custom Resource Definition) and the controller that drives it.

The starting point is the kubebuilder binary, which scaffolds a skeleton project for us. https://github.com/kubernetes-sigs/kubebuilder/releases

$:  kubebuilder init --domain blog.webischia.com                              
Writing scaffold for you to edit...
Get controller runtime:
$ go get sigs.k8s.io/controller-runtime@v0.5.0
Update go.mod:
$ go mod tidy
Running make:
$ make
/home/f/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go
kubrNext: define a resource with:
$ kubebuilder create api

This generates the necessary classes and pulls the required libraries into our go.mod file.

With the project scaffolded, the next thing to add is an API. In Kubernetes, resources have controllers behind them, just like Deployments do. https://kubernetes.io/docs/concepts/architecture/controller/

“A controller tracks at least one Kubernetes resource type. These objects have a spec field that represents the desired state. The controller(s) for that resource are responsible for making the current state come closer to that desired state.”

To make this concrete (and a little more fun), the resource I’m going to model is The Expanse’s spaceships, running right inside my Kubernetes cluster.

Creating the API

$ kubebuilder create api --group expanse --version v1beta1 --kind SpaceShips
Create Resource [y/n]
y
Create Controller [y/n]
y
Writing scaffold for you to edit...
api/v1beta1/spaceships_types.go
controllers/spaceships_controller.go
Running make:
$ make
/home/f/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
go fmt ./...
go vet ./...
go build -o bin/manager main.go

Once the API is scaffolded, we fill in our fields in the types class, and the make command takes care of generating the corresponding classes and YAML files.

apiVersion: expanse.blog.webischia.com/v1beta1
kind: SpaceShips
metadata:
  name: spaceships-sample
spec:
  name: MyShip
  class: Corvette
  owner: Earth

As the code above makes clear, our SpaceShips resource carries three fields: Name, Class, and Owner.

Installing in Kubernetes

For testing, I have Minikube running locally on Kubernetes v1.17.11, which is plenty to exercise both the resources and the controller logic.

With that ready, let’s install the CRDs.

$ make install
/home/f/go/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
kustomize build config/crd | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/spaceships.expanse.blog.webischia.com created

Listing all CRDs now turns up ours among them.

$ kubectl get crd |grep webischia
spaceships.expanse.blog.webischia.com         2020-10-31T11:04:34Z

And describing it echoes back exactly the fields we set in the types.go file.

$ kubectl describe crd spaceships.expanse.blog.webischia.com
Name:         spaceships.expanse.blog.webischia.com
Namespace:    
Labels:       <none>
Annotations:  controller-gen.kubebuilder.io/version: v0.2.5
API Version:  apiextensions.k8s.io/v1
Kind:         CustomResourceDefinition
Metadata:
  Creation Timestamp:  2020-10-31T11:04:34Z
..
Spec:
Description:  SpaceShipsSpec defines the desired state of SpaceShips
Properties:
  Class:
    Type:  string
  Name:
    Type:  string
  Owner:
    Type:  string

At this point we’ve defined our resources and installed the CustomResourceDefinition into a cluster. The missing half is the controller, and once it’s running we’ll see every event we wired up for our resource.

For permissions, kubebuilder sets these by default:

//+kubebuilder:rbac:groups=expanse.blog.webischia.com,resources=spaceships,verbs=get;list;watch;create;update;patch;delete
//+kubebuilder:rbac:groups=expanse.blog.webischia.com,resources=spaceships/status,verbs=get;update;patch

With that in place, let’s start our simple manager binary.

2020-10-31T14:10:11.180+0300    INFO    controller-runtime.metrics      metrics server is starting to listen    {"addr": ":8080"}
2020-10-31T14:10:11.180+0300    INFO    setup   starting manager
2020-10-31T14:10:11.181+0300    INFO    controller-runtime.manager      starting metrics server {"path": "/metrics"}
2020-10-31T14:10:11.181+0300    INFO    controller-runtime.controller   Starting EventSource    {"controller": "spaceships", "source": "kind source: /, Kind="}
2020-10-31T14:10:11.281+0300    INFO    controller-runtime.controller   Starting Controller     {"controller": "spaceships"}
2020-10-31T14:10:11.281+0300    INFO    controller-runtime.controller   Starting workers        {"controller": "spaceships", "worker count": 1}

Awesome, everything is OK. It connected to our cluster and started watching. Now to give it something to react to, let’s apply a sample YAML for our spaceships.

$ kubectl apply -f sample.yml 
spaceships.expanse.blog.webischia.com/spaceships-sample created

Because we added a log line to the controller, the object should show up there.

2020-10-31T14:10:52.004+0300    INFO    controllers.SpaceShips  Controller received : default/spaceships-sample
2020-10-31T14:10:52.004+0300    DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "spaceships", "request": "default/spaceships-sample"}

Awesome, and notice the new keyword that shows up: Reconcile. Because we returned no error, the object was reconciled successfully, and the controller stops watching it until some status or API operation stirs things up again.

And of course we can pull our object back out with a plain kubectl get.

$ kubectl get spaceships
NAME                AGE
spaceships-sample   7m20s

Statuses

A controller is only as useful as its grasp of the application’s state. We read that state (the status) and act on it according to our logic, so the next step is giving our resource a status to track. To do that, I added a subresource by following the kubebuilder book.

Here’s how the status field ended up looking for my SpaceShip resource.

const Created SpaceShipPhase = "Created"
const Active SpaceShipPhase = "Active"
const Passive SpaceShipPhase = "Passive"
// SpaceShipsStatus defines the observed state of SpaceShips
type SpaceShipsStatus struct {
	Phase SpaceShipPhase `json:"phase,omitempty"`
}
// +kubebuilder:subresource:status
type SpaceShips struct {
...
.
Status SpaceShipsStatus `json:"status,omitempty"`
}

Now that the controller can see the object’s current state, it needs to branch on it, so I added the switch block below.

switch spaceship.Status.Phase {
	case "":
		// newly created ones don't have any status, so it's empty
		return r.createSpaceship(spaceship)
	case expansev1beta1.Created:
		return r.startTheEngines(spaceship)
	}

One thing worth knowing here: when you create an object in Kubernetes, you don’t supply the status field yourself. The Kubernetes system fills it in.

The status describes the current state of the object, supplied and updated by the Kubernetes system and its components. The Kubernetes control plane continually and actively manages every object's actual state to match the desired state you supplied. Link to Kubernetes Docs

DynamoDB

To give the controller something tangible to do, I had it persist the YAML content to DynamoDB. That meant adding a small package wrapping a simple DynamoDB PutItem operation.

func (d *DBConfig) Write(ships v1beta1.SpaceShips) error {
	putItem := &dynamodb.PutItemInput{
		Item: map[string]*dynamodb.AttributeValue{
			"name": {
				S: aws.String(ships.Spec.Name),
			},
			"class": {
				S: aws.String(ships.Spec.Class),
			},
			"owner": {
				S: aws.String(ships.Spec.Owner),
			},
			"status": {
				S: aws.String(string(ships.Status.Phase)),
			},
		},
		TableName: aws.String(TABLE_NAME),
	}
	_, err := d.DynamoDB.PutItem(putItem)
	if err != nil {
		return err
	}
	return nil
}

Goal

Pulling it all together, the goal is to keep Expanse ship records in Kubernetes and DynamoDB. So let’s create our ship.

apiVersion: expanse.blog.webischia.com/v1beta1
kind: SpaceShips
metadata:
  name: spaceships-sample
spec:
  name: MyShip
  class: Corvette
  owner: Earth

Applying this fires the controller. Since the object has no status yet, the switch falls through to the empty case and calls r.createSpaceship(spaceship), which writes the initial DynamoDB entry and bumps the object’s status to CREATED.

That status update is itself a change, so the controller is triggered a second time. Now there’s a status to match, and it calls r.startTheEngines(spaceship), which updates the status field in DynamoDB and moves the CRD’s status to ACTIVE.

Kubernetes:

$ kubectl describe spaceships
Name:         spaceships-sample
Namespace:    default
Labels:       <none>
Annotations:  API Version:  expanse.blog.webischia.com/v1beta1
Kind:         SpaceShips
Metadata:
  Creation Timestamp:  2020-10-31T13:09:57Z
  Generation:          1
  Resource Version:    35155
  Self Link:           /apis/expanse.blog.webischia.com/v1beta1/namespaces/default/spaceships/spaceships-sample
  UID:                 591b4e89-31f0-4a63-82f4-6b284cfd148a
Spec:
  Class:  Corvette
  Name:   MyShip
  Owner:  Earth
Status:
  Phase:  Active
Events:   <none>

DynamoDB:

You can access the whole code here

Future Work

• Implement Delete/Stop Engines