Backing up a Kubernetes cluster isn’t just about disaster recovery, it also gives you a clean way to clone or migrate workloads between clusters. In this post, I’ll show how to back up Kubernetes objects to AWS S3 and then restore that backup, whether to a different cluster or the same one. In other words, we get a simple way to copy, move, and restore our objects.

The tool that makes this possible is Velero, so that’s the first thing we need in place. The quickest path is to download the Velero binary, though you can always build it yourself.

$ wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz && tar -xvf velero-v1.0.0-linux-amd64.tar.gz

Once we have the binary, the next step is installing the Velero server into our cluster.

One thing worth clearing up first: the cluster itself doesn’t need to live in AWS. The AWS credentials here are only for storing the backed-up objects in S3. To prove the point, the cluster for this blog post is actually a DigitalOcean Kubernetes cluster.

./velero install     --provider aws     --bucket webischia-velero-backup     --secret-file ./credentials-velero     --backup-location-config region=eu-central-1     --snapshot-location-config region=eu-central-1
CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource
CustomResourceDefinition/podvolumebackups.velero.io: created
CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource
CustomResourceDefinition/backupstoragelocations.velero.io: created
CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource
CustomResourceDefinition/volumesnapshotlocations.velero.io: created
CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: created
CustomResourceDefinition/schedules.velero.io: attempting to create resource
CustomResourceDefinition/schedules.velero.io: created
CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource
CustomResourceDefinition/downloadrequests.velero.io: created
CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource
CustomResourceDefinition/deletebackuprequests.velero.io: created
CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource
CustomResourceDefinition/podvolumerestores.velero.io: created
CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource
CustomResourceDefinition/resticrepositories.velero.io: created
CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource
CustomResourceDefinition/serverstatusrequests.velero.io: created
CustomResourceDefinition/restores.velero.io: attempting to create resource
CustomResourceDefinition/restores.velero.io: created
Waiting for resources to be ready in cluster...
Namespace/velero: attempting to create resource
Namespace/velero: created
ClusterRoleBinding/velero: attempting to create resource
ClusterRoleBinding/velero: created
ServiceAccount/velero: attempting to create resource
ServiceAccount/velero: created
Secret/cloud-credentials: attempting to create resource
Secret/cloud-credentials: created
BackupStorageLocation/default: attempting to create resource
BackupStorageLocation/default: created
VolumeSnapshotLocation/default: attempting to create resource
VolumeSnapshotLocation/default: created
Deployment/velero: attempting to create resource
Deployment/velero: created
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

Velero is running, yay! Now we can actually back up some objects. Every Velero tutorial reaches for an nginx backup, so let’s keep with tradition and run nginx too. First, though, let’s confirm where our backups will land.

 kubectl describe backupstoragelocation -n velero
Name:         default
Namespace:    velero
Labels:       component=velero
Annotations:  <none>
API Version:  velero.io/v1
Kind:         BackupStorageLocation
Metadata:
  Creation Timestamp:  2019-05-30T10:59:29Z
  Generation:          9
  Resource Version:    1849
  Self Link:           /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
  UID:                 ff74d742-82c9-11e9-a801-16e16d5f9ddf
Spec:
  Config:
    Region:  eu-central-1
  Object Storage:
    Bucket:  webischia-velero-backup
    Prefix:  
  Provider:  aws
Status:
  Last Synced Revision:  
  Last Synced Time:      2019-05-30T11:06:40.568956943Z
Events:                  <none>

The Velero GitHub repo ships an example nginx YAML, so let’s apply it. It sets up a namespace, a deployment, and a load balancer service for nginx in one go. The pinned nginx:1.7.9 image can give you trouble, so it’s worth bumping the deployment to the latest version.

kubectl get po -n nginx-example
NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-5f88c697f-mqlzb   1/1     Running   0          2m6s

With nginx running, we can create our backup, and for that we lean on the Velero binary again.

./velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.

./velero backup describe nginx-backup
Name:         nginx-backup
Namespace:    velero
Labels:       velero.io/storage-location=default
Annotations:  <none>

Phase:  Completed

Namespaces:
  Included:  nginx-example
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        <none>
  Cluster-scoped:  auto

Label selector:  <none>

Storage Location:  default

Snapshot PVs:  auto

TTL:  720h0m0s

Hooks:  <none>

Backup Format Version:  1

Started:    2019-05-30 14:19:08 +0300 +03
Completed:  2019-05-30 14:19:09 +0300 +03

Expiration:  2019-06-29 14:19:08 +0300 +03

Persistent Volumes: <none included>

The phase reads Completed, so the backup went through. To see exactly what it did, let’s check the logs.

./velero backup logs nginx-backup
time="2019-05-30T11:19:08Z" level=info msg="Setting up backup temp file" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:454"
time="2019-05-30T11:19:08Z" level=info msg="Setting up plugin manager" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:461"
time="2019-05-30T11:19:08Z" level=info msg="Getting backup item actions" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:465"
time="2019-05-30T11:19:08Z" level=info msg="Setting up backup store" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:471"
time="2019-05-30T11:19:08Z" level=info msg="Writing backup version file" backup=velero/nginx-backup logSource="pkg/backup/backup.go:219"
time="2019-05-30T11:19:08Z" level=info msg="Including namespaces: nginx-example" backup=velero/nginx-backup logSource="pkg/backup/backup.go:225"
time="2019-05-30T11:19:08Z" level=info msg="Excluding namespaces: <none>" backup=velero/nginx-backup logSource="pkg/backup/backup.go:226"
time="2019-05-30T11:19:08Z" level=info msg="Including resources: *" backup=velero/nginx-backup logSource="pkg/backup/backup.go:229"
time="2019-05-30T11:19:08Z" level=info msg="Excluding resources: <none>" backup=velero/nginx-backup logSource="pkg/backup/backup.go:230"
time="2019-05-30T11:19:08Z" level=info msg="Backing up group" backup=velero/nginx-backup group=v1 logSource="pkg/backup/group_backupper.go:105"
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Executing custom action" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:307" name=nginx-deployment-5f88c697f-mqlzb namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Executing podAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Done executing podAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=persistentvolumes
time="2019-05-30T11:19:08Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:130" resource=persistentvolumes
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=default namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Executing custom action" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:307" name=default namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Running ServiceAccountAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/service_account_action.go:77" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Done running ServiceAccountAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/service_account_action.go:120" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=events
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 41 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a37163dc86c899 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a37163ed686120 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a3718817cb8fa9 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a37181aab8a5c9 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a3718214fdddc3 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a3718387b659de namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a371838d8c4d0c namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a37183a46803bf namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37183f0d4ea15 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a371846897ae3d namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185c3afab98 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185d4615367 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185e51af034 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a371a8f6c8acd1 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a37181a8a7b0b4 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a37183f03d1b0b namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a371a8f6c51984 namespace=nginx-example resource=events

Nice. Now let’s look inside our S3 bucket to see what actually landed there.

aws s3 ls webischia-velero-backup --recursive
2019-05-30 14:19:10       2817 backups/nginx-backup/nginx-backup-logs.gz
2019-05-30 14:19:10         29 backups/nginx-backup/nginx-backup-volumesnapshots.json.gz
2019-05-30 14:19:10       8722 backups/nginx-backup/nginx-backup.tar.gz
2019-05-30 14:19:10       1154 backups/nginx-backup/velero-backup.json
2019-05-30 14:19:10         36 metadata/revision

Between them, these files hold our deployment, service, namespace, and the app logs JSON. To make the restore meaningful, let’s wipe the namespace first.

$kubectl delete namespace nginx-example
namespace "nginx-example" deleted

With the backup safely in S3, restoring it is straightforward. We can even point it at a different namespace on the way back in.

./velero restore create --from-backup nginx-backup --namespace-mappings nginx-example:fahri
Restore request "nginx-backup-20190530142623" submitted successfully.
Run `velero restore describe nginx-backup-20190530142623` or `velero restore logs nginx-backup-20190530142623` for more details.

./velero restore describe nginx-backup-20190530142623
Name:         nginx-backup-20190530142623
Namespace:    velero
Labels:       
Annotations:  
Phase:  Completed

kubectl get po -n fahri
NAME                               READY   STATUS    RESTARTS   AGE
nginx-deployment-5f88c697f-mqlzb   1/1     Running   0          73s

And there it is, the restore completed cleanly. If we had been using persistent volumes, the volume snapshot would have come back along with it. And notice the detail that ties it all together: the deployment pod names match exactly what we had in the backup.