Backup Kubernetes Object to AWS S3 using Heptio Velero(Ark)
Thu, May 30, 2019
6-minute read
In this post i’ll show how to backup Kubernetes objects to AWS S3 and restoring this backup to another cluster or same cluster. This allows us to copy,move,restore our objects.
First of we need Velero in our system to do that download Velero binary (or we can make).
$ wget https://github.com/heptio/velero/releases/download/v1.0.0/velero-v1.0.0-linux-amd64.tar.gz && tar -xvf velero-v1.0.0-linux-amd64.tar.gz
After that we have binary we need to install velero server to our cluster.
BTW: we dont need any cluster inside AWS. This credentials for backup objects. For this blog post i created DigitalOcean Kubernetes.
./velero install --provider aws --bucket webischia-velero-backup --secret-file ./credentials-velero --backup-location-config region=eu-central-1 --snapshot-location-config region=eu-central-1 CustomResourceDefinition/podvolumebackups.velero.io: attempting to create resource CustomResourceDefinition/podvolumebackups.velero.io: created CustomResourceDefinition/backupstoragelocations.velero.io: attempting to create resource CustomResourceDefinition/backupstoragelocations.velero.io: created CustomResourceDefinition/volumesnapshotlocations.velero.io: attempting to create resource CustomResourceDefinition/volumesnapshotlocations.velero.io: created CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: created CustomResourceDefinition/schedules.velero.io: attempting to create resource CustomResourceDefinition/schedules.velero.io: created CustomResourceDefinition/downloadrequests.velero.io: attempting to create resource CustomResourceDefinition/downloadrequests.velero.io: created CustomResourceDefinition/deletebackuprequests.velero.io: attempting to create resource CustomResourceDefinition/deletebackuprequests.velero.io: created CustomResourceDefinition/podvolumerestores.velero.io: attempting to create resource CustomResourceDefinition/podvolumerestores.velero.io: created CustomResourceDefinition/resticrepositories.velero.io: attempting to create resource CustomResourceDefinition/resticrepositories.velero.io: created CustomResourceDefinition/serverstatusrequests.velero.io: attempting to create resource CustomResourceDefinition/serverstatusrequests.velero.io: created CustomResourceDefinition/restores.velero.io: attempting to create resource CustomResourceDefinition/restores.velero.io: created Waiting for resources to be ready in cluster... Namespace/velero: attempting to create resource Namespace/velero: created ClusterRoleBinding/velero: attempting to create resource ClusterRoleBinding/velero: created ServiceAccount/velero: attempting to create resource ServiceAccount/velero: created Secret/cloud-credentials: attempting to create resource Secret/cloud-credentials: created BackupStorageLocation/default: attempting to create resource BackupStorageLocation/default: created VolumeSnapshotLocation/default: attempting to create resource VolumeSnapshotLocation/default: created Deployment/velero: attempting to create resource Deployment/velero: created Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
Velero is running yay!. Lets backup some objects. Every velero tutorial is using nginx backup and hey lets run nginx.
Lets see our backup location.
kubectl describe backupstoragelocation -n velero
Name: default
Namespace: velero
Labels: component=velero
Annotations: <none>
API Version: velero.io/v1
Kind: BackupStorageLocation
Metadata:
Creation Timestamp: 2019-05-30T10:59:29Z
Generation: 9
Resource Version: 1849
Self Link: /apis/velero.io/v1/namespaces/velero/backupstoragelocations/default
UID: ff74d742-82c9-11e9-a801-16e16d5f9ddf
Spec:
Config:
Region: eu-central-1
Object Storage:
Bucket: webischia-velero-backup
Prefix:
Provider: aws
Status:
Last Synced Revision:
Last Synced Time: 2019-05-30T11:06:40.568956943Z
Events: <none>
Velero github repo has example nginx yaml. Lets apply this. This yaml will create namespace,deployment and loadbalancer service for nginx. You may having problem with nginx:1.7.9 lets change it to latest for deployment.
kubectl get po -n nginx-example
NAME READY STATUS RESTARTS AGE
nginx-deployment-5f88c697f-mqlzb 1/1 Running 0 2m6s
Now we can create backup. For creating backup we are going to use velero binary
./velero backup create nginx-backup --include-namespaces nginx-example
Backup request "nginx-backup" submitted successfully.
Run `velero backup describe nginx-backup` or `velero backup logs nginx-backup` for more details.
./velero backup describe nginx-backup
Name: nginx-backup
Namespace: velero
Labels: velero.io/storage-location=default
Annotations: <none>
Phase: Completed
Namespaces:
Included: nginx-example
Excluded: <none>
Resources:
Included: *
Excluded: <none>
Cluster-scoped: auto
Label selector: <none>
Storage Location: default
Snapshot PVs: auto
TTL: 720h0m0s
Hooks: <none>
Backup Format Version: 1
Started: 2019-05-30 14:19:08 +0300 +03
Completed: 2019-05-30 14:19:09 +0300 +03
Expiration: 2019-06-29 14:19:08 +0300 +03
Persistent Volumes: <none included>
As we can see our backup is completed. Lets check out logs
./velero backup logs nginx-backup
time="2019-05-30T11:19:08Z" level=info msg="Setting up backup temp file" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:454"
time="2019-05-30T11:19:08Z" level=info msg="Setting up plugin manager" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:461"
time="2019-05-30T11:19:08Z" level=info msg="Getting backup item actions" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:465"
time="2019-05-30T11:19:08Z" level=info msg="Setting up backup store" backup=velero/nginx-backup logSource="pkg/controller/backup_controller.go:471"
time="2019-05-30T11:19:08Z" level=info msg="Writing backup version file" backup=velero/nginx-backup logSource="pkg/backup/backup.go:219"
time="2019-05-30T11:19:08Z" level=info msg="Including namespaces: nginx-example" backup=velero/nginx-backup logSource="pkg/backup/backup.go:225"
time="2019-05-30T11:19:08Z" level=info msg="Excluding namespaces: <none>" backup=velero/nginx-backup logSource="pkg/backup/backup.go:226"
time="2019-05-30T11:19:08Z" level=info msg="Including resources: *" backup=velero/nginx-backup logSource="pkg/backup/backup.go:229"
time="2019-05-30T11:19:08Z" level=info msg="Excluding resources: <none>" backup=velero/nginx-backup logSource="pkg/backup/backup.go:230"
time="2019-05-30T11:19:08Z" level=info msg="Backing up group" backup=velero/nginx-backup group=v1 logSource="pkg/backup/group_backupper.go:105"
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Executing custom action" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:307" name=nginx-deployment-5f88c697f-mqlzb namespace=nginx-example resource=pods
time="2019-05-30T11:19:08Z" level=info msg="Executing podAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/pod_action.go:51" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Done executing podAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/pod_action.go:77" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=persistentvolumeclaims
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=persistentvolumes
time="2019-05-30T11:19:08Z" level=info msg="Skipping resource because it's cluster-scoped and only specific namespaces are included in the backup" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:130" resource=persistentvolumes
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=podtemplates
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=limitranges
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=configmaps
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 0 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=resourcequotas
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=default namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Executing custom action" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:307" name=default namespace=nginx-example resource=serviceaccounts
time="2019-05-30T11:19:08Z" level=info msg="Running ServiceAccountAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/service_account_action.go:77" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Done running ServiceAccountAction" backup=velero/nginx-backup cmd=/velero logSource="pkg/backup/service_account_action.go:120" pluginName=velero
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 1 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx namespace=nginx-example resource=endpoints
time="2019-05-30T11:19:08Z" level=info msg="Backing up resource" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:108" resource=events
time="2019-05-30T11:19:08Z" level=info msg="Listing items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:229" namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Retrieved 41 items" backup=velero/nginx-backup group=v1 logSource="pkg/backup/resource_backupper.go:243" namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a37163dc86c899 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a37163ed686120 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=my-nginx.15a3718817cb8fa9 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a37181aab8a5c9 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a3718214fdddc3 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a3718387b659de namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a371838d8c4d0c namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-mqlzb.15a37183a46803bf namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37183f0d4ea15 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a371846897ae3d namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185c3afab98 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185d4615367 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a37185e51af034 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f-xxfff.15a371a8f6c8acd1 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a37181a8a7b0b4 namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a37183f03d1b0b namespace=nginx-example resource=events
time="2019-05-30T11:19:08Z" level=info msg="Backing up item" backup=velero/nginx-backup group=v1 logSource="pkg/backup/item_backupper.go:163" name=nginx-deployment-5f88c697f.15a371a8f6c51984 namespace=nginx-example resource=events
Nice. Lets look our s3 bucket to see whats inside.
aws s3 ls webischia-velero-backup --recursive
2019-05-30 14:19:10 2817 backups/nginx-backup/nginx-backup-logs.gz
2019-05-30 14:19:10 29 backups/nginx-backup/nginx-backup-volumesnapshots.json.gz
2019-05-30 14:19:10 8722 backups/nginx-backup/nginx-backup.tar.gz
2019-05-30 14:19:10 1154 backups/nginx-backup/velero-backup.json
2019-05-30 14:19:10 36 metadata/revision
These files contains our deployment,service,namespace and our app logs json.
$kubectl delete namespace nginx-example
namespace “nginx-example” deleted
We already have backup so lets restore it. Also we can select different namespace to restore this.
./velero restore create --from-backup nginx-backup --namespace-mappings nginx-example:fahri
Restore request "nginx-backup-20190530142623" submitted successfully.
Run `velero restore describe nginx-backup-20190530142623` or `velero restore logs nginx-backup-20190530142623` for more details.
./velero restore describe nginx-backup-20190530142623
Name: nginx-backup-20190530142623
Namespace: velero
Labels:
Annotations:
Phase: Completed
kubectl get po -n fahri
NAME READY STATUS RESTARTS AGE
nginx-deployment-5f88c697f-mqlzb 1/1 Running 0 73s
As you can see restore completed.
If we will use persistent volumes this volume snapshot also will be restored. Have you noticed deployment pod names are same with backup.