Velero ¶

This package provides safe operations to backup, restore, perform disaster recovery, and migrate Tanzu Community Edition cluster resources and persistent volumes using the open source tool Velero.

Supported Providers ¶

The following table shows the providers this package can work with.

Components ¶

• Velero Deployment
• Restic DaemonSet

Custom Resources ¶

Each Velero operation – on-demand backup, scheduled backup, restore – is a custom resource, defined with a Kubernetes Custom Resource Definition (CRD) and stored in etcd. Each time these operations are run, an equivalent Kubernetes object is created and saved to storage.

Because of this Kubernetes native way that Velero operates, you are not restricted to backing up the entire etcd. You can back up or restore all objects in a Tanzu Community Edition cluster, or you can also filter what object to operate on by type, namespace, and/or label.

CLI ¶

Once the Velero package is installed, you will need to have the Velero CLI to run operations on the command line. Please see this documentation for how to install it: Velero CLI install.

Storage ¶

Velero needs an object storage where to save all resource backups and the information about snapshot backups.

Velero treats object storage as the source of truth. It continuously checks to see that the correct backup resources are always present. If there is a properly formatted backup file in the storage bucket, but no corresponding backup resource in the Kubernetes API, Velero synchronizes the information from object storage to Kubernetes. This allows restore functionality to work in a cluster migration scenario, where the original backup objects do not exist in the new cluster. Likewise, if a backup object exists in Kubernetes but not in object storage, it will be deleted from Kubernetes since the backup tarball no longer exists. To learn more, please see the documentation: How Velero works.

Server ¶

Velero runs on the cluster as a deployment alongside installed plugins that are specific to storage providers for backup and snapshot operations. It also includes controllers that process the custom resources to perform backups, restores, and all related operations.

Providers ¶

The Tanzu Community Edition Velero package provides support for these providers out of the box with minimal configuration.

Some other third-party storage providers, like MinIO, DigitalOcean, and others, support the same S3 API that the AWS Velero plugin uses. For more information please see: S3-Compatible object store providers for Velero.

Configuration ¶

The following configuration values can be set to customize the Velero installation for the different components.

Global ¶

Storage settings ¶

Global configurations for storage ¶

AWS storage ¶

Azure storage ¶

Volume snapshot settings ¶

Global configurations for snapshotting ¶

AWS volumes ¶

Azure volumes ¶

vSphere volumes ¶

vSphere specific ¶

MinIO ¶

Images ¶

Advanced ¶

Installation ¶

The Velero package can easily be installed to a cluster. However, for the package to function, you must configure settings for your cloud provider.

AWS Configuration ¶

To configure Velero for AWS, you will first need to setup your AWS account. Your account will need the following:

• An S3 bucket to store the backups
• IAM user with permissions to access EC2 and S3

There are multiple ways to configure your AWS account. Detailed instructions and other options can be found in the velero-plugin-for-aws setup instructions. This guide follows the first option in those instructions.

You will also need the AWS CLI to complete these steps. Follow the user guide to install it if needed.

1. Create an S3 bucket to store backups.

   export BUCKET=<< bucket name >>
   export REGION=<< region >>
   aws s3api create-bucket \
       --bucket ${BUCKET} \
       --region ${REGION}
   

2. Create an IAM user. This user will be explicitly for use with Velero, and given the appropriate permissions to perform backups.

   aws iam create-user --user-name velero
   

3. Create the policy that gives the necessary S3 and EC2 permissions.

   cat > velero-policy.json <<EOF
   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Effect": "Allow",
               "Action": [
                   "ec2:DescribeVolumes",
                   "ec2:DescribeSnapshots",
                   "ec2:CreateTags",
                   "ec2:CreateVolume",
                   "ec2:CreateSnapshot",
                   "ec2:DeleteSnapshot"
               ],
               "Resource": "*"
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:AbortMultipartUpload",
                   "s3:ListMultipartUploadParts"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}/*"
               ]
           },
           {
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::${BUCKET}"
               ]
           }
       ]
   }
   EOF
   

   Apply the policy to the velero IAM user.

   aws iam put-user-policy \
     --user-name velero \
     --policy-name velero \
     --policy-document file://velero-policy.json
   

4. Create an access key for the velero user.

   aws iam create-access-key --user-name velero
   

   The response will look like the following:

   {
       "AccessKey": {
           "UserName": "velero",
           "AccessKeyId": "AKIA4CRA12345EXAMPLE",
           "Status": "Active",
           "SecretAccessKey": "RbJSykwB1Z3c094BKgfJi2YzBCG12345EXAMPLE",
           "CreateDate": "2021-09-23T15:32:15+00:00"
       }
   }
   

   Be sure to save the AccessKeyId and SecretAccessKey values in a safe place. This is the only time that you will have access to these values.

5. Create a values.yaml file to configure the package to use AWS. Values that you will need to provide are:

   • Bucket name
   • Region
   • AWS Access Key
   • AWS Secret Access Key

   export AWS_ACCESS_KEY=<< AWS ACCESS KEY >>
   export AWS_SECRET_ACCESS_KEY=<< AWS SECRET ACCESS KEY >>
   
   cat > values.yaml <<EOF
   ---
   namespace: velero
   restic:
     create: false
   credential:
     useDefaultSecret: true
     name: cloud-credentials
     secretContents:
       cloud: |
         [default]
         aws_access_key_id=${AWS_ACCESS_KEY}
         aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}
   backupStorageLocation:
     name: backup
     spec:
       provider: aws
       default: true
       objectStorage:
         bucket: ${BUCKET}
         prefix: backup
       configAWS:
         region: ${REGION}
   volumeSnapshotLocation:
     snapshotsEnabled: true
     name: default
     spec:
       provider: aws
       configAWS:
         region: ${REGION}
   EOF
   

6. Install the Velero package.

   tanzu package install velero --package-name velero.community.tanzu.vmware.com --version 1.8.0 --values-file values.yaml
   

7. Verify that the Velero package was properly installed.

   tanzu package installed list
   | Retrieving installed packages...
     NAME    PACKAGE-NAME                       PACKAGE-VERSION  STATUS
     velero  velero.community.tanzu.vmware.com  1.8.0            Reconcile succeeded
   

vSphere configuration ¶

This is for using Velero in TCE vSphere environment, so first, need to create a TCE cluster.

For creating vSphere TCE cluster, please reference to document: Prepare to Deploy a Management Cluster to vSphere

Please be aware the vCenter IAAS infrastructure should be created before going through the guide.

For vSphere and vCenter guidance, please reference to: VMware vSphere Document

1. Due to vSphere environment doesn’t provide a S3 compatible object storage, the Carvel package provides a way to deploy a default MinIO for that job. Just set DEPLOY_MINIO to true and provide AWS_ACCESS_KEY and AWS_SECRET_ACCESS_KEY in the next step. Of course, you can also choose to use your own existing S3 compatible object storage.

2. Create a values.yaml file to configure the package to use vSphere. Values that you will need to provide are:

   • S3 storage provider’s Bucket name
   • S3 storage provider’s Region
   • S3 storage provider’s Access Key
   • S3 storage provider’s Secret Access Key
   • Namespace that vsphere is installed in the workload cluster
   • Workload cluster name. If your environment already set the vSphere workload cluster as the default cluster to kubectl, cluster name can be retrieved by this command: kubectl config current-context | awk -F@ '{ print $1 }'
   • vCenter server’s IP address
   • vCenter login user name
   • vCenter login password
   • Datacenter created in vCenter to deploy workload cluster
   • Public network used by vCenter
   • Whether to deploy a default MinIO as S3 provider. If deployed, the default MinIO will use provided S3 access key and secret key.

   export AWS_ACCESS_KEY=< S3 ACCESS KEY >
   export AWS_SECRET_ACCESS_KEY=< S3 SECRET ACCESS KEY >
   export BUCKET=< S3 BUCKET >
   export REGION=< S3 REGION >
   export VSPHERE_NAMESPACE=< VSPHERE NAMESPACE IN CLUSTER >
   export WORKLOAD_CLUSTER_NAME=`kubectl config current-context | awk -F@ '{ print $1 }'`
   export VC_SERVER=< VCENTER SERVER IP >
   export VC_USERNAME=< VCENTER USERNAME >
   export VC_PASSWORD=< VCENTER PASSWORD >
   export VC_DATACENTER=< VCENTER DATACENTER >
   export VC_PUBLIC_NETWORK=< VCENTER PUBLIC NETWORK >
   export DEPLOY_MINIO=< DEPLOY MINIO >
   
   cat > values.yaml <<EOF
   ---
   namespace: velero
   restic:
     create: false
   credential:
     useDefaultSecret: true
     name: cloud-credentials
     secretContents:
       cloud: |
         [default]
         aws_access_key_id=${AWS_ACCESS_KEY}
         aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}
   backupStorageLocation:
     name: backup
     spec:
       provider: aws
       default: true
       objectStorage:
         bucket: ${BUCKET}
       configAWS:
         region: ${REGION}
         s3ForcePathStyle: true
         s3Url: ${S3URL}
   volumeSnapshotLocation:
     snapshotsEnabled: true
     name: default
     spec:
       provider: vsphere
       configAWS:
         region: ${REGION}
         bucket: ${BUCKET}
   vsphere:
     create: true
     namespace: ${VSPHERE_NAMESPACE}
     clusterName: ${WORKLOAD_CLUSTER_NAME}
     server: ${VC_SERVER}
     username: ${VC_USERNAME}
     password: ${VC_PASSWORD}
     datacenter: ${VC_DATACENTER}
     publiceNetwork: ${$VC_PUBLIC_NETWORK}
   minio:
     deployDefaultMinio: ${DEPLOY_MINIO}
     namespace: ${AWS_ACCESS_KEY}
     accessKey: ${AWS_ACCESS_KEY}
     secretAccessKey: minio123
   EOF
   

3. Install the Velero package.

   tanzu package install velero --package-name velero.community.tanzu.vmware.com --version 1.8.0 --values-file values.yaml
   

4. Verify that the Velero package was properly installed.

   tanzu package installed list
   | Retrieving installed packages...
     NAME    PACKAGE-NAME                       PACKAGE-VERSION  STATUS
     velero  velero.community.tanzu.vmware.com  1.8.0            Reconcile succeeded
   

Usage Example ¶

This walkthrough guides you through an example disaster recovery scenario that leverages the Velero package. You must deploy the package before attempting this walkthrough.

⚠️ Note: For more advanced use cases and documentation, see the official Velero documentation.

In the following steps, you will simulate a disaster scenario. Specifically, you will deploy a stateless workload, create a backup, delete the workload, and restore it from the backup.

1. Download the Velero CLI from the GitHub releases page. The following steps assume you have installed Velero into your PATH.

2. Create a new namespace for this example:

   kubectl create ns velero-example
   

3. Deploy a sample workload into the new namespace:

   kubectl create deploy -n velero-example nginx --image=nginx
   

4. Verify the workload is up and running:

   kubectl get pods -n velero-example
   

   The output should be similar to the following:

   NAME                     READY   STATUS    RESTARTS   AGE
   nginx-6799fc88d8-mm47k   1/1     Running   0          7s
   

5. Create a backup of the velero-example namespace:

   velero create backup velero-example --include-namespaces velero-example
   

6. Verify the backup completed successfully:

   velero describe backup velero-example
   

   The output shows the “Phase” of the backup, which should be Completed.

7. Delete the velero-example namespace to simulate a disaster scenario:

   kubectl delete ns velero-example
   

8. Verify that the namespace has been deleted:

   kubectl get ns
   

9. Restore the namespace from the velero backup:

   velero create restore --from-backup velero-example
   

10. Validate that the velero-example namespace has been restored:

    kubectl get ns velero-example
    

11. Validate that the workload has been restored:

    kubectl get pods -n velero-example
    

Limitation ¶

Velero Carvel package is still not full fledged yet. The followings are features not included but supported by velero client:

• Multiple BackupStorageLocation provision.
• Specify adding which plugins to Velero.