Tanzu Community Edition

Documentation

Velero

This package provides safe operations to backup, restore, perform disaster recovery, and migrate Tanzu Community Edition cluster resources and persistent volumes using the open source tool Velero.

Supported Providers

The following table shows the providers this package can work with.

AWSAzurevSphereDocker
🚫

Components

  • Velero Deployment
  • Restic DaemonSet

Custom Resources

Each Velero operation – on-demand backup, scheduled backup, restore – is a custom resource, defined with a Kubernetes Custom Resource Definition (CRD) and stored in etcd. Each time these operations are run, an equivalent Kubernetes object is created and saved to storage.

Because of this Kubernetes native way that Velero operates, you are not restricted to backing up the entire etcd. You can back up or restore all objects in a Tanzu Community Edition cluster, or you can also filter what object to operate on by type, namespace, and/or label.

CLI

Once the Velero package is installed, you will need to have the Velero CLI to run operations on the command line. Please see this documentation for how to install it: Velero CLI install.

Storage

Velero needs an object storage where to save all resource backups and the information about snapshot backups.

Velero treats object storage as the source of truth. It continuously checks to see that the correct backup resources are always present. If there is a properly formatted backup file in the storage bucket, but no corresponding backup resource in the Kubernetes API, Velero synchronizes the information from object storage to Kubernetes. This allows restore functionality to work in a cluster migration scenario, where the original backup objects do not exist in the new cluster. Likewise, if a backup object exists in Kubernetes but not in object storage, it will be deleted from Kubernetes since the backup tarball no longer exists. To learn more, please see the documentation: How Velero works.

Server

Velero runs on the cluster as a deployment alongside installed plugins that are specific to storage providers for backup and snapshot operations. It also includes controllers that process the custom resources to perform backups, restores, and all related operations.

Providers

The Tanzu Community Edition Velero package provides support for these providers out of the box with minimal configuration.

ProviderObject StoreVolume SnapshotterPlugin Provider RepoSetup Instructions
Amazon Web Services (AWS)AWS S3AWS EBSVelero plugin for AWSAWS Plugin Setup
Microsoft AzureAzure Blob StorageAzure Managed DisksVelero plugin for Microsoft AzureAzure Plugin Setup
VMware vSphereN/A🚫 vSphere Volumes (on the roadmap)VMware vSpherevSphere Plugin Setup

Some other third-party storage providers, like MinIO, DigitalOcean, and others, support the same S3 API that the AWS Velero plugin uses. For more information please see: S3-Compatible object store providers for Velero.

Configuration

The following configuration values can be set to customize the Velero installation for the different components.

Global

ValueRequired/OptionalDescription
namespaceOptionalThe namespace in which to deploy Velero.

Storage settings

Global configurations for storage

ValueRequired/OptionalDescription
backupStorageLocation.nameRequiredThe name of the Backup Storage Location.
backupStorageLocation.providerRequiredThe cloud provider to use. One of: aws, azure.
backupStorageLocation.defaultOptionalIndicates if this location is the default backup storage location.
backupStorageLocation.objectStorage.bucketRequiredThe storage bucket where backups are to be uploaded.
backupStorageLocation.objectStorage.prefixOptionalThe directory inside a storage bucket where backups are to be uploaded.

AWS storage

ValueRequired/OptionalDescription
backupStorageLocation.configAWS.regionRequiredThe AWS region where the S3 bucket is located.

Azure storage

ValueRequired/OptionalDescription
backupStorageLocation.configAzure.resourceGroupRequiredThe name of the resource group containing the storage account for this backup storage location.
backupStorageLocation.configAzure.storageAccountRequiredThe name of the storage account for this backup storage location.
backupStorageLocation.configAzure.storageAccountKeyEnvVarRequiredRequired if using a storage account access key to authenticate rather than a service principal.
backupStorageLocation.configAzure.subscriptionIdOptionalThe the ID of the subscription for this backup storage location.

Volume snapshot settings

Global configurations for snapshotting

ValueRequired/OptionalDescription
volumeSnapshotLocation.snapshotsEnabledRequiredIndicates whether to create a volumesnapshotlocation CR. If false => disable the snapshot feature.
volumeSnapshotLocation.nameRequiredThe name of the volume snapshot location where snapshots are being taken.
volumeSnapshotLocation.spec.providerRequiredThe name for the volume snapshot provider. Required if snapshots are enabled. Valid values are aws and azure

AWS volumes

ValueRequired/OptionalDescription
volumeSnapshotLocation.spec.configAWS.regionRequiredThe AWS region where the volumes/snapshots are located
volumeSnapshotLocation.spec.configAWS.profileOptionalThe AWS profile within the credentials file to use for the volume snapshot location. Default is “default”.

Azure volumes

ValueRequired/OptionalDescription
volumeSnapshotLocation.spec.configAzure.apiTimeoutOptionalIndicates how long to wait for an Azure API request to complete before timeout. Defaults to 2m0s.
volumeSnapshotLocation.spec.configAzure.resourceGroupOptionalThe name of the resource group where volume snapshots should be stored, if different from the cluster’s resource group.
volumeSnapshotLocation.spec.configAzure.subscriptionIdOptionalThe ID of the subscription where volume snapshots should be stored, if different from the cluster’s subscription. Requires “resourceGroup” to also be set.
volumeSnapshotLocation.spec.configAzure.incrementalOptionalAzure offers the option to take full or incremental snapshots of managed disks. Set this parameter to true, to take incremental snapshots. If the parameter is omitted or set to false, full snapshots are taken (default).

Advanced

ValueRequired/OptionalDescription
rbac.createOptionalWhether to create the Velero Role and RoleBinding to give all permissions to the namespace to Velero.
rbac.nameOptionalA new name for the cluster RolBinding. Default is velero.
rbac.clusterAdministratorOptionalWhether to create the ClusterRoleBinding to give administrator permissions to Velero. rbac.create must also be set to true.
rbac.roleRefNameOptionalName of the cluster role to reference. Default is cluster-admin.
rbac.clusterRoleAPIGroupsOptionalThe name of the API groups that contain the resources for the cluster role.
rbac.clusterRoleVerbsOptionalThe set of verbs that apply to the secret resources contained in this rule.
serviceAccount.nameOptionalThe name of the ServiceAccount the RoleBinding should reference.
serviceAccount.annotationsOptionalAnnotations for the ServiceAccount the RoleBinding should reference.
serviceAccount.labelsOptionalLabels for the ServiceAccount the RoleBinding should reference.
restic.createOptionalWhether to deploy the restic daemonset.
restic.defaultVolumesToResticOptionalBool flag to configure Velero server to use restic by default to backup all pod volumes on all backups.
restic.defaultResticPruneFrequencyOptionalHow often ‘restic prune’ is run for restic repositories by default.
restic.cpuLimitOptionalCPU limit for restic pod. A value of “0” is treated as unbounded. (default “1000m”).
restic.cpuRequestOptionalCPU request for restic pod. A value of “0” is treated as unbounded. (default “500m”).
restic.memoryLimitOptionalMemory limit for restic pod. A value of “0” is treated as unbounded. (default “1Gi”).
restic.memoryRequestOptionalMemory request for restic pod. A value of “0” is treated as unbounded. (default “512Mi”).

Installation

The Velero package can easily be installed to a cluster. However, for the package to function, you must configure settings for your cloud provider.

AWS Configuration

To configure Velero for AWS, you will first need to setup your AWS account. Your account will need the following:

  • An S3 bucket to store the backups
  • IAM user with permissions to access EC2 and S3

There are multiple ways to configure your AWS account. Detailed instructions and other options can be found in the velero-plugin-for-aws setup instructions. This guide follows the first option in those instructions.

You will also need the AWS CLI to complete these steps. Follow the user guide to install it if needed.

  1. Create an S3 bucket to store backups.

    export BUCKET=<< bucket name >>
    export REGION=<< region >>
    aws s3api create-bucket \
        --bucket ${BUCKET} \
        --region ${REGION}
    
  2. Create an IAM user. This user will be explicitly for use with Velero, and given the appropriate permissions to perform backups.

    aws iam create-user --user-name velero
    
  3. Create the policy that gives the necessary S3 and EC2 permissions.

    cat > velero-policy.json <<EOF
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Action": [
                    "ec2:DescribeVolumes",
                    "ec2:DescribeSnapshots",
                    "ec2:CreateTags",
                    "ec2:CreateVolume",
                    "ec2:CreateSnapshot",
                    "ec2:DeleteSnapshot"
                ],
                "Resource": "*"
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:GetObject",
                    "s3:DeleteObject",
                    "s3:PutObject",
                    "s3:AbortMultipartUpload",
                    "s3:ListMultipartUploadParts"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}/*"
                ]
            },
            {
                "Effect": "Allow",
                "Action": [
                    "s3:ListBucket"
                ],
                "Resource": [
                    "arn:aws:s3:::${BUCKET}"
                ]
            }
        ]
    }
    EOF
    

    Apply the policy to the velero IAM user.

    aws iam put-user-policy \
      --user-name velero \
      --policy-name velero \
      --policy-document file://velero-policy.json
    
  4. Create an access key for the velero user.

    aws iam create-access-key --user-name velero
    

    The response will look like the following:

    {
        "AccessKey": {
            "UserName": "velero",
            "AccessKeyId": "AKIA4CRA12345EXAMPLE",
            "Status": "Active",
            "SecretAccessKey": "RbJSykwB1Z3c094BKgfJi2YzBCG12345EXAMPLE",
            "CreateDate": "2021-09-23T15:32:15+00:00"
        }
    }
    

    Be sure to save the AccessKeyId and SecretAccessKey values in a safe place. This is the only time that you will have access to these values.

  5. Create a values.yaml file to configure the package to use AWS. Values that you will need to provide are:

    • Bucket name
    • Region
    • AWS Access Key
    • AWS Secret Access Key
    export AWS_ACCESS_KEY=<< AWS ACCESS KEY >>
    export AWS_SECRET_ACCESS_KEY=<< AWS SECRET ACCESS KEY >>
    
    cat > values.yaml <<EOF
    ---
    namespace: velero
    restic:
      create: false
    credential:
      useDefaultSecret: true
      name: cloud-credentials
      secretContents:
        cloud: |
          [default]
          aws_access_key_id=${AWS_ACCESS_KEY}
          aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}
    backupStorageLocation:
      name: backup
      spec:
        provider: aws
        default: true
        objectStorage:
          bucket: ${BUCKET}
          prefix: backup
        configAWS:
          region: ${REGION}
    volumeSnapshotLocation:
      snapshotsEnabled: true
      name: default
      spec:
        provider: aws
        configAWS:
          region: ${REGION}
    EOF
    
  6. Install the Velero package.

    tanzu package install velero --package-name velero.community.tanzu.vmware.com --version 1.7.1 --values-file values.yaml
    
  7. Verify that the Velero package was properly installed.

    tanzu package installed list
    | Retrieving installed packages...
      NAME    PACKAGE-NAME                       PACKAGE-VERSION  STATUS
      velero  velero.community.tanzu.vmware.com  1.7.1            Reconcile succeeded
    

Usage Example

This walkthrough guides you through an example disaster recovery scenario that leverages the Velero package. You must deploy the package before attempting this walkthrough.

⚠️ Note: For more advanced use cases and documentation, see the official Velero documentation.

In the following steps, you will simulate a disaster scenario. Specifically, you will deploy a stateless workload, create a backup, delete the workload, and restore it from the backup.

  1. Download the Velero CLI from the GitHub releases page. The following steps assume you have installed Velero into your PATH.

  2. Create a new namespace for this example:

    kubectl create ns velero-example
    
  3. Deploy a sample workload into the new namespace:

    kubectl create deploy -n velero-example nginx --image=nginx
    
  4. Verify the workload is up and running:

    kubectl get pods -n velero-example
    

    The output should be similar to the following:

    NAME                     READY   STATUS    RESTARTS   AGE
    nginx-6799fc88d8-mm47k   1/1     Running   0          7s
    
  5. Create a backup of the velero-example namespace:

    velero create backup velero-example --include-namespaces velero-example
    
  6. Verify the backup completed successfully:

    velero describe backup velero-example
    

    The output shows the “Phase” of the backup, which should be Completed.

  7. Delete the velero-example namespace to simulate a disaster scenario:

    kubectl delete ns velero-example
    
  8. Verify that the namespace has been deleted:

    kubectl get ns
    
  9. Restore the namespace from the velero backup:

    velero create restore --from-backup velero-example
    
  10. Validate that the velero-example namespace has been restored:

    kubectl get ns velero-example
    
  11. Validate that the workload has been restored:

    kubectl get pods -n velero-example
    

Join us!

Our open community welcomes all users and contributors

Community