Migration with Stork on Kubernetes

This document will walk you through how to migrate your Portworx volumes between clusters with Stork on Kubernetes.

Prerequisites

Before we begin, please make sure the following prerequisites are met:

• Version: To take advantage of the features described in this document, the source AND destination clusters must have Stork 2.12.0 or later, Operator 1.10.0 or later, and Portworx Enterprise 2.11.0 or later.
• Stork helper : storkctl is a command-line tool for interacting with a set of scheduler extensions.
• Secret Store : Make sure you have configured a secret store on both clusters. This will be used to store the credentials for the objectstore.
• Network Connectivity:
  • Portworx: Port 9001 on the destination cluster should be reachable by the source cluster. For OpenShift, Port 17001 should be reachable.
  • Kubernetes API End Point: Port 6443, 443, or 8443 should be reachable. Check with your Kubernetes administrator to determine which port you need.

Download storkctl

Perform the following steps to download storkctl from the Stork pod:

• Linux:

  STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
  kubectl cp -n <namespace> $STORK_POD:/storkctl/linux/storkctl ./storkctl
  sudo mv storkctl /usr/local/bin &&
  sudo chmod +x /usr/local/bin/storkctl

• OS X:

  STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
  kubectl cp -n <namespace> $STORK_POD:/storkctl/darwin/storkctl ./storkctl
  sudo mv storkctl /usr/local/bin &&
  sudo chmod +x /usr/local/bin/storkctl

• Windows:

  1. Copy storkctl.exe from the stork pod:

     STORK_POD=$(kubectl get pods -n <namespace> -l name=stork -o jsonpath='{.items[0].metadata.name}') &&
     kubectl cp -n <namespace> $STORK_POD:/storkctl/windows/storkctl.exe ./storkctl.exe

  2. Move storkctl.exe to a directory in your PATH.

In Kubernetes, you must define a trust object called ClusterPair. Portworx requires this object to communicate with the destination cluster. The ClusterPair object pairs the Portworx storage driver with the Kubernetes scheduler, allowing volumes and resources to be migrated between clusters.

The ClusterPair is generated and used in the following way:

• It requires storkctl to be running on the Kubernetes control plane node in both the destination and source clusters
• The ClusterPair spec is generated on the destination cluster
• The generated spec is then applied on the source cluster

Perform the following steps to create a cluster pair:

Create object store credentials for cloud clusters

Create object store credentials on your source and destination clusters. The options you use to create your object store credentials differ based on which object store you use.

Create Amazon s3 or s3 compatible object store credentials

1. Find the UUID of your destination cluster by running the following command:

     PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
     kubectl exec $PX_POD -n <namespace> --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'

2. Enter the pxctl credentials create command, specifying the following:

   • The --provider flag with the name of the cloud provider (s3).
   • The --s3-access-key flag with your secret access key
   • The --s3-secret-key flag with your access key ID
   • The --s3-region flag with the name of the S3 region (us-east-1)
   • The --s3-endpoint flag with the name of the endpoint (s3.amazonaws.com)
   • The optional --s3-storage-class flag with either the STANDARD or STANDARD-IA value, depending on which storage class you prefer

   • clusterPair_<UUID_of_destination_cluster> Enter the UUID of your destination cluster collected in step 1.

     /opt/pwx/bin/pxctl credentials create \
     --provider s3 \
     --s3-access-key <aws_access_key> \
     --s3-secret-key <aws_secret_key> \
     --s3-region us-east-1  \
     --s3-endpoint s3.amazonaws.com \
     --s3-storage-class STANDARD \
     clusterPair_<UUID_of_destination_cluster>

Create Microsoft Azure credentials

1. Find the UUID of your destination cluster by running the following command:

     PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
     kubectl exec $PX_POD -n <namespace> --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'

2. Enter the pxctl credentials create command, specifying the following:

   • --provider as azure
   • --azure-account-name with the name of your Azure account
   • --azure-account-key with your Azure account key

   • clusterPair_<UUID_of_destination_cluster> Enter the UUID of your destination cluster collected in step 1.

     /opt/pwx/bin/pxctl credentials create \
     --provider azure \
     --azure-account-name <your_azure_account_name> \
     --azure-account-key <your_azure_account_key> \
     clusterPair_<UUID_of_destination_cluster>

Create Google Cloud Platform credentials

1. Find the UUID of your destination cluster by running the following command:

     PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
     kubectl exec $PX_POD -n <namespace> --  /opt/pwx/bin/pxctl status | grep UUID | awk '{print $3}'

2. Enter the pxctl credentials create command, specifying the following:

   • --provider as google
   • --google-project-id with the string of your Google project ID
   • --google-json-key-file with the filename of your GCP JSON key file

   • clusterPair_<UUID_of_destination_cluster> Enter the UUID of your destination cluster collected in step 1.

     /opt/pwx/bin/pxctl credentials create \
     --provider google \
     --google-project-id <your_google_project_ID> \
     --google-json-key-file <your_GCP_JSON_key_file> \
     clusterPair_<UUID_of_destination_cluster>

Using Rancher Projects with ClusterPair

Rancher has a concept of Projects that allow grouping of resources and Kubernetes namespaces. Depending on the resource and how it is created, Rancher adds the following label or annotation:

field.cattle.io/projectID: <project-short-UUID>

The projectID uniquely identifies the project, and the annotation or label on the Kubernetes object provides a way to tie a Kubernetes object back to a Rancher project.

From version 2.11.2 or newer, Stork has the capability to map projects from the source cluster to the destination cluster when it migrates Kubernetes resources. It will ensure that the following are transformed when migrating Kubernetes resources to a destination cluster:

• Labels and annotations for projectID field.cattle.io/projectID on any Kubernetes resource on the source cluster are transformed to their respective projectIDs on the destination cluster.
• Namespace Selectors on a NetworkPolicy object which refer to the field.cattle.io/projectID label will be transformed to their respective projectIDs on the destination cluster.
• Namespace Selectors on a Pod object (Kubernetes version 1.24 or newer) which refer to the field.cattle.io/projectID label will be transformed to their respective projectIDs on the destination cluster.

While creating the ClusterPair, use the argument --project-mappings to indicate which projectID on the source cluster maps to a projectID on the destination cluster.

For example:

storkctl generate clusterpair -n <migrationnamespace> <remotecluster> --project-mappings  <projectID-A1>=<projectID-A2>,<projectID-B1>=<projectID-B2>

The project mappings are provided as a comma-separate key=value pairs. In this example, projectID-A1 on source cluster maps to projectID-A2 on the destination cluster, while projectID-B1 on the source cluster maps to projectID-B2 on the destination cluster.

Generate a ClusterPair from the destination cluster

To generate the ClusterPair spec, run the following command on the destination cluster:

storkctl generate clusterpair -n <migrationnamespace> <remotecluster>

Here, remotecluster is the Kubernetes object that will be created on the source cluster representing the pair relationship, and migrationnamespace is the Kubernetes namespace of the source cluster that you want to migrate to the destination cluster.

During the actual migration, you will reference this name to identify the destination of your migration.

  annotations:
    openstorage.io/auth-secret-namespace -> Points to the namespace where the k8s secret holding the auth token resides.
    openstorage.io/auth-secret-name -> Points to the name of the k8s secret which holds the auth token.

Save the resulting spec to a file named clusterpair.yaml.

Example:

storkctl generate clusterpair -n <migrationnamespace> <remotecluster> -o yaml > clusterpair.yaml

apiVersion: stork.libopenstorage.org/v1alpha1
kind: ClusterPair
metadata:
    creationTimestamp: null
    name: <remotecluster>
    namespace: <migrationnamespace>
    annotations:
      # Add the below annotations when PX-Security is enabled on both the clusters
      #openstorage.io/auth-secret-namespace -> Points to the namespace where the k8s secret holding the auth token resides
      #openstorage.io/auth-secret-name -> Points to the name of the k8s secret which holds the auth token.
spec:
   config:
      clusters:
         kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            certificate-authority-data: <CA_DATA>
            server: https://192.168.56.74:6443
      contexts:
         kubernetes-admin@kubernetes:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            cluster: kubernetes
            user: kubernetes-admin
      current-context: kubernetes-admin@kubernetes
      preferences: {}
      users:
         kubernetes-admin:
            LocationOfOrigin: /etc/kubernetes/admin.conf
            client-certificate-data: <CLIENT_CERT_DATA>
            client-key-data: <CLIENT_KEY_DATA>
    options:
       <insert_storage_options_here>: ""
status:
  remoteStorageId: ""
  schedulerStatus: ""
  storageStatus: ""

Get the destination cluster token

On the destination cluster, run the following command from one of the Portworx nodes to get the cluster token. You’ll need this token in later steps:

PX_POD=$(kubectl get pods -l name=portworx -n <namespace> -o jsonpath='{.items[0].metadata.name}')
kubectl exec $PX_POD -n <namespace> --  /opt/pwx/bin/pxctl cluster token show

Insert Storage Options

To pair the storage, specify the following fields in the options section of the clusterpair.yaml file saved previously:

• ip: The IP address of any of the remote Portworx nodes. If using external load balancer, specify the IP address of the external load balancer.
• port: The port of the remote Portworx node. See Network Connectivity in the Prerequisites section of this document.
• token: The token of the destination cluster, which you obtained in the previous step.

See example below:

  options:
    token: "34b2fd8df839650ed8f9b5cd5a70e1ca6d79c1ebadb5f50e29759525a20aee7daa5aae35e1e23729a4d9f673ce465dbe3679032d1be7a1bcc489049a3c23a460"
    ip: "10.13.21.125"
    port: "9001"

Apply the ClusterPair spec on the source cluster

To create the ClusterPair, apply the ClusterPair YAML spec on the source cluster. Run the following command from a location where you have kubectl access to the source cluster:

kubectl create -f clusterpair.yaml

Verify the ClusterPair

To verify that you have generated the ClusterPair and that it is ready, run the following command:

storkctl get clusterpair -n <migrationnamespace>

NAME            STORAGE-STATUS   SCHEDULER-STATUS   CREATED
remotecluster   Ready            Ready              09 Nov 22 00:22 UTC

On a successful pairing, you should see the STORAGE-STATUS and SCHEDULER-STATUS as Ready.

If you see an error instead, you can get more information by running the following command:

kubectl describe clusterpair remotecluster -n <migrationnamespace>

Migrating volumes and resources

Once the pairing is configured, applications can be migrated repeatedly to the destination cluster.

Start a migration

1. Create a file called migration.yaml file, specifying the following fields and values:

   • apiVersion: as stork.libopenstorage.org/v1alpha1
   • kind: as Migration
   • metadata.name: with the name of the object that performs the migration
   • metadata.namespace: with the name of the namespace in which you want to create the object
   • spec.clusterPair: with the name of the ClusterPair object created in the Generate a ClusterPair from the destination cluster section
   • spec.includeResources: with a boolean value specifying if the migration should include PVCs and other applications specs. If you set this field to false, Portworx will only migrate your volumes.
   • spec.startApplications: with a boolean value specifying if Portworx should automatically start applications on the destination cluster. If you set this field to false, then the Deployment and StatefulSet objects on the destination cluster will be set to zero. Note that, on the destination cluster, Portworx uses the stork.openstorage.org/migrationReplicas annotation to store the number of replicas from the source cluster.
   • spec.namespaces: with the list of namespaces you want to migrate
   • spec.adminClusterPair: with the name of the ClusterPair object created in the admin namespace that Portworx should use to migrate your cluster-scoped resources. Use this if your regular users lack permission to create or delete cluster-scoped resources on the destination cluster. If you don’t specify this field, then Stork will use the object specified in the spec.clusterPair field. This feature was introduced in Stork version 2.3.0.

   • spec.purgeDeletedResources: with a boolean value specifying if Stork should automatically purge a resource from the destination cluster when you delete it from the source cluster. The default value is false. This feature was introduced in Stork version 2.3.2.

     apiVersion: stork.libopenstorage.org/v1alpha1
     kind: Migration
     metadata:
     name: <YOUR_MIGRATION_OBJECT>
     namespace: <YOUR_MIGRATION_NAMESPACE>
     spec:
     clusterPair: <YOUR_CLUSTER_PAIR>
     includeResources: true
     startApplications: true
     namespaces:
     - <NAMESPACE_TO_MIGRATE>
     adminClusterPair: <YOUR_ADMIN_CLUSTER_PAIR>
     purgeDeletedResources: false

2. Apply the spec by entering the following command:

   kubectl apply -f migration.yaml

storkctl create migration <YOUR_MIGRATION_OBJECT> --clusterPair <YOUR_CLUSTER_PAIR> --namespaces <NAMESPACE_TO_MIGRATE> --includeResources --startApplications -n <YOUR_NAMESPACE>

Migration scope

Currently, you can only migrate namespaces in which the object is created. You can also designate one namespace as an admin namespace. This will allow an admin who has access to that namespace to migrate any namespace from the source cluster. Instructions for setting this admin namespace to stork can be found in Set up a cluster admin namespace.

Monitoring a migration

Once the migration has been started using the previous commands, you can check the status using storkctl. For example:

storkctl get migration -n <migrationnamespace>

First, you should see something like the following:

NAME                                             CLUSTERPAIR     STAGE     STATUS       VOLUMES   RESOURCES   CREATED
zoomigrationschedule-interval-2022-12-12-200210  remotecluster   Volumes   InProgress   0/3       0/0         12 Dec 22 11:45 PST

If the migration is successful, the STAGE will go from Volumes to Application to Final.

Here is how the example output of a successful migration looks:

NAME                                              CLUSTERPAIR   STAGE   STATUS       VOLUMES   RESOURCES   CREATED               ELAPSED
zoomigrationschedule-interval-2022-12-12-200210   remotecluster Final   Successful   3/3       10/10       12 Dec 22 12:02 PST   1m23s

Troubleshooting

If there is a failure or you want more information about what resources were migrated you can describe the migration object using kubectl:

kubectl describe migration <migrationschedulename> -n <migrationnamespace>

Name:         zoomigrationschedule-interval-2022-12-12-220239
Namespace:    zookeeper
Labels:       <none>
Annotations:  stork.libopenstorage.org/migration-schedule-name: zoomigrationschedule
API Version:  stork.libopenstorage.org/v1alpha1
Kind:         Migration
Metadata:
  Creation Timestamp:  2022-12-12T22:02:39Z
  Finalizers:
    stork.libopenstorage.org/finalizer-cleanup
  Generation:  202
  Managed Fields:
    API Version:  stork.libopenstorage.org/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
    ......
    ...........
    Manager:    stork
    Operation:  Update
    Time:       2022-12-12T22:02:40Z
  Owner References:
    API Version:     stork.libopenstorage.org/v1alpha1
    Kind:            MigrationSchedule
    Name:            zoomigrationschedule
    UID:             5ae0cdb9-e4ba-4562-8094-8d7763d99ed1
  Resource Version:  31906052
  UID:               89c0ae5e-f08a-454f-b7b3-59951662ab0b
Spec:
  Admin Cluster Pair:                
  Cluster Pair:                      remotecluster
  Include Network Policy With CIDR:  false
  Include Optional Resource Types:   <nil>
  Include Resources:                 true
  Include Volumes:                   true
  Namespaces:
    zookeeper
  Post Exec Rule:           
  Pre Exec Rule:            
  Purge Deleted Resources:  false
  Selectors:                <nil>
  Skip Deleted Namespaces:  <nil>
  Skip Service Update:      false
  Start Applications:       false
Status:
  Finish Timestamp:                     2022-12-12T22:04:40Z
  Resource Migration Finish Timestamp:  2022-12-12T22:04:40Z
  Resources:
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d
    Namespace:  
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c
    Namespace:  
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolume
    Name:       pvc-fdd559fa-b42c-4424-9699-cf13bc14a77e
    Namespace:  
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       datadir-zk-0
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       datadir-zk-1
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       PersistentVolumeClaim
    Name:       datadir-zk-2
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Service
    Name:       zk-cs
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      core
    Kind:       Service
    Name:       zk-hs
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      apps
    Kind:       StatefulSet
    Name:       zk
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
    Group:      policy
    Kind:       PodDisruptionBudget
    Name:       zk-pdb
    Namespace:  zookeeper
    Reason:     Resource migrated successfully
    Status:     Successful
    Version:    v1
  Stage:        Final
  Status:       Successful
  Summary:
    Elapsed Time For Resource Migration:  7s
    Elapsed Time For Volume Migration:    1m54s
    Num Of Migrated Resources:            10
    Num Of Migrated Volumes:              3
    Total Bytes Migrated:                 12288
    Total Number Of Resources:            10
    Total Number Of Volumes:              3
  Volume Migration Finish Timestamp:      2022-12-12T22:04:33Z
  Volumes:
    Bytes Total:              4096
    Namespace:                zookeeper
    Persistent Volume Claim:  datadir-zk-0
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d
    Bytes Total:              4096
    Namespace:                zookeeper
    Persistent Volume Claim:  datadir-zk-1
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c
    Bytes Total:              4096
    Namespace:                zookeeper
    Persistent Volume Claim:  datadir-zk-2
    Reason:                   Migration successful for volume
    Status:                   Successful
    Volume:                   pvc-fdd559fa-b42c-4424-9699-cf13bc14a77e
Events:
  Type     Reason      Age                 From   Message
  ----     ------      ----                ----   -------
  Warning  Failed      20m                 stork  Error migrating volumes: Operation cannot be fulfilled on migrations.stork.libopenstorage.org "zoomigrationschedule-interval-2022-12-12-220239": the object has been modified; please apply your changes to the latest version and try again
  Normal   Successful  19m (x13 over 19m)  stork  Volume pvc-0a330e8b-d5f3-4c65-bca6-3eb37adcfb2d migrated successfully
  Normal   Successful  19m (x11 over 19m)  stork  Volume pvc-62ad252d-2218-4d33-9c57-8e61641b1a8c migrated successfully

Pre and Post Exec rules

Similar to snapshots, a PreExec and PostExec rule can be specified when creating a Migration object. This will result in the PreExec rule being run before the migration is triggered and the PostExec rule to be run after the Migration has been triggered. If the rules do not exist, the Migration will log an event and will stop.

If the PreExec rule fails for any reason, it will log an event against the object and retry. The Migration will not be marked as failed.

If the PostExec rule fails for any reason, it will log an event and mark the Migration as failed. It will also try to cancel the migration that was started from the underlying storage driver.

In the example below with mysql, to add pre and post rules to our migration, we could edit our migration.yaml file like this:

apiVersion: stork.libopenstorage.org/v1alpha1
kind: Migration
metadata:
  name: mysqlmigration
  namespace: mysql
spec:
  clusterPair: remotecluster
  includeResources: true
  startApplications: true
  preExecRule: mysql-pre-rule
  postExecRule: mysql-post-rule
  namespaces:
  - mysql

Advanced Operations

• Migration with Stork on EKS
• Migration with Stork on GKE
• Migration with Stork on OKE
• Configuring a namespace as a cluster namespace