Failover an application

In case of a disaster, where one of your Kubernetes clusters is down and inaccessible, you can failover the applications running on it to an operational Kubernetes cluster. To achieve this, you should stop your application on the source cluster and start the application on an active Kubernetes cluster.

The following considerations are used in the examples on this page. Update them to the appropriate values for your environment.

• Source Cluster is the Kubernetes cluster which is down and where your applications were originally running.
• Destination Cluster is the Kubernetes cluster where the applications will be failed over.
• The Zookeeper application is being failed over to the destination cluster.

Stop the application on the source cluster (if accessible or applicable)

If your source Kubernetes cluster is still alive and is accessible, Portworx, Inc. recommends you to stop the applications before failing them over to the destination cluster.

Scale down the replica count of your application:

kubectl scale --replicas 0 statefulset/zk -n zookeeper

As the zookeeper namespace is being used in the above command, it will scale down the replica count for the Zookeeper application. Update the namespace to your application namespace.

Suspend the migrations on the source cluster (if accessible)

1. Run the following command to suspend the migration schedule. Once the replicas for your application’s statefulset are set to 0, you need to suspend the migration schedule on the source cluster. This is done so that your application’s stateful sets are not updated to 0 replicas on the destination cluster:

   storkctl suspend migrationschedule migrationschedule -n <migrationnamespace>

2. Verify if the schedule has been suspended:

   storkctl get migrationschedule -n <migrationnamespace>

   NAME                POLICYNAME   CLUSTERPAIR     SUSPEND   LAST-SUCCESS-TIME     LAST-SUCCESS-DURATION
   migrationschedule   <your-schedule-policy>   remotecluster   true      01 Dec 22 23:31 UTC   10s

Start the application on the destination cluster

You can allow Stork to activate migration either on all namespaces or one namespace at a time. For performance reasons, if you have a high number of namespaces in your migration schedule, Portworx, Inc. recommends you migrate one namespace at a time.

1. Each application spec will have the annotation stork.openstorage.org/migrationReplicas indicating the replica count on the source cluster. Run the following command to update the replica count of your app to the same number as on your source cluster:

   storkctl activate migration -n zookeeper

   Run the following command to migrate all namespaces:

   storkctl activate migration --all-namespaces

   Stork will look for that annotation and scale it to the correct number automatically. Once the replica count is updated, the application will start running, and the failover will be completed.

2. Verify that your application is up and running:

   kubectl get pods -n zookeeper

   NAME   READY   STATUS    RESTARTS      AGE
   zk-0   1/1     Running   0             3m18s
   zk-1   1/1     Running   0             2m54s
   zk-2   1/1     Running   0             99s

   You can see that the status of all the pods for Zookeeper shows running, indicating that your application is operational.