Failover

Portworx Documentation has moved to https://docs.portworx.com

Portworx Enterprise version 2.13 has reached end of life and end of extended maintenance. Refer to the release support policy doc here.
Upgrade to the latest version of Portworx Enterprise for continued support. Documentation for the latest version of Portworx Enterprise can be found here.

Pod failover

Verify that your Cassandra cluster is formed of five nodes:

kubectl get pods -l "app=cassandra"

NAME          READY     STATUS    RESTARTS   AGE
cassandra-0   1/1       Running   0          1h
cassandra-1   1/1       Running   0          10m
cassandra-2   1/1       Running   0          18h
cassandra-3   1/1       Running   0          17h
cassandra-4   1/1       Running   0          13h

Add data to Cassandra

Run the bash command on one of your Pods. The following example command runs the bash command on the cassandra-2 Pod:
```
kubectl exec -it cassandra-2 -- bash
```

Start cqlsh, the command line shell for interacting with Cassandra:

cqlsh

Connected to TestCluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.

Enter the following example command to add data to a keyspace called demodb:

CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 };
use demodb;
CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint);
INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES(123423445,'Steve', 'Denver', 5910234452, 50000);

Run the exit command to terminate cqlsh and return to the shell session.
Display the list of nodes that host the data in your Cassandra ring based on its partition key:
```
nodetool getendpoints demodb emp 123423445
```
```
10.0.112.1
10.0.160.1
```
Terminate the shell session:
```
exit
```

Use the following command to list the nodes and the Pods they host:

kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'

{
  "name": "cassandra-0",
  "hostname": "k8s-5",
  "hostIP": "10.140.0.8",
  "PodIP": "10.0.112.1"
}
{
  "name": "cassandra-1",
  "hostname": "k8s-0",
  "hostIP": "10.140.0.3",
  "PodIP": "10.0.160.1"
}
{
  "name": "cassandra-2",
  "hostname": "k8s-1",
  "hostIP": "10.140.0.5",
  "PodIP": "10.0.64.3"
}
{
  "name": "cassandra-3",
  "hostname": "k8s-3",
  "hostIP": "10.140.0.6",
  "PodIP": "10.0.240.1"
}
{
  "name": "cassandra-4",
  "hostname": "k8s-4",
  "hostIP": "10.140.0.7",
  "PodIP": "10.0.128.1"
}

Note that the k8s-0 node hosts the cassandra1 Pod.

Delete a Cassandra Pod

Cordon a node where one of the replicas resides. The Kubernetes stateful set will schedule the Pod to another node. The following kubectl cordon command cordons the k8s-0 node:
```
kubectl cordon k8s-0
```
```
node "k8s-0" cordoned
```
Use the kubectl delete pods command to delete the cassandra-1 Pod:
```
kubectl delete pods cassandra-1
```
```
pod "cassandra-1" deleted
```

The Kubernetes stateful set schedules the cassandra-1 Pod on a different host. You can use the kubectl get pods -w command to see where the Pod is in its lifecycle:

kubectl get pods -w

NAME          READY     STATUS              RESTARTS   AGE
cassandra-0   1/1       Running             0          1h
cassandra-1   0/1       ContainerCreating   0          1s
cassandra-2   1/1       Running             0          19h
cassandra-3   1/1       Running             0          17h
cassandra-4   1/1       Running             0          14h
cassandra-1   0/1       Running   0         4s
cassandra-1   1/1       Running   0         28s

To see the node on which the Kubernetes stateful set schedules the cassandra-1 Pod, enter the following command:

kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'

{
  "name": "cassandra-0",
  "hostname": "k8s-5",
  "hostIP": "10.140.0.8",
  "PodIP": "10.0.112.1"
}
{
  "name": "cassandra-1",
  "hostname": "k8s-2",
  "hostIP": "10.140.0.4",
  "PodIP": "10.0.192.2"
}
{
  "name": "cassandra-2",
  "hostname": "k8s-1",
  "hostIP": "10.140.0.5",
  "PodIP": "10.0.64.3"
}
{
  "name": "cassandra-3",
  "hostname": "k8s-3",
  "hostIP": "10.140.0.6",
  "PodIP": "10.0.240.1"
}
{
  "name": "cassandra-4",
  "hostname": "k8s-4",
  "hostIP": "10.140.0.7",
  "PodIP": "10.0.128.1"
}

Note that the cassandra-1 Pod is now scheduled on the k8s-2 node.

Verify that there is no data loss by entering the following command:

kubectl exec cassandra-1 -- cqlsh -e 'select * from demodb.emp'

emp_id    | emp_city | emp_name | emp_phone  | emp_sal
-----------+----------+----------+------------+---------
123423445 |   Denver |    Steve | 5910234452 |   50000

(1 rows)

Node failover

List the Pods in your cluster by entering the following command:

kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'

{
  "name": "cassandra-0",
  "hostname": "k8s-5",
  "hostIP": "10.140.0.8",
  "PodIP": "10.0.112.1"
}
{
  "name": "cassandra-1",
  "hostname": "k8s-2",
  "hostIP": "10.140.0.4",
  "PodIP": "10.0.192.2"
}
{
  "name": "cassandra-2",
  "hostname": "k8s-1",
  "hostIP": "10.140.0.5",
  "PodIP": "10.0.64.3"
}
{
  "name": "cassandra-3",
  "hostname": "k8s-3",
  "hostIP": "10.140.0.6",
  "PodIP": "10.0.240.1"
}
{
  "name": "cassandra-4",
  "hostname": "k8s-4",
  "hostIP": "10.140.0.7",
  "PodIP": "10.0.128.1"
}

Note that Kubernetes scheduled the cassandra-2 Pod on the k8s-1 node.

Display the list of nodes and their labels:

kubectl get nodes --show-labels

NAME         STATUS        LABELS
k8s-0        Ready         cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true
k8s-1        Ready         cassandra-data-cassandra-1=true,cassandra-data-cassandra-4=true
k8s-2        Ready         cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
k8s-3        Ready         cassandra-data-cassandra-3=true
k8s-4        Ready         cassandra-data-cassandra-4=true
k8s-5        Ready
k8s-master   Ready         cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true

NOTE: This example output is truncated for brevity.

Decommission the k8s-1 Portworx node by following the steps in the Decommission a Node section.
Decommission the k8s-1 Kubernetes node by entering the kubectl delete node command with k8s-1 as an argument:
```
kubectl delete node k8s-1
```

List the Pods in your cluster by entering the following command:

kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'

{
  "name": "cassandra-0",
  "hostname": "k8s-5",
  "hostIP": "10.140.0.8",
  "PodIP": "10.0.112.1"
}
{
  "name": "cassandra-1",
  "hostname": "k8s-2",
  "hostIP": "10.140.0.4",
  "PodIP": "10.0.192.2"
}
{
  "name": "cassandra-2",
  "hostname": "k8s-0",
  "hostIP": "10.140.0.3",
  "PodIP": "10.0.160.2"
}
{
  "name": "cassandra-3",
  "hostname": "k8s-3",
  "hostIP": "10.140.0.6",
  "PodIP": "10.0.240.1"
}
{
  "name": "cassandra-4",
  "hostname": "k8s-4",
  "hostIP": "10.140.0.7",
  "PodIP": "10.0.128.1"
}

Note that the cassandra-2 pod is scheduled on the k8s-0 node.

Display the list of nodes and their labels:

kubectl get nodes --show-labels

NAME         STATUS        LABELS
k8s-0        Ready         cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true
k8s-2        Ready         cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
k8s-3        Ready         cassandra-data-cassandra-3=true
k8s-4        Ready         cassandra-data-cassandra-4=true
k8s-5        Ready
k8s-master   Ready         cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true

Scale up

Snapshots