Failover
Pod failover
Verify that your Cassandra cluster is formed of five nodes:
kubectl get pods -l "app=cassandra"
NAME READY STATUS RESTARTS AGE
cassandra-0 1/1 Running 0 1h
cassandra-1 1/1 Running 0 10m
cassandra-2 1/1 Running 0 18h
cassandra-3 1/1 Running 0 17h
cassandra-4 1/1 Running 0 13h
Add data to Cassandra
Run the
bash
command on one of your Pods. The following example command runs thebash
command on thecassandra-2
Pod:kubectl exec -it cassandra-2 -- bash
Start
cqlsh
, the command line shell for interacting with Cassandra:cqlsh
Connected to TestCluster at 127.0.0.1:9042. [cqlsh 5.0.1 | Cassandra 3.11.4 | CQL spec 3.4.4 | Native protocol v4] Use HELP for help.
Enter the following example command to add data to a keyspace called
demodb
:CREATE KEYSPACE demodb WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 2 }; use demodb; CREATE TABLE emp(emp_id int PRIMARY KEY, emp_name text, emp_city text, emp_sal varint,emp_phone varint); INSERT INTO emp (emp_id, emp_name, emp_city, emp_phone, emp_sal) VALUES(123423445,'Steve', 'Denver', 5910234452, 50000);
Run the
exit
command to terminatecqlsh
and return to the shell session.Display the list of nodes that host the data in your Cassandra ring based on its partition key:
nodetool getendpoints demodb emp 123423445
10.0.112.1 10.0.160.1
Terminate the shell session:
exit
Use the following command to list the nodes and the Pods they host:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
{ "name": "cassandra-0", "hostname": "k8s-5", "hostIP": "10.140.0.8", "PodIP": "10.0.112.1" } { "name": "cassandra-1", "hostname": "k8s-0", "hostIP": "10.140.0.3", "PodIP": "10.0.160.1" } { "name": "cassandra-2", "hostname": "k8s-1", "hostIP": "10.140.0.5", "PodIP": "10.0.64.3" } { "name": "cassandra-3", "hostname": "k8s-3", "hostIP": "10.140.0.6", "PodIP": "10.0.240.1" } { "name": "cassandra-4", "hostname": "k8s-4", "hostIP": "10.140.0.7", "PodIP": "10.0.128.1" }
Note that the
k8s-0
node hosts thecassandra1
Pod.
Delete a Cassandra Pod
Cordon a node where one of the replicas resides. The Kubernetes stateful set will schedule the Pod to another node. The following
kubectl cordon
command cordons thek8s-0
node:kubectl cordon k8s-0
node "k8s-0" cordoned
Use the
kubectl delete pods
command to delete thecassandra-1
Pod:kubectl delete pods cassandra-1
pod "cassandra-1" deleted
The Kubernetes stateful set schedules the
cassandra-1
Pod on a different host. You can use thekubectl get pods -w
command to see where the Pod is in its lifecycle:kubectl get pods -w
NAME READY STATUS RESTARTS AGE cassandra-0 1/1 Running 0 1h cassandra-1 0/1 ContainerCreating 0 1s cassandra-2 1/1 Running 0 19h cassandra-3 1/1 Running 0 17h cassandra-4 1/1 Running 0 14h cassandra-1 0/1 Running 0 4s cassandra-1 1/1 Running 0 28s
To see the node on which the Kubernetes stateful set schedules the
cassandra-1
Pod, enter the following command:kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'
{ "name": "cassandra-0", "hostname": "k8s-5", "hostIP": "10.140.0.8", "PodIP": "10.0.112.1" } { "name": "cassandra-1", "hostname": "k8s-2", "hostIP": "10.140.0.4", "PodIP": "10.0.192.2" } { "name": "cassandra-2", "hostname": "k8s-1", "hostIP": "10.140.0.5", "PodIP": "10.0.64.3" } { "name": "cassandra-3", "hostname": "k8s-3", "hostIP": "10.140.0.6", "PodIP": "10.0.240.1" } { "name": "cassandra-4", "hostname": "k8s-4", "hostIP": "10.140.0.7", "PodIP": "10.0.128.1" }
Note that the
cassandra-1
Pod is now scheduled on thek8s-2
node.Verify that there is no data loss by entering the following command:
kubectl exec cassandra-1 -- cqlsh -e 'select * from demodb.emp'
emp_id | emp_city | emp_name | emp_phone | emp_sal -----------+----------+----------+------------+--------- 123423445 | Denver | Steve | 5910234452 | 50000 (1 rows)
Node failover
List the Pods in your cluster by entering the following command:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": status.podIP}'
{ "name": "cassandra-0", "hostname": "k8s-5", "hostIP": "10.140.0.8", "PodIP": "10.0.112.1" } { "name": "cassandra-1", "hostname": "k8s-2", "hostIP": "10.140.0.4", "PodIP": "10.0.192.2" } { "name": "cassandra-2", "hostname": "k8s-1", "hostIP": "10.140.0.5", "PodIP": "10.0.64.3" } { "name": "cassandra-3", "hostname": "k8s-3", "hostIP": "10.140.0.6", "PodIP": "10.0.240.1" } { "name": "cassandra-4", "hostname": "k8s-4", "hostIP": "10.140.0.7", "PodIP": "10.0.128.1" }
Note that Kubernetes scheduled the
cassandra-2
Pod on thek8s-1
node.Display the list of nodes and their labels:
kubectl get nodes --show-labels
NAME STATUS LABELS k8s-0 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true k8s-1 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-4=true k8s-2 Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true k8s-3 Ready cassandra-data-cassandra-3=true k8s-4 Ready cassandra-data-cassandra-4=true k8s-5 Ready k8s-master Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true
NOTE: This example output is truncated for brevity.Decommission the
k8s-1
Portworx node by following the steps in the Decommission a Node section.Decommission the
k8s-1
Kubernetes node by entering thekubectl delete node
command withk8s-1
as an argument:kubectl delete node k8s-1
List the Pods in your cluster by entering the following command:
kubectl get pods -l app=cassandra -o json | jq '.items[] | {"name": .metadata.name,"hostname": .spec.nodeName, "hostIP": .status.hostIP, "PodIP": .status.podIP}'
{ "name": "cassandra-0", "hostname": "k8s-5", "hostIP": "10.140.0.8", "PodIP": "10.0.112.1" } { "name": "cassandra-1", "hostname": "k8s-2", "hostIP": "10.140.0.4", "PodIP": "10.0.192.2" } { "name": "cassandra-2", "hostname": "k8s-0", "hostIP": "10.140.0.3", "PodIP": "10.0.160.2" } { "name": "cassandra-3", "hostname": "k8s-3", "hostIP": "10.140.0.6", "PodIP": "10.0.240.1" } { "name": "cassandra-4", "hostname": "k8s-4", "hostIP": "10.140.0.7", "PodIP": "10.0.128.1" }
Note that the
cassandra-2
pod is scheduled on thek8s-0
node.Display the list of nodes and their labels:
kubectl get nodes --show-labels
NAME STATUS LABELS k8s-0 Ready cassandra-data-cassandra-1=true,cassandra-data-cassandra-3=true k8s-2 Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true k8s-3 Ready cassandra-data-cassandra-3=true k8s-4 Ready cassandra-data-cassandra-4=true k8s-5 Ready k8s-master Ready cassandra-data-cassandra-0=true,cassandra-data-cassandra-2=true