Solr on Kubernetes on Portworx

The examples provided create a Solr cluster running in Kubernetes, which uses Portworx volumes for Zookeeper and Solr data.

The Portworx StorageClass for Volume provisioning

Check your cluster nodes

kubectl get nodes -o wide

 NAME                           STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION              CONTAINER-RUNTIME
ravi-blr-dev-dour-shoulder-0   Ready    master   44d   v1.14.1   70.0.87.119   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
ravi-blr-dev-dour-shoulder-1   Ready    <none>   44d   v1.14.1   70.0.87.82    <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
ravi-blr-dev-dour-shoulder-2   Ready    <none>   44d   v1.14.1   70.0.87.118   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6
ravi-blr-dev-dour-shoulder-3   Ready    <none>   44d   v1.14.1   70.0.87.120   <none>        CentOS Linux 7 (Core)   3.10.0-862.3.2.el7.x86_64   docker://18.9.6

Define StorageClass.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: portworx-sc
provisioner: kubernetes.io/portworx-volume
parameters:
  repl: "2"
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: Immediate

Apply the StorageClass configuration

kubectl apply -f portworx-sc.yml

Install Zookeeper cluster

Zookeeper ensemble is used for managing the confinguration for Solr. Define config properties for zookeeper configmap zk-config.properties.

zooLogDir=/store/logs
zooDataLogDir=/store/datalog
zooDataDir=/store/data
zooStandaloneEnabled=false
zooServers=server.1=zk-0.zkensemble:2888:3888 server.2=zk-1.zkensemble:2888:3888 server.3=zk-2.zkensemble:2888:3888

Create a configmap

kubectl create configmap zookeeper-ensemble-config --from-env-file=zk-config.properties

The Spec below deines a headless service, PodDisruptionBudget and zookeeper statefulset.

apiVersion: v1
kind: Service
metadata:
  name: zkensemble
  labels:
    app: zookeeper-app
spec:
  clusterIP: None
  selector:
    app: zookeeper-app
---
apiVersion: v1
kind: Service
metadata:
  name: zk-service
  labels:
    app: zookeeper-app
spec:
  ports:
  - protocol: TCP
    port: 2181
    targetPort: 2181
  type: LoadBalancer
  selector:
    app: zookeeper-app
---
apiVersion: v1
kind: Service
metadata:
  name: zk-headless
  labels:
    app: zk-headless
spec:
  ports:
  - port: 2888
    name: server
  - port: 3888
    name: leader-election
  clusterIP: None
  selector:
    app: zk
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: zk-config
data:
  ensemble: "zk-0;zk-1;zk-2"
  jvm.heap: "2G"
  tick: "2000"
  init: "10"
  sync: "5"
  client.cnxns: "60"
  snap.retain: "3"
  purge.interval: "1"
---
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: zk-budget
spec:
  selector:
    matchLabels:
      app: zk
  minAvailable: 2
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: zk
spec:
  selector:
    matchLabels:
      app: zk
  serviceName: zk-headless
  replicas: 3
  template:
    metadata:
      labels:
        app: zk
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
    spec:
      # Use the stork scheduler to enable more efficient placement of the pods
      schedulerName: stork
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: px/enabled
                operator: NotIn
                values:
                - "false"
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                    - zk-headless
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: k8szk
        imagePullPolicy: Always
        image: gcr.io/google_samples/k8szk:v1
        ports:
        - containerPort: 2181
          name: client
        - containerPort: 2888
          name: server
        - containerPort: 3888
          name: leader-election
        env:
        - name : ZK_ENSEMBLE
          valueFrom:
            configMapKeyRef:
              name: zk-config
              key: ensemble
        - name : ZK_HEAP_SIZE
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: jvm.heap
        - name : ZK_TICK_TIME
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: tick
        - name : ZK_INIT_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: init
        - name : ZK_SYNC_LIMIT
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: tick
        - name : ZK_MAX_CLIENT_CNXNS
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: client.cnxns
        - name: ZK_SNAP_RETAIN_COUNT
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: snap.retain
        - name: ZK_PURGE_INTERVAL
          valueFrom:
            configMapKeyRef:
                name: zk-config
                key: purge.interval
        - name: ZK_CLIENT_PORT
          value: "2181"
        - name: ZK_SERVER_PORT
          value: "2888"
        - name: ZK_ELECTION_PORT
          value: "3888"
        command:
        - sh
        - -c
        - zkGenConfig.sh && zkServer.sh start-foreground
        readinessProbe:
          exec:
            command:
            - "zkOk.sh"
          initialDelaySeconds: 15
          timeoutSeconds: 5
        livenessProbe:
          exec:
            command:
            - "zkOk.sh"
          initialDelaySeconds: 15
          timeoutSeconds: 5
        volumeMounts:
        - name: datadir
          mountPath: /var/lib/zookeeper
      securityContext:
        runAsUser: 1000
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: datadir
      annotations:
        volume.beta.kubernetes.io/storage-class: portworx-sc
    spec:
      storageClassName: portworx-sc
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 2Gi

Apply the above configuration

kubectl create -f zookeeper-ensemble.yml

Post Zookeeper install

Verify the Zookeeper pods are up and running with Portworx volumes

kubectl get pods

NAME       READY   STATUS              RESTARTS   AGE
zk-0       1/1     Running             0          39m
zk-1       1/1     Running             0          6h16m
zk-2       1/1     Running             0          6h15m

Check pvc

kubectl get pvc

[root@ravi-blr-dev-dour-shoulder-0 ~]# kubectl get pvc
NAME                   STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-zk-0           Bound         pvc-492cc607-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h14m
datadir-zk-1           Bound         pvc-556adfd1-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h13m
datadir-zk-2           Bound         pvc-6dca0c3b-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h13m

Check statefulset

kubectl get sts

NAME     READY   AGE
zk       3/3     6h17m

Observe that volumes are bound to zookeeper pods.

pxctl volume inspect pvc-492cc607-db62-11e9-a83a-000c29886e3e

Volume	:  534644031852659169
	Name            	 :  pvc-492cc607-db62-11e9-a83a-000c29886e3e
	Size            	 :  2.0 GiB
	Format          	 :  ext4
	HA              	 :  2
	IO Priority     	 :  LOW
	Creation time   	 :  Sep 20 04:51:19 UTC 2019
	Shared          	 :  no
	Status          	 :  up
	State           	 :  Attached: acba9a03-6781-4962-8f7c-3507eafa45ea (70.0.87.118)
	Device Path     	 :  /dev/pxd/pxd534644031852659169
	Labels          	 :  repl=2,namespace=default,pvc=datadir-zk-0
	Reads           	 :  59
	Reads MS        	 :  60
	Bytes Read      	 :  1126400
	Writes          	 :  21
	Writes MS       	 :  103
	Bytes Written   	 :  172032
	IOs in progress 	 :  0
	Bytes used      	 :  896 KiB
	Replica sets on nodes:
		Set 0
		  Node 		 : 70.0.87.120 (Pool 1)
		  Node 		 : 70.0.87.118 (Pool 2)
	Replication Status	 :  Up
	Volume consumers	 : 
		- Name           : zk-0 (787278d3-db91-11e9-a83a-000c29886e3e) (Pod)
		  Namespace      : default
		  Running on     : ravi-blr-dev-dour-shoulder-2
		  Controlled by  : zk (StatefulSet)

Verify that zookeeper ensemble is working.

kubectl exec zk-0 -- /opt/zookeeper/bin/zkCli.sh create /foo bar

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
Created /foo

kubectl exec zk-2 -- /opt/zookeeper/bin/zkCli.sh get /foo

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
cZxid = 0x200000002
bar
ctime = Thu Sep 19 05:23:10 UTC 2019
mZxid = 0x200000002
mtime = Thu Sep 19 05:23:10 UTC 2019
pZxid = 0x200000002
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 3
numChildren = 0

Install Solr

Define config properties for solr configmap solr-config.properties.

solrHome=/store/data
solrPort=8983
zkHost=zk-0.zkensemble:2181,zk-1.zkensemble:2181,zk-2.zkensemble:2181
solrLogsDir=/store/logs
solrHeap=1g

Create configmap solr-cluster-config.

kubectl create -f configmap solr-cluster-config --from-env-file=solr-config.properties

The following spec defines solr service, PodDisruptionBudget, 2 node solr cluster.

apiVersion: v1
kind: Service
metadata:
  name: solrcluster
  labels:
    app: solr-app
spec:
  clusterIP: None
  selector:
    app: solr-app
---
apiVersion: v1
kind: Service
metadata:
  name: solr-service
  labels:
    app: solr-app
spec:
  ports:
  - protocol: TCP
    port: 8983
    targetPort: 8983
  type: LoadBalancer
  selector:
    app: solr-app
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: solr
spec:
  selector:
    matchLabels:
      app: solr-app # has to match .spec.template.metadata.labels
  serviceName: "solrcluster"
  replicas: 2 # by default is 1
  template:
    metadata:
      labels:
        app: solr-app # has to match .spec.selector.matchLabels
    spec:
      terminationGracePeriodSeconds: 10
      restartPolicy: Always
      containers:
      - name: solr
        image: solr:8.1.1
        imagePullPolicy: IfNotPresent
        readinessProbe:
          tcpSocket:
            port: 8983
          initialDelaySeconds: 5
          periodSeconds: 10
        livenessProbe:
          tcpSocket:
            port: 8983
          initialDelaySeconds: 15
          periodSeconds: 20
        volumeMounts:
        - name: volsolr
          mountPath: /store
        ports:
        - name: solrport
          containerPort: 8983
        env:
          - name: MY_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: MY_POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
          - name: MY_POD_IP
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: SOLR_HOME
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrHome
          - name: ZK_HOST
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: zkHost
          - name: POD_HOST_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: SOLR_HOST
            value: "$(POD_HOST_NAME).solrcluster"
          - name: SOLR_LOGS_DIR
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrLogsDir
          - name: SOLR_HEAP
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrHeap
      initContainers:
      - name: init-solr-data
        image: busybox
        command:
        - "/bin/sh"
        - "-c"
        - "if [ ! -d $SOLR_HOME/lib ] ; then mkdir -p $SOLR_HOME/lib && chown -R 8983:8983 $SOLR_HOME ; else true; fi"
        env:
          - name: SOLR_HOME
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrHome
        volumeMounts:
        - name: volsolr
          mountPath: /store
      - name: init-solr-logs
        image: busybox
        command:
        - "/bin/sh"
        - "-c"
        - "if [ ! -d $SOLR_LOGS_DIR ] ; then mkdir -p $SOLR_LOGS_DIR && chown 8983:8983 $SOLR_LOGS_DIR ; else true; fi"
        env:
          - name: SOLR_LOGS_DIR
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrLogsDir
        volumeMounts:
        - name: volsolr
          mountPath: /store
      - name: init-solr-xml
        image: solr:8.1.1
        command:
        - "/bin/sh"
        - "-c"
        - "if [ ! -f $SOLR_HOME/solr.xml ] ; then cp /opt/solr/server/solr/solr.xml $SOLR_HOME/solr.xml;\
               sed -i \"s/<solr>/<solr><str name='sharedLib'>\\/store\\/data\\/lib<\\/str>/g\" $SOLR_HOME/solr.xml ; else true; fi "
        env:
          - name: SOLR_HOME
            valueFrom:
              configMapKeyRef:
                name: solr-cluster-config
                key: solrHome
        volumeMounts:
        - name: volsolr
          mountPath: /store
  volumeClaimTemplates:
  - metadata:
      name: volsolr
    spec:
      storageClassName: portworx-sc
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 32Gi

Apply above configuration.

kubectl create -f solr-cluster.yml

Post install status - Solr

Verify Solr resources created on the cluster.

kubectl get pods

NAME       READY   STATUS              RESTARTS   AGE
solr-0     1/1     Running             0          78m
solr-1     1/1     Running             0          106m
solr-2     1/1     Running             0          105m
zk-0       1/1     Running             0          5h41m
zk-1       1/1     Running             0          5h41m
zk-2       1/1     Running             0          28m

kubectl get pvc

NAME                   STATUS        VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
datadir-zk-0           Bound         pvc-492cc607-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h25m
datadir-zk-1           Bound         pvc-556adfd1-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h25m
datadir-zk-2           Bound         pvc-6dca0c3b-db62-11e9-a83a-000c29886e3e   2Gi        RWO            portworx-sc    6h24m
volsolr-solr-0         Bound         pvc-2d10ff4f-db6c-11e9-a83a-000c29886e3e   32Gi       RWO            portworx-sc    5h14m
volsolr-solr-1         Bound         pvc-506f787d-db6c-11e9-a83a-000c29886e3e   32Gi       RWO            portworx-sc    5h13m

pxctl volume list

ID			NAME						SIZE	HA	SHARED	ENCRYPTED	IO_PRIORITY	STATUS			SNAP-ENABLED	
865481696383214454	pvc-2d10ff4f-db6c-11e9-a83a-000c29886e3e	32 GiB	2	no	no		LOW		up - attached on 70.0.87.120	no
534644031852659169	pvc-492cc607-db62-11e9-a83a-000c29886e3e	2 GiB	2	no	no		LOW		up - attached on 70.0.87.118	no
416426097321211582	pvc-506f787d-db6c-11e9-a83a-000c29886e3e	32 GiB	2	no	no		LOW		up - attached on 70.0.87.118	no
235013471273806575	pvc-556adfd1-db62-11e9-a83a-000c29886e3e	2 GiB	2	no	no		LOW		up - attached on 70.0.87.118	no
143480503226632164	pvc-6dca0c3b-db62-11e9-a83a-000c29886e3e	2 GiB	2	no	no		LOW		up - attached on 70.0.87.82	no

pxctl volume inspect pvc-2d10ff4f-db6c-11e9-a83a-000c29886e3e

Volume	:  865481696383214454
	Name            	 :  pvc-2d10ff4f-db6c-11e9-a83a-000c29886e3e
	Size            	 :  32 GiB
	Format          	 :  ext4
	HA              	 :  2
	IO Priority     	 :  LOW
	Creation time   	 :  Sep 20 06:02:07 UTC 2019
	Shared          	 :  no
	Status          	 :  up
	State           	 :  Attached: fe471f15-d91c-4f94-900e-fdb2c8379541 (70.0.87.120)
	Device Path     	 :  /dev/pxd/pxd865481696383214454
	Labels          	 :  repl=2,volume.beta.kubernetes.io/storage-class=portworx-sc,namespace=default,pvc=volsolr-solr-0
	Reads           	 :  55
	Reads MS        	 :  24
	Bytes Read      	 :  1110016
	Writes          	 :  27
	Writes MS       	 :  241
	Bytes Written   	 :  249856
	IOs in progress 	 :  0
	Bytes used      	 :  4.8 MiB
	Replica sets on nodes:
		Set 0
		  Node 		 : 70.0.87.120 (Pool 2)
		  Node 		 : 70.0.87.118 (Pool 2)
	Replication Status	 :  Up
	Volume consumers	 : 
		- Name           : solr-0 (c49ce09c-db91-11e9-a83a-000c29886e3e) (Pod)
		  Namespace      : default
		  Running on     : ravi-blr-dev-dour-shoulder-3
		  Controlled by  : solr (StatefulSet)

Failover - Replacing lost stateful replicas

Scenario-1: Node down

We can see below the respective nodes on which solr pods are running and the Stateful Sets.

kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name

NODE                           NAME
ravi-blr-dev-dour-shoulder-3   solr-0
ravi-blr-dev-dour-shoulder-2   solr-1
ravi-blr-dev-dour-shoulder-2   zk-0
ravi-blr-dev-dour-shoulder-2   zk-1
ravi-blr-dev-dour-shoulder-1   zk-2

Check the stateful set status

kubectl get sts

NAME     READY   AGE
solr     2/2     5h19m
zk       3/3     6h30m

Now lets bring down the node which is hosting solr-0.

kubectl drain ravi-blr-dev-dour-shoulder-3 --ignore-daemonsets --delete-local-data --force

Inspect the Stateful Sets and the pods and observe that solr StatefulSet is ¹⁄₂ and solr-0 has moved to node-1.

kubectl get sts

NAME     READY   AGE
solr     1/2     5h20m
zk       3/3     6h31m

output NAME READY AGE solr ²⁄₂ 5h21m zk ³⁄₃ 6h32m

text kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name

output NODE NAME ravi-blr-dev-dour-shoulder-1 solr-0 ravi-blr-dev-dour-shoulder-2 solr-1 ravi-blr-dev-dour-shoulder-2 zk-0 ravi-blr-dev-dour-shoulder-2 zk-1 ravi-blr-dev-dour-shoulder-1 zk-2

Now lets bring back the node and observe the pod placements

text kubectl uncordon ravi-blr-dev-dour-shoulder-3

text kubectl get pod -o=custom-columns=NODE:.spec.nodeName,NAME:.metadata.name

output NODE NAME ravi-blr-dev-dour-shoulder-1 solr-0 ravi-blr-dev-dour-shoulder-2 solr-1 ravi-blr-dev-dour-shoulder-2 zk-0 ravi-blr-dev-dour-shoulder-2 zk-1 ravi-blr-dev-dour-shoulder-1 zk-2

### Scenario-2: Pod down
Now lets kill solr-0 and observe that solr-0 will reinitialize on its original node-3

text kubectl delete pod solr-0

text kubectl get pods -o wide

output NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES solr-0 ¹⁄₁ Running 0 2m17s 10.233.121.52 ravi-blr-dev-dour-shoulder-3 solr-1 ¹⁄₁ Running 0 5h28m 10.233.127.65 ravi-blr-dev-dour-shoulder-2 zk-0 ¹⁄₁ Running 0 62m 10.233.127.66 ravi-blr-dev-dour-shoulder-2 zk-1 ¹⁄₁ Running 0 6h39m 10.233.127.64 ravi-blr-dev-dour-shoulder-2 zk-2 ¹⁄₁ Running 0 6h39m 10.233.76.16 ravi-blr-dev-dour-shoulder-1 ```