Etcd for Portworx

Portworx requires a key-value database such as etcd for configuring storage. A highly available clustered etcd with persistent storage is preferred.

This page list various approaches for installing an external ETCD cluster and provides recommendations on best practices.

Requirements

For production Portworx clusters, Portworx, Inc. recommends the following configuration of an etcd cluster:

1. Etcd Version > 3.1.x
2. Minimum 3 nodes
3. Minimum 8G of memory dedicated to each etcd node.
4. Each Etcd node in the etcd cluster backed with storage disks (minimum 100GB)

More detailed set of hardware requirements as recommended by etcd can be found here.

Setup

You can use one of the following methods to setup an etcd cluster

Setup an ETCD cluster using the official documentation

Follow this detailed step by step process provided by etcd to setup a brand new multi-node cluster.

Setup an ETCD cluster using Ansible Playbook

Follow this ansible playbook to install a 3 node etcd cluster.

Tuning Etcd

Etcd provides multiple knobs to fine tune the cluster based on your needs. Portworx, Inc. recommends fine tuning the following three settings.

Compaction

etcd keeps an exact history of its keyspace, this history should be periodically compacted to avoid performance degradation and eventual storage space exhaustion. Regular compaction ensures that the memory usage of the etcd process is under check. The keyspace can be compacted automatically with etcd’s time windowed history retention policy, or manually with etcd.

Portworx, Inc. recommends keeping history for last 3 hours. While setting up etcd you can specify the retention policy in the following way:

etcd --auto-compaction-retention=3

Database Size (Space Quota)

The space quota in etcd ensures the cluster operates in a reliable fashion. Without a space quota, etcd may suffer from poor performance if the keyspace grows excessively large, or it may simply run out of storage space, leading to unpredictable cluster behavior.

Portworx, Inc. recommends setting the space quota to max value of 8Gi. While setting up etcd you can specify the space quota in the following way:

etcd --quota-backend-bytes=$((8*1024*1024*1024))

Snapshots

Etcd provides a command to take snapshots of its keyspace which can be used to restore the etcd cluster in case of a complete disaster. Portworx, Inc. recommends running the following command as a part of a cron job which will take periodic snapshots

ETCDCTL_API=3 etcdctl --endpoints="<comma-separated-etcd-url>" snapshot save </path/to/snapshot-file> --command-timeout=60s

You can run the above command either on the etcd nodes or on a separate node where you would want to store these etcd snapshots.

For a more detailed setup, maintenance and tuning information refer the following etcd reference docs.

• Maintenance
• Tuning
• Recovery from failure

Securing with certificates in Kubernetes

SSL certificates for etcd can be stored as Kubernetes secrets. Three files are required - in this example, the CA certificate is kvdb-ca.crt, the etcd certificate kvdb.crt and the etcd key kvdb.key. These files should be copied to a directory on the Kubernetes control plane node (kvdb-secrets). Next, create a secret from these files:

kubectl -n kube-system create secret generic px-kvdb-auth --from-file=kvdb-secrets/

secret/px-kvdb-auth created

kubectl -n kube-system describe secret px-kvdb-auth

Name:         px-kvdb-auth
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
kvdb-ca.crt:      1679 bytes
kvdb.crt:  1680 bytes
kvdb.key:  414  bytes

Use the Portworx spec generator in PX-Central, selecting “Certificate Auth” under the etcd section, ensuring the filenames match those specified.