Manage the number of storage nodes in a cluster
This page explains how Portworx automatically manages the number of storage nodes in a cluster. It also explains the need for this behavior and how you can override it.
Why limit the number of storage nodes?
The value of the
spec.cloudStorage.maxStorageNodesPerZone parameter gates the number of storage nodes that can exist in a single failure domain. The following example describes an issue that you can encounter when this parameter is not set.
Consider a cluster with a single storage node per zone where
maxStorageNodesPerZone is not set:
If your cloud provider decides to recycle the N1 node, then N1-new is the new node that replaces the node N1. It is often observed that the N1-new node starts before the N1 node is terminated, and that the N1 node may not have relinquished its disk. Thus, Portworx cannot reuse those disks in the new node N1-new, which results in creation of a new storage node with a new cloud drive, as shown in the following diagram:
In this case, if the value of
maxStorageNodesPerZone had seen set to 1, then N1-new node would not have created the new cloud drive Px disk-4.
How does Portworx manage storage nodes?
Portworx automatically manages the number of storage nodes in a new cluster1. This is achieved by choosing a value for the
maxStorageNodesPerZone parameter2 as follows and avoids the issue explained above:
|Disaggregated install||(Total nodes marked as
|Converged install||(Total nodes)4 / (Total zones)3|
Here are some examples of how Portworx chooses the default value of
maxStorageNodesPerZone using the above formulas:
|Number of zones||Total nodes or Total Nodes marked storage||Default value of
How do I override the value of maxStorageNodesPerZone?
Depending on whether you are performing a fresh install or modifying an existing installation, proceed to one of the following sections.
A fresh installation
You can use any one of the following methods to set a value for
maxStorageNodesPerZone so that Portworx does not set this value automatically:
While generating a Portworx spec from PX-Central, perform the following steps:
- On the Basic page, select the latest Portworx version from the Portworx Version dropdown and click Next.
- On the Storage page, select a cloud provider and then specify a number in the Max storage nodes per availability zone option.
Add the following to your generated Portworx spec:
spec: cloudStorage: maxStorageNodesPerZone: <custom-value>
maxStorageNodesPerZone, which keeps the number of storage nodes unbounded in a cluster. Every node that joins the cluster will be a storage node. This value must be used with great caution to avoid the situation mentioned previously.
An existing installation
Add more storage nodes by increasing the value of
maxStorageNodesPerZone in the StorageCluster spec.
How is a zone determined?
For different environments, the following table shows how zones are determined:
|Public clouds, such as AWS, Azure, Google Cloud, or IBM Cloud||Detected by Portworx using cloud providers APIs.|
|VMware vSphere or VMware Tanzu||
|Pure FlashArray||Single zone. No zoning labels are supported.|
- When an existing cluster upgrades to any newer version of Portworx,
maxStorageNodesPerZoneis updated using the same formula mentioned above. Portworx can perform this update only if
maxStorageNodesPerZoneis not specified for the existing cluster. [return]
- If the user account has permissions to access Auto Scaling Groups (ASG) information, Portworx honors the ASG settings. If the minimum value in ASG is lower than
maxStorageNodesPerZone, Portworx creates storage nodes equal to the minimum value. [return]
- If the number of nodes is unevenly distributed, the zone with the smallest number of nodes is chosen. For example, if there are 10 nodes across 3 zones, the value 3 is selected. [return]
- For the total number of nodes, Kubernetes master and control plane nodes are excluded if they are part of the cluster. [return]
- For Portworx versions 2.12.0 or earlier, the supported label is