From 8d43d435c174e6765d9f8569c6423bb835baa582 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Fri, 15 Aug 2025 01:21:56 -0700 Subject: [PATCH 1/4] docs: Correct EC pipeline calculation and limits Discovered and corrected the documentation for how the number of EC pipelines is calculated. The previous analysis was incorrect. - `ErasureCoding.md` is updated to describe the two new properties `ozone.scm.ec.pipeline.minimum` and `ozone.scm.ec.pipeline.per.volume.factor` and the `max()` logic used to determine the target number of pipelines. - `ProductionDeployment.md` is updated to reference the correct and existing configuration property for tuning EC pipelines. Change-Id: I393dc60d8745da2b2bb7899530665a108956446d --- .../content/feature/multi-raft-support.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/hadoop-hdds/docs/content/feature/multi-raft-support.md b/hadoop-hdds/docs/content/feature/multi-raft-support.md index c0cd1b4c0433..3f8d6858cca8 100644 --- a/hadoop-hdds/docs/content/feature/multi-raft-support.md +++ b/hadoop-hdds/docs/content/feature/multi-raft-support.md @@ -69,6 +69,51 @@ Ratis handles concurrent logs per node. This property is effective only when the previous property is set to 0. The value of this property must be greater than 0. +### Calculating Ratis Pipeline Limits + +The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively. SCM will create pipelines until the most restrictive limit is met. + +1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)** + * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines across the entire cluster. This acts as a final cap on the total number of pipelines. + * **Default Value**: `0` (which means no global limit is enforced by default). + +2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)** + * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for every datanode. This is one of two ways to calculate a cluster-wide target. + * **Default Value**: `2` + * **Calculation**: If this is set, the target is `( * ) / 3`. + +3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)** + * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`. It calculates a dynamic limit for each datanode based on its available metadata disks. + * **Default Value**: `2` + * **Calculation**: The limit for each datanode is `( * )`. The total cluster-wide target is the sum of all individual datanode limits, divided by 3. + +#### How Limits are Applied + +SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the **Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The **lowest value** is used as the final target for the number of open pipelines. + +**Example (Dynamic Limit):** + +Consider a cluster with **10 healthy datanodes**. +* **8 datanodes** have 4 metadata disks each. +* **2 datanodes** have 2 metadata disks each. + +And the configuration is: +* `ozone.scm.ratis.pipeline.limit` = **30** (A global cap is set) +* `ozone.scm.datanode.pipeline.limit` = **0** (Use dynamic calculation) +* `ozone.scm.pipeline.per.metadata.disk` = **2** (Default) + +**Calculation Steps:** +1. Calculate the limit for the first group of datanodes: `8 datanodes * (2 pipelines/disk * 4 disks/datanode) = 64 pipelines` +2. Calculate the limit for the second group of datanodes: `2 datanodes * (2 pipelines/disk * 2 disks/datanode) = 8 pipelines` +3. Calculate the total raw target from the dynamic limit: `(64 + 8) / 3 = 24` +4. Compare with the global limit: `min(24, 30) = 24` + +SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE Ratis pipelines. + +**Production Recommendation:** + +For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for `ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the actual number of pipelines aligns with your configured targets. + ## How to Use 1. Configure Datanode metadata directories: ```xml From c4deea787c9a5a6f5e7c40e48c0be90d4c59b5ec Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Sat, 3 Jan 2026 16:48:35 -0800 Subject: [PATCH 2/4] Limit to 120 characters Change-Id: I0ec667c21155436eb6a0654782b43b48636f75d5 --- .../content/feature/multi-raft-support.md | 27 ++++++++++++++----- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/hadoop-hdds/docs/content/feature/multi-raft-support.md b/hadoop-hdds/docs/content/feature/multi-raft-support.md index 3f8d6858cca8..bd5ef7e38981 100644 --- a/hadoop-hdds/docs/content/feature/multi-raft-support.md +++ b/hadoop-hdds/docs/content/feature/multi-raft-support.md @@ -71,25 +71,34 @@ Ratis handles concurrent logs per node. ### Calculating Ratis Pipeline Limits -The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively. SCM will create pipelines until the most restrictive limit is met. +The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum +number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively. +SCM will create pipelines until the most restrictive limit is met. 1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)** - * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines across the entire cluster. This acts as a final cap on the total number of pipelines. + * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines + across the entire cluster. This acts as a final cap on the total number of pipelines. * **Default Value**: `0` (which means no global limit is enforced by default). 2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)** - * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for every datanode. This is one of two ways to calculate a cluster-wide target. + * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for + every datanode. This is one of two ways to calculate a cluster-wide target. * **Default Value**: `2` * **Calculation**: If this is set, the target is `( * ) / 3`. 3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)** - * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`. It calculates a dynamic limit for each datanode based on its available metadata disks. + * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`. + It calculates a dynamic limit for each datanode based on its available metadata disks. * **Default Value**: `2` - * **Calculation**: The limit for each datanode is `( * )`. The total cluster-wide target is the sum of all individual datanode limits, divided by 3. + * **Calculation**: The limit for each datanode is + `( * )`. + The total cluster-wide target is the sum of all individual datanode limits, divided by 3. #### How Limits are Applied -SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the **Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The **lowest value** is used as the final target for the number of open pipelines. +SCM first calculates a target number of pipelines based on either the **Datanode-level Fixed Limit** or the +**Datanode-level Dynamic Limit**. It then compares this calculated target to the **Cluster-wide Limit**. The +**lowest value** is used as the final target for the number of open pipelines. **Example (Dynamic Limit):** @@ -112,7 +121,11 @@ SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE **Production Recommendation:** -For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for `ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the actual number of pipelines aligns with your configured targets. +For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is +recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the +global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for +`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the +actual number of pipelines aligns with your configured targets. ## How to Use 1. Configure Datanode metadata directories: From f8060d994c1159c6de4e595196429864a05757fc Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Tue, 6 Jan 2026 13:38:38 -0800 Subject: [PATCH 3/4] Update text for monitoring number of pipelines. Change-Id: I2537905761cf45d23cdb3701b2f0c94e7ff2485a --- hadoop-hdds/docs/content/feature/multi-raft-support.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hadoop-hdds/docs/content/feature/multi-raft-support.md b/hadoop-hdds/docs/content/feature/multi-raft-support.md index bd5ef7e38981..e7ddf052eb08 100644 --- a/hadoop-hdds/docs/content/feature/multi-raft-support.md +++ b/hadoop-hdds/docs/content/feature/multi-raft-support.md @@ -124,8 +124,8 @@ SCM will attempt to create and maintain approximately **24** open, FACTOR_THREE For most production deployments, using the dynamic per-disk limit (`ozone.scm.datanode.pipeline.limit=0`) is recommended, as it allows the cluster to scale pipeline capacity naturally with its resources. You can use the global limit (`ozone.scm.ratis.pipeline.limit`) as a safety cap if needed. A good starting value for -`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the `NumOpenPipelines` metric in SCM to see if the -actual number of pipelines aligns with your configured targets. +`ozone.scm.pipeline.per.metadata.disk` is **2**. Monitor the section **Pipeline Statistics** in SCM web UI, or run +the command `ozone admin pipeline list` to see if the actual number of pipelines aligns with your configured targets. ## How to Use 1. Configure Datanode metadata directories: From b7eefa22ad22109c3d88562a7de7b813d7da6b06 Mon Sep 17 00:00:00 2001 From: Wei-Chiu Chuang Date: Fri, 16 Jan 2026 17:06:36 -0800 Subject: [PATCH 4/4] Update based on Nicholas's comments. Change-Id: Ieff1a5ffe67c4eed32a7c61c655c19ba9f307d26 --- .../content/feature/multi-raft-support.md | 21 +++++++++---------- 1 file changed, 10 insertions(+), 11 deletions(-) diff --git a/hadoop-hdds/docs/content/feature/multi-raft-support.md b/hadoop-hdds/docs/content/feature/multi-raft-support.md index e7ddf052eb08..d44cd2f75075 100644 --- a/hadoop-hdds/docs/content/feature/multi-raft-support.md +++ b/hadoop-hdds/docs/content/feature/multi-raft-support.md @@ -71,28 +71,27 @@ Ratis handles concurrent logs per node. ### Calculating Ratis Pipeline Limits -The target number of open, FACTOR_THREE Ratis pipelines is controlled by three properties that define the maximum -number of pipelines in the cluster at a cluster-wide level, datanode level, and metadata disk level, respectively. -SCM will create pipelines until the most restrictive limit is met. +ReplicationFactor.THREE is controlled by three configuration properties that limit the +number of pipelines in the cluster at a cluster-wide level and a datanode level, respectively. +The number of pipelines created by SCM is restricted by these limits. 1. **Cluster-wide Limit (`ozone.scm.ratis.pipeline.limit`)** - * **Description**: An absolute, global limit for the total number of open, FACTOR_THREE Ratis pipelines + * **Description**: An absolute, global limit for the total number of open Ratis pipelines across the entire cluster. This acts as a final cap on the total number of pipelines. - * **Default Value**: `0` (which means no global limit is enforced by default). + * **Default Value**: `0` (which means no global limit by default). 2. **Datanode-level Fixed Limit (`ozone.scm.datanode.pipeline.limit`)** * **Description**: When set to a positive number, this property defines a fixed maximum number of pipelines for - every datanode. This is one of two ways to calculate a cluster-wide target. + every datanode. * **Default Value**: `2` - * **Calculation**: If this is set, the target is `( * ) / 3`. + * **Cluster-wide Limit Calculation**: If this property is set, + the number of pipelines in the cluster is in addition limited by + `( * ) / 3`. 3. **Datanode-level Dynamic Limit (`ozone.scm.pipeline.per.metadata.disk`)** - * **Description**: This property is used only when `ozone.scm.datanode.pipeline.limit` is explicitly set to `0`. + * **Description**: This property takes effect when `ozone.scm.datanode.pipeline.limit` is not set to a positive number. It calculates a dynamic limit for each datanode based on its available metadata disks. * **Default Value**: `2` - * **Calculation**: The limit for each datanode is - `( * )`. - The total cluster-wide target is the sum of all individual datanode limits, divided by 3. #### How Limits are Applied