huaweicloud/huawei-cloud-dws-io-diag

Huawei Cloud DWS I/O Overload Diagnosis Skill

Overview

This skill is dedicated to DWS cluster I/O overload root cause diagnosis. When a cluster triggers an I/O usage too high alarm, it automatically collects metric data, analyzes root causes (customer-side/system-side) via a three-stage decision tree, and outputs a standardized diagnosis report.

Architecture: KooCLI (hcloud) → DWS Autopilot API → Cluster monitoring metrics; MCP Server (dws_autopilot) → Fallback channel for the same API

Applicable Scenarios:

• DWS cluster I/O usage too high alarm
• Disk I/O load anomaly investigation
• User-initiated I/O diagnosis request

Typical Use Cases:

• "My DWS cluster I/O usage is very high, help diagnose"
• "Received an I/O alarm, cluster ID is xxx, help analyze the cause"
• "DWS cluster disk util is maxed out, see what's causing it"

Skill Scope & Boundaries:

• Service boundary: This skill only supports Huawei Cloud DWS (GaussDB(DWS)) data warehouse service. It does NOT apply to non-DWS services such as ECS, RDS, CCE, or others.
• Problem-type boundary: This skill only diagnoses DWS cluster I/O overload / I/O usage too high / disk I/O load anomaly (I/O overload, I/O anomaly, high await, high util, etc.). It does NOT handle CPU overload, memory high, or other non-I/O problems (refer to the dedicated CPU/memory diagnosis skills).
• Operation boundary: This skill is read-only diagnosis. It does NOT execute any management or maintenance operations such as cluster restart, scale-out/in, node replacement, kill SQL, or parameter modification.

Important Rules: All diagnosis conclusions must come from actual tool return results. Fabricating or assuming values is prohibited. Output only contains the diagnosis report; adding remediation suggestions, outputting SQL optimization statements, and using emoji are prohibited.

Background Knowledge: RAID architecture, I/O key metric thresholds, three major I/O scenario indicator features, and important principles are documented in I/O Background Knowledge. Must read before diagnosis.

Prerequisites

1. CLI Requirements

• KooCLI (hcloud) >= 3.2.0
• Verify installation: hcloud version
• If not installed or version too low, see CLI Installation Guide

2. MCP Server Configuration (Fallback)

• When KooCLI is unavailable, use MCP Server as an alternative
• See MCP Installation Guide

3. Authentication Configuration

• Valid Huawei Cloud credentials (AK/SK mode or IAM Token)
• Security Rules:
  • Never expose AK/SK values in conversations or commands
  • Never ask users to input AK/SK directly in conversation
  • Only use hcloud configure list to check credential status

4. IAM Permission Requirements

• dws:clusters:get, dws:clusters:list
• dws:metricData:get, dws:hostOverview:get
• See IAM Policies

KooCLI Command Format Standard

Command Format

# Query metric data
hcloud DWS ListMetricsData --cli-region=<region> --cluster_id=<id> --metric_name=<name> --project_id=<pid> --offset=0 --limit=200 --from=<from_ts> --to=<to_ts>

# Query host information
hcloud DWS ListHostOverview --cli-region=<region> --project_id=<pid> --cluster_id=<cid> --offset=0 --limit=200

Tool Selection Strategy

Choose between KooCLI and MCP Server, preferring KooCLI. Step 0 checks hcloud availability:

• hcloud available (version >= 3.2.0) → Use KooCLI command line calls for subsequent steps
• hcloud unavailable (not installed or version < 3.2.0) → Use MCP Server tool calls for subsequent steps

Fallback and Termination Strategy:

• After selecting hcloud, if the first call returns NETWORK_ERROR (connection timeout, network unreachable, etc.), automatically fall back to MCP Server and use MCP mode for all subsequent steps
• If hcloud returns NETWORK_ERROR and MCP Server is also unavailable (not configured or returns authentication/connection error), terminate this skill and output: KooCLI network unavailable and MCP Server connection failed. Please check KooCLI network configuration or DWS Autopilot MCP Server configuration and retry
• After selecting MCP Server, if the first call returns an authentication error (e.g., 401 Unauthorized), do not fall back to hcloud. Terminate this skill directly and output: MCP Server authentication failed. Please check DWS Autopilot MCP Server authentication configuration and retry
• hcloud call failures that are not NETWORK_ERROR (e.g., parameter errors, insufficient permissions) do not trigger fallback; follow existing retry logic

Once a tool is selected, use it throughout without switching (except for NETWORK_ERROR fallback). If a call fails, retry once (maximum 2 attempts). If still failing, mark the metric as "unavailable" and continue to the next step. When all metric queries fail, generate the diagnosis report directly.

Parameter Mapping

Common Parameter	hcloud Parameter	MCP Parameter
Region	--cli-region	(built into MCP connection)
Project ID	--project_id	project_id
Cluster ID	--cluster_id	cluster_id
Metric Name	--metric_name	metric_name
Start Time	--from	from_ts
End Time	--to	to_ts
Pagination Offset	--offset	offset
Pagination Limit	--limit	limit
Sort Field	(not supported)	order_by
Sort Direction	(not supported)	sort_by

MCP Tools

Tool Name	Purpose	Key Parameters
dws_autopilot_get_clusters	Query cluster list	project_id, cluster_id, limit, offset
dws_autopilot_get_hosts	Query host information	project_id, cluster_id, limit, offset
dws_autopilot_get_metric	Query metric data	project_id, cluster_id, metric_name, from_ts, to_ts, limit, offset, order_by, sort_by

metric_data Parameter Notes: metric_data does not support filtering by instance_name (no such parameter); query returns full cluster data and must be filtered by inst_name field for target instances; metric_data does not support period parameter (sampling period is automatically determined by the platform).

Available metric_name: IOStat, CpuStat, cpu_io_diagnose_detail, business_concurrency, business_query_monitor, business_thread_wait, bussiness_conflict_lock

Time Protocol: from_ts/to_ts must use Unix millisecond timestamps; all times are in UTC timezone; recommended time window: from 5 minutes before alarm time to alarm time (from_ts = first_alarm_time - 300000).

Return Format: Success {"code": 0, "data": [...]}; Failure {"code": -1, "message": "error description"}. On failure, retry once; if still failing, use degradation path and mark as "unavailable" in the report.

Key Fields per Metric:

• IOStat: ctime, host_id, disk_name, io_data{util, await, r_await, w_await, r_s, w_s, read_iops, write_iops, rMB_s, wMB_s, read_throughput, write_throughput, avgrq_sz, avgqu_sz} → core I/O metrics
• CpuStat: ctime, host_id, cpu_data{usr, sys, idle, iowait} → iowait > 30% indicates significant I/O pressure
• cpu_io_diagnose_detail: ctime, host_id, active_queries[{query, query_id, userName, cpu_rate, state, inst_name, duration_ms}], io_stats{read_iops, write_iops, read_throughput, write_throughput} → I/O diagnosis details + active queries
• business_concurrency: ctime, concurrency_value, active_connections → business concurrency
• business_query_monitor: ctime, query, query_id, username, duration_ms, host_id → historical real-time queries
• business_thread_wait: ctime, wait_event, count, username, host_id → wait events (wait wal sync, WALWriteLock, etc.)
• bussiness_conflict_lock: ctime, lock_type, blocker, waiter, username, host_id → lock conflicts

For query differences per step, see Metric Reference.

Pagination Specification

All tool calls (MCP and hcloud) must use paginated queries to prevent single responses from being too large and exceeding token limits.

Pagination Rules:

• Use limit=200 uniformly (do not use values larger than 200, except the declared exception below)
• Explicit exception: cpu_io_diagnose_detail uses limit=800 pagination (due to its largest data volume, multi-node × multi-instance × multi-timepoint × multi-query); single page of 200 may not be enough
• First page offset=0; if returned data count = limit, then offset+=limit and continue querying
• Repeat until returned data count < limit, then merge all paginated data
• When merging, concatenate all page data arrays into a complete dataset

MCP Call Example (using IOStat):

Page 1: dws_autopilot_get_metric(project_id, cluster_id, metric_name="IOStat", from_ts, to_ts, limit=200, offset=0)
If returned data length = 200:
Page 2: dws_autopilot_get_metric(project_id, cluster_id, metric_name="IOStat", from_ts, to_ts, limit=200, offset=200)
If returned data length < 200: Stop pagination, merge page 1 + page 2 data

hcloud Call Example (using IOStat):

Page 1: hcloud DWS ListMetricsData --cli-region=<region> --cluster_id=<id> --metric_name=IOStat --project_id=<pid> --offset=0 --limit=200 --from=<from_ts> --to=<to_ts>
If returned data count = 200:
Page 2: hcloud DWS ListMetricsData --cli-region=<region> --cluster_id=<id> --metric_name=IOStat --project_id=<pid> --offset=200 --limit=200 --from=<from_ts> --to=<to_ts>
If returned data count < 200: Stop pagination, merge data

Workflow

Before diagnosis, create an execution plan based on Steps 0-7, then execute sequentially. For tool selection strategy, see the "KooCLI Command Format Standard" section; subsequent steps will not repeat this. All MCP tool calls follow the "Pagination Specification" section; subsequent steps will not repeat pagination details.

Step 0: Environment Detection & Parameter Resolution

Required Parameter Validation: Before proceeding, verify that at least one of cluster_id or cluster_name is provided (both may be provided, but at least one is mandatory). If both are missing, prompt the user to provide a valid cluster_id or cluster_name and terminate the diagnosis until it is provided.

Cluster Existence Validation: Use the cluster list query (hcloud DWS ListClusters in hcloud mode, or dws_autopilot_get_clusters in MCP mode) to confirm the target cluster exists:

• If the query returns the error codes DWS.13026 or DWS.13032 (cluster not found), or the returned cluster list does not contain the specified cluster_id/cluster_name → the cluster does not exist. Terminate the diagnosis and prompt: The specified DWS cluster does not exist. Please check the cluster_id/cluster_name and retry.
• If the query succeeds and the cluster exists → continue with the following steps.

Parameter Resolution: If project_id or region_id is not provided by the user, read from MCP Server config file conf/dws_config.yaml:

• When project_id is missing: Execute python -c "import yaml; c=yaml.safe_load(open('conf/dws_config.yaml')); print(c.get('project_id',''))" to obtain it
• When region_id is missing: Execute python -c "import yaml; c=yaml.safe_load(open('conf/dws_config.yaml')); print(c.get('region_id',''))" to obtain it
• If the corresponding field in the config file is also empty, prompt the user to provide the parameter

Execute hcloud version; version >= 3.2.0 → tool_mode=hcloud, otherwise tool_mode=mcp.

hcloud Network Availability Probe: If tool_mode=hcloud, execute a lightweight API call (e.g., hcloud DWS ListClusters --cli-region=<region> --project_id=<project_id>) to verify network connectivity:

• Returns normal or business error (e.g., insufficient permissions, parameter error) → Network available, keep tool_mode=hcloud
• Returns NETWORK_ERROR → Network unavailable, fall back to tool_mode=mcp
• After fallback, if MCP is also unavailable → Terminate execution, prompt user to check KooCLI network configuration or MCP Server configuration

Step 1: Query IOStat

Call metric query with metric_name="IOStat", time window: from_ts=first_alarm_time - 300000, to_ts=first_alarm_time. In MCP mode, use limit=200 paginated query.

Autopilot Unavailable Determination: Returns 50201/RDS.9999 error → Skip Steps 1-6, mark all metrics as "unavailable", proceed directly to Step 7.

Parsing: Group by host_id and disk_name, find the latest I/O data for each disk on each node. Extract util, await, r_await, w_await, r_s, w_s, read_iops, write_iops, rMB_s, wMB_s, read_throughput, write_throughput, avgrq_sz, avgqu_sz for each disk.

I/O Overload Determination: Whether any disk has util > 90% or await > 100 or throughput/IOPS reaching upper limit.

I/O Imbalance Determination: Whether the deviation of util or await between nodes > 20%.

TOP Node Filtering: Find the two nodes with the highest I/O load.

Phenomenon Distribution Determination: Whether a node has single disk I/O high or all disks I/O high (used for Step 4 scenario routing).

High IO Time Points: From the IOStat time series data, find all ctime points where any disk util > 90% on the top 2 IO nodes. Collect these ctime values as high_io_ctimes (list of millisecond timestamps). Also record the ctime with the highest util as peak_io_ctime. If no ctime has util > 90%, high_io_ctimes is empty and peak_io_ctime is the ctime with the highest util value.

Output: io_by_host_disk, io_by_host, max_io_hosts, problem_host_id, problem_host_util, problem_host_await, problem_host_avgrq_sz, problem_host_avgqu_sz, cluster_avg_util, is_imbalanced, io_deviation, high_io_nodes, single_disk_high_nodes, all_disk_high_nodes, high_io_ctimes, peak_io_ctime

Step 2: Get Host Names

Call host information query (MCP mode limit=200 paginated), build host_id → {ho

…

[truncated for translation]