tencent cloud

Tencent Kubernetes Engine

DokumentasiTencent Kubernetes EngineCluster ConfigurationGPU Resource ManagementGPU Fault Detection and Self-Healing

GPU Fault Detection and Self-Healing

Unduh
Mode fokus
Ukuran font
Terakhir diperbarui: 2026-10-23 15:49:20
Diterjemahkan oleh AI

Scenarios

During the operation of a GPU cluster, GPU hardware may experience various failures, such as XID errors, ECC memory errors, overheating, or device loss. If these failures are not detected and handled promptly, they can lead to training job failures, data corruption, or even cluster instability.
TKE provides a comprehensive GPU fault detection and self-healing solution. It uses the Node Problem Detector (NPD) component to detect GPU exceptions in real time and integrates with the Node-Healing component to automatically perform self-healing operations such as isolation, draining, and restarting. This helps you quickly restore GPU nodes to their normal operating state and ensures the stability of AI training and inference tasks. TKE supports fault detection and self-healing for two mainstream GPU models: NVIDIA and Hygon DCU.

Prerequisites

A TKE cluster has been created and contains GPU nodes. For details, see Create a Cluster.
The cluster's Kubernetes version must be >= 1.18.
The fault self-healing feature works only on native nodes.

Feature Overview

NVIDIA GPU Fault Detection Capability

TKE supports the detection of the following 12 NVIDIA GPU fault types:
Fault Type
Description
GPUDeviceLost
The GPU device is lost and cannot be recognized by the system.
GPUDoubleBitEccError
GPU double-bit ECC error, an uncorrectable memory error
GPUHighSingleBitEccError
GPU single-bit ECC error exceeds the threshold.
GPUInfoROMCorrupted
GPU InfoROM corruption
GPUNvlinkError
NVLink communication error
GPUPCIeError
PCIe interface abnormality
GPUPendingRetiredPages
GPU Pending Retired Pages
GPUPowerError
GPU power issue
GPURemappingRowsError
GPU memory row remapping error
GPUTempError
GPU temperature is too high
GPUXIDFatalError
GPU fatal XID error
GPUXIDWarningError
GPU warning XID error

Hygon DCU GPU Fault Detection Capability

TKE supports the detection of the following 5 Hygon DCU GPU fault types:
Fault Type
Description
HygonGPUDeviceLost
The GPU device is lost, and driver or PCIe issues cause the device to be unrecognizable.
HygonGPUDriverError
The GPU driver is abnormal and can be restored after a restart.
HygonGPUTempError
GPU temperature is too high, and heat dissipation needs to be checked.
HygonGPUEccError
GPU memory ECC error, which may indicate hardware damage.
HygonGPUPowerError
GPU power is abnormal, and the power supply needs to be checked.

Self-Healing Policy Overview

The system adopts different self-healing policies based on the fault type:
NVIDIA GPU Self-Healing Policy
Self-Healing Policy
Execution Steps
Applicable Fault Types
Complete Self-Healing Process
Isolate → Drain → Restart → Unseal
GPUDeviceLost,GPUNvlinkError,GPUPCIeError,GPUPendingRetiredPages,GPUXIDWarningError
Isolation and Drainage Only
Isolate → Drain
GPUDoubleBitEccError,GPUHighSingleBitEccError,GPUInfoROMCorrupted,GPUPowerError,GPURemappingRowsError,GPUTempError,GPUXIDFatalError
Hygon DCU GPU Self-Healing Policy
Self-Healing Policy
Execution Steps
Applicable Fault Types
Complete Self-Healing Process
Isolate → Drain → Restart → Unseal
HygonGPUDeviceLost,HygonGPUDriverError
Isolation and Drainage Only
Isolate → Drain
HygonGPUTempError,HygonGPUEccError,HygonGPUPowerError
Note:
Complete Self-Healing Process: Applies to software or recoverable hardware issues that can be automatically restored after a restart.
Isolation and Draining Only: This applies to severe hardware faults that require manual inspection and repair. The system only performs isolation and draining operations to prevent new tasks from being scheduled to the faulty node.

Operation Steps

Step 1: Installing the Node Problem Detector Component

Node Problem Detector (NPD) is the foundational component for node fault detection. It is responsible for monitoring node status in real time and reporting exceptions.
1. Log in to the TKE console and select Cluster in the left sidebar.
2. In the cluster list, click the target cluster ID to go to the cluster details page.
3. In the left sidebar, choose Component Management. On the displayed page, click Create.
4. On the Create Component Management page, select NodeProblemDetectorPlus.
5. Click Complete.
After installation, the NPD component runs as a DaemonSet on every node in the cluster. It automatically detects GPU-related faults and updates the node Condition.

Step 2: Installing the Node-Healing Component

The Node-Healing component is responsible for performing self-healing operations based on detected faults.
1. On the component management page, click Create.
2. On the Create Component Management page, select Node-Healing.
3. Click Complete.

Step 3: Configuring the GPU Node Self-Healing Policy

After installing the component, you need to create a NodeHealer resource to define the self-healing policy for GPU nodes.

NVIDIA GPU Node Configuration Example

1. Create a NodeHealer configuration file named gpu-node-healer.yaml.
apiVersion: node.healing.tke.cloud.tencent.com/v1
kind: NodeHealer
metadata:
  name: gpu-cluster-healer
spec:
  nodeSelector:
    matchLabels:
      node-healing.tke.cloud.tencent.com/belong-to: gpu-cluster-healer
      cloud.tencent.com/provider: tencentcloud
      node.tke.cloud.tencent.com/accelerator-type: gpu
  machineHealingTemplate:
    name: "default-healing-template"
    kind: "MachineHealingTemplate"
    apiVersion: "node.healing.tke.cloud.tencent.com/v1"
  unhealthyConditions:
    - type: GPUXIDFatalError      # GPU fatal error
      enableCheck: true
      enableHealing: true
      durationTime: "1m"
      priority: 100
    - type: GPUDeviceLost         # GPU device lost
      enableCheck: true
      enableHealing: true
      durationTime: "5m"
      priority: 80
  nodeHealingPolicy:
    maxHealConcurrency: 3
    minNodeCoolingMinutes: 30
Note:
The preceding example configures only two typical GPU fault types. You can add more fault types based on your actual requirements. For the complete list of fault types, see GPU Fault Detection Capabilities.
2. Run the following command to apply the configuration.
kubectl apply -f gpu-node-healer.yaml

Hygon DCU GPU Node Configuration Example

1. Create a NodeHealer configuration file named hygon-gpu-node-healer.yaml.
apiVersion: node.healing.tke.cloud.tencent.com/v1
kind: NodeHealer
metadata:
  name: hygon-gpu-cluster-healer
spec:
  nodeSelector:
    matchLabels:
      node-healing.tke.cloud.tencent.com/belong-to: hygon-gpu-cluster-healer
      cloud.tencent.com/provider: tencentcloud
      node.tke.cloud.tencent.com/accelerator-type: dcu
  machineHealingTemplate:
    name: "default-healing-template"
    kind: "MachineHealingTemplate"
    apiVersion: "node.healing.tke.cloud.tencent.com/v1"
  unhealthyConditions:
    - type: HygonGPUDeviceLost    # Hygon GPU device lost (complete self-healing)
      enableCheck: true
      enableHealing: true
      durationTime: "5m"
      priority: 85
    - type: HygonGPUDriverError   # Hygon GPU driver exception (complete self-healing)
      enableCheck: true
      enableHealing: true
      durationTime: "5m"
      priority: 80
    - type: HygonGPUTempError     # Hygon GPU temperature too high (isolation and draining only)
      enableCheck: true
      enableHealing: true
      durationTime: "5m"
      priority: 85
    - type: HygonGPUEccError      # Hygon GPU ECC error (isolation and draining only)
      enableCheck: true
      enableHealing: true
      durationTime: "3m"
      priority: 90
    - type: HygonGPUPowerError    # Hygon GPU power exception (isolation and draining only)
      enableCheck: true
      enableHealing: true
      durationTime: "3m"
      priority: 90
  nodeHealingPolicy:
    maxHealConcurrency: 3
    minNodeCoolingMinutes: 30
2. Run the following command to apply the configuration.
kubectl apply -f hygon-gpu-node-healer.yaml

Step 4: Adding Tags to GPU Nodes

NodeHealer selects nodes that require self-healing based on the Tags defined in nodeSelector. You need to add Tags to the target nodes that match those in nodeSelector.matchLabels in the configuration file. Using the following example configuration as a reference, run the following command to add Tags to the GPU nodes:
kubectl label node <node-name> node-healing.tke.cloud.tencent.com/belong-to=gpu-cluster-healer
Or add Tags to all native GPU nodes in batches:
kubectl label node \\
  -l node.tke.cloud.tencent.com/accelerator-type=gpu \\
  node-healing.tke.cloud.tencent.com/belong-to=gpu-cluster-healer
Note:
The Tag key-value pairs can be customized, as long as the node Tags match those in the nodeSelector.matchLabels section of the NodeHealer configuration.

Verifying the Configuration

Checking NodeHealer Status

kubectl get nodehealer
Expected output:
NAME                 AGE
gpu-cluster-healer   1h

Checking Node Conditions

kubectl get node <gpu-node-name> -o jsonpath='{.status.conditions}' | jq
If NPD detects a GPU fault, it displays the corresponding fault type in the node Condition.

Viewing Self-Healing Tasks

kubectl get healingtask -n kube-system
When self-healing is triggered, the system automatically creates a HealingTask to record the task execution status.

Handling GPU Hardware Failures

For severe GPU hardware failures (such as double-bit ECC errors, InfoROM corruption, and so on), the system only performs isolation and draining operations and does not automatically restart. You need to manually handle them according to the following procedure:
1. Confirm the fault: Check the node status in the TKE console, or run the kubectl describe node command to view the node Condition and events, thereby identifying the specific fault type.
2. Check GPU status: Log in to the node and use the nvidia-smi -q command (for NVIDIA GPUs) or the hy-smi -q command (for Hygon DCU GPUs) to check the detailed GPU status.
3. Contact for repair: If a hardware fault is confirmed, submit a ticket to contact the Tencent Cloud after-sales team for inspection and replacement.
4. Restore the node: After the fault is repaired, use the kubectl uncordon command to uncordon the node and remove the self-healing-related Taint.

Configuration Instructions

unhealthyConditions Parameter Description

Define the fault types to be monitored and their corresponding self-healing actions:
Parameter
Type
Description
type
string
Fault type. For the complete list, see Node-Healing Supported Fault Types.
enableCheck
bool
Whether to enable detection.
enableHealing
bool
Whether to enable self-healing.
durationTime
string
The duration after which self-healing is triggered when a fault persists. Format examples: 1m, 5m.
priority
int
Priority. A higher value indicates a higher priority (0-100).
Recommended Configuration for durationTime:
Fault Type
Recommended Value
Description
Critical hardware failures (such as ECC errors, XID fatal errors, and so on)
1m
Respond quickly to avoid data corruption.
General hardware failures (such as device loss and NVLink errors)
5m
Observe appropriately to avoid misjudgment.
Recoverable issues (such as warning-level errors)
10m
Extend observation, as it may recover automatically.

nodeHealingPolicy Parameter Description

Rate limiting policy to prevent cluster cascading failures during large-scale faults:
Parameter
Description
Recommended Value
maxHealNodesPerDay
Maximum number of self-healing nodes per day
10-20
maxHealNodesPerHour
Maximum number of self-healing nodes per hour
3-5
maxHealConcurrency
Maximum concurrent self-healing quantity
2-3
minNodeCoolingMinutes
Minimum interval between two self-healing operations on the same node (minutes)
30-60

FAQs

Why Is GPU Failure Detection Not Working?

1. Confirm that the NPD component is correctly installed and running:
kubectl get ds -n kube-system | grep node-problem-detector
2. Confirm that the NPD Pod on the node is in a normal state:
kubectl get pod -n kube-system -l app=node-problem-detector -o wide

Why Is the Self-Healing Task Not Triggered?

1. Confirm that the node has been tagged with the labels that match the NodeHealer configuration:
kubectl get node <node-name> --show-labels
2. Confirm that the node is a native node (with the cloud.tencent.com/provider=tencentcloud Tag).
3. Check whether the fault duration exceeds the durationTime configuration.
4. Check whether it is being limited by the rate limiting policy:
kubectl describe nodehealer gpu-cluster-healer

How to Disable Self-Healing for a Node

Remove the Tags on the node that match the NodeHealer configuration, for example:
kubectl label node <node-name> node-healing.tke.cloud.tencent.com/belong-to-

References

Node-Healing Description
NodeProblemDetectorPlus Description
GPU Monitoring Metrics Acquisition
Topik sebelumnya: Custom Image SpecificationsTopik selanjutnya: qGPU Overview

Bantuan dan Dukungan

Apakah halaman ini membantu?

masukan

Search
    Enter keywords to search documentation