Release Notes and Announcements
- Release Notes
- Announcements
- Release Notes
Product Introduction
Purchase Guide
- Purchase Instructions
- Purchase a TKE General Cluster
- Purchasing Native Nodes
- Purchasing a Super Node
Getting Started
Cluster Configuration
- General Cluster Overview
- Cluster Management
- Network Management
- Storage Management
- Node Management
- Images
- GPU Resource Management
- Remote Terminals
Application Configuration
- Workload Management
- Service and Configuration Management
- Component and Application Management
- Auto Scaling
- Container Login Methods
Observability Configuration
- Ops Observability
- Cost Insights and Optimization
Scheduler Configuration
- Scheduling Component Overview
- Resource Utilization Optimization Scheduling
- Business Priority Assurance Scheduling
- QoS Awareness Scheduling
Security and Stability
- TKE Security Group Settings
- Identity Authentication and Authorization
- Control Plane Security
- Application Security
- Cluster Security Enhancement Capability
Multi-cluster Management
- Planned Upgrade
- Backup Center
Cloud Native Service Guide
- Cloud Service for etcd
- TMP
- TKE Serverless Cluster Guide
- TKE Registered Cluster Guide
Use Cases
- Cluster
- Serverless Cluster
- Scheduling
- Security
- Service Deployment
- Network
- Release
- Logs
- Monitoring
- OPS
- Terraform
- DevOps
- Auto Scaling
- Containerization
- Cost Management
- Hybrid Cloud
- AI
Troubleshooting
API Documentation
- History
- Introduction
- API Category
- Making API Requests
- Elastic Cluster APIs
- Resource Reserved Coupon APIs
- Cluster APIs
- Third-party Node APIs
- Relevant APIs for Addon
- Node Pool APIs
- Network APIs
- Node APIs
- Node Pool APIs
- TKE Edge Cluster APIs
- Cloud Native Monitoring APIs
- Scaling group APIs
- Super Node APIs
- Other APIs
- Data Types
- Error Codes
- TKE API 2022-05-01
FAQs
- TKE Standard Cluster
- TKE Serverless Cluster
- About OPS
- Hidden Danger Handling
- About Services
- Image Repositories
- About Remote Terminals
- Event FAQs
- Resource Management
Service Agreement
- TKE Service Level Agreement
- TKE Serverless Service Level Agreement
Contact Us
Glossary

NodeProblemDetectorPlus (Node Monitoring)

Baixar

Modo Foco

Tamanho da Fonte

Última atualização: 2026-10-23 15:27:44

Traduzido e Verificado por IA

Component Introduction
For component details, see NodeProblemDetectorPlus Description.
Changelog
Release Date
Version Number
Change Content
Limitations and Impacts
2026-06-12
1.1.5
1. Added Hygon DCU fault detection: supports detecting DCU device anomalies through two methods, hycu-smi active polling and XID/SXID log parsing, covering 12 NodeConditions, including device loss, high temperature, uncorrectable ECC errors, PCIe link loss, and HySwitch switch chip anomalies.
2. Optimize containerd runtime detection: improve the containerd health check logic to enhance detection accuracy.
3. Optimize network device detection: use route -n instead of the original method to avoid delays caused by DNS queries.
4. Fixed the FrequentDockerRestart detection logic and the kubelet healthchecker judgment issue.
The Hygon DCU detection feature requires the hycu-smi tool to be installed on nodes. Nodes without the hycu-smi tool are not affected, and DCU-related conditions will not be registered. Other changes do not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-04-20
1.1.4
1. GPU XID detection completion: warningXid adds 10 error codes, XID 13, 31, 43, 45, 63, 68, 94, 110, 120, and 137, aligning with the latest NVIDIA XID classification standard. When these error codes are hit, a permanent GPUXIDWarningError Condition is triggered.
2. Added SRAM Aggregate ECC error detection: prioritizes the GetSramEccErrorStatus API (Ampere+ architecture), covering all SRAM ECC metrics output by nvidia-smi (Uncorrectable Parity, SEC-DED, Aggregate Sources, and so on). For architectures before Ampere, it falls back to the GetMemoryErrorCounter method. Added the GPUEccModeError Condition to detect when ECC mode is disabled.
3. Automatic recovery of non-permanent GPU Conditions: when recoverable alarms such as NvlinkError, TempError, and PowerError disappear in subsequent check cycles, the Condition is automatically restored from True to False.
4. StuckProcess monitoring fix: fixes the issue where the StuckProcess Condition repeatedly flips between True/False, including incorrectly returning OK after a timeout process is killed by SIGKILL, two rules sharing a checkpoint file and overwriting each other, and uncleaned checkpoints causing false PID recycling alarms when no D-state processes exist.
5. StuckProcess checkpoint anti-false-cleanup: adds a consecutive empty scan threshold (default 2 times) to prevent premature checkpoint cleanup when a single /proc scan is incomplete under IO pressure.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-03-26
1.1.3
1. Enhanced GPU loss detection: after GetUUID, adds a GetTemperature secondary probe for the partial GPU loss scenario. When multiple NVML APIs return GPU_IS_LOST, the system aggregates the results and determines a permanent DeviceLost error. The Power sliding window calculates thresholds based on the actual number of samples rather than assuming 1 sample/second.
2. ECC detection completion: adds SRAM (L2 Cache, Register File) uncorrectable error detection to address the issue of SRAM errors being missed when only DRAM is checked. Adds the GPUEccModeError condition to detect when ECC mode is disabled.
3. Comprehensive StuckProcess monitoring fix: fixes 9 defects, including PID reuse false alarms, /proc/PID/stat parsing misalignment, cgroup v2 incompatibility, and hardcoded docker inspect on containerd nodes.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-03-10
1.1.2
1. Add a retry mechanism for log file discovery to avoid startup failures caused by temporary unavailability.
2. Add a fallback mechanism for candidate log paths to automatically adapt to different Linux distributions.
3. Improve error handling and log output to facilitate troubleshooting.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-01-20
1.1.1
Problem fixes:
1. Fix the issue of inaccurate detection of abnormal GPU disconnections.
2. Fix the issue of inaccurate log time format matching in systemd_ops detection.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-01-13
1.1.0
Major feature changes:
1. Added GPU hardware monitoring: supports comprehensive detection of temperature, power, NVLink, ECC errors, row remapping, retired pages, PCIe, and more.
2. Added XID error monitoring: supports fatal/warning/application-level XID classification detection.
3. Added CRI interface health check: detects the health status of the container runtime interface.
4. Added NTP service monitoring: supports both ntpd and chronyd NTP services.
5. Fine-grained monitor control: supports enabling/disabling each monitor individually.
6. Configuration structure optimization: refactor values.yaml, group configurations by monitoring type, and add custom metric configurations.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2025-09-05
1.0.6
Fixed the issue of abnormal reporting of FrequentKubeletRestart/FrequentDockerRestart/FrequentContainerdRestart events to the node.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2025-08-19
1.0.5
Enhanced NPD's capability to directly report events to the corresponding Pod object when Pod OOM is detected.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2024-12-05
1.0.4
Upgrade the base image to TencentOS Server 4, fix Ubuntu 18.08 vulnerabilities, and improve image security.
Enhanced monitoring capabilities: added Metrics API support to provide Prometheus-format monitoring metrics for easy integration into monitoring systems.
Service check optimization: fix the Docker and Containerd service status check logic to eliminate redundant warning messages caused by misjudgment.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2023-07-26
1.0.3
Kernel compatibility fix: Resolved the issue of abnormal OOM (Out Of Memory) event information retrieval in the Linux 5.4 kernel environment.
This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.

Ajuda e Suporte

Esta página foi útil?

Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.

comentários

tencent cloud

Tencent Kubernetes Engine

NodeProblemDetectorPlus (Node Monitoring)

Component Introduction

Changelog

Ajuda e Suporte

Release Date	Version Number	Change Content	Limitations and Impacts
2026-06-12	1.1.5	1. Added Hygon DCU fault detection: supports detecting DCU device anomalies through two methods, hycu-smi active polling and XID/SXID log parsing, covering 12 NodeConditions, including device loss, high temperature, uncorrectable ECC errors, PCIe link loss, and HySwitch switch chip anomalies. 2. Optimize containerd runtime detection: improve the containerd health check logic to enhance detection accuracy. 3. Optimize network device detection: use `route -n` instead of the original method to avoid delays caused by DNS queries. 4. Fixed the FrequentDockerRestart detection logic and the kubelet healthchecker judgment issue.	The Hygon DCU detection feature requires the hycu-smi tool to be installed on nodes. Nodes without the hycu-smi tool are not affected, and DCU-related conditions will not be registered. Other changes do not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-04-20	1.1.4	1. GPU XID detection completion: warningXid adds 10 error codes, XID 13, 31, 43, 45, 63, 68, 94, 110, 120, and 137, aligning with the latest NVIDIA XID classification standard. When these error codes are hit, a permanent GPUXIDWarningError Condition is triggered. 2. Added SRAM Aggregate ECC error detection: prioritizes the GetSramEccErrorStatus API (Ampere+ architecture), covering all SRAM ECC metrics output by nvidia-smi (Uncorrectable Parity, SEC-DED, Aggregate Sources, and so on). For architectures before Ampere, it falls back to the GetMemoryErrorCounter method. Added the GPUEccModeError Condition to detect when ECC mode is disabled. 3. Automatic recovery of non-permanent GPU Conditions: when recoverable alarms such as NvlinkError, TempError, and PowerError disappear in subsequent check cycles, the Condition is automatically restored from True to False. 4. StuckProcess monitoring fix: fixes the issue where the StuckProcess Condition repeatedly flips between True/False, including incorrectly returning OK after a timeout process is killed by SIGKILL, two rules sharing a checkpoint file and overwriting each other, and uncleaned checkpoints causing false PID recycling alarms when no D-state processes exist. 5. StuckProcess checkpoint anti-false-cleanup: adds a consecutive empty scan threshold (default 2 times) to prevent premature checkpoint cleanup when a single /proc scan is incomplete under IO pressure.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-03-26	1.1.3	1. Enhanced GPU loss detection: after GetUUID, adds a GetTemperature secondary probe for the partial GPU loss scenario. When multiple NVML APIs return GPU_IS_LOST, the system aggregates the results and determines a permanent DeviceLost error. The Power sliding window calculates thresholds based on the actual number of samples rather than assuming 1 sample/second. 2. ECC detection completion: adds SRAM (L2 Cache, Register File) uncorrectable error detection to address the issue of SRAM errors being missed when only DRAM is checked. Adds the GPUEccModeError condition to detect when ECC mode is disabled. 3. Comprehensive StuckProcess monitoring fix: fixes 9 defects, including PID reuse false alarms, /proc/PID/stat parsing misalignment, cgroup v2 incompatibility, and hardcoded docker inspect on containerd nodes.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-03-10	1.1.2	1. Add a retry mechanism for log file discovery to avoid startup failures caused by temporary unavailability. 2. Add a fallback mechanism for candidate log paths to automatically adapt to different Linux distributions. 3. Improve error handling and log output to facilitate troubleshooting.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-01-20	1.1.1	Problem fixes: 1. Fix the issue of inaccurate detection of abnormal GPU disconnections. 2. Fix the issue of inaccurate log time format matching in `systemd_ops` detection.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2026-01-13	1.1.0	Major feature changes: 1. Added GPU hardware monitoring: supports comprehensive detection of temperature, power, NVLink, ECC errors, row remapping, retired pages, PCIe, and more. 2. Added XID error monitoring: supports fatal/warning/application-level XID classification detection. 3. Added CRI interface health check: detects the health status of the container runtime interface. 4. Added NTP service monitoring: supports both ntpd and chronyd NTP services. 5. Fine-grained monitor control: supports enabling/disabling each monitor individually. 6. Configuration structure optimization: refactor values.yaml, group configurations by monitoring type, and add custom metric configurations.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2025-09-05	1.0.6	Fixed the issue of abnormal reporting of FrequentKubeletRestart/FrequentDockerRestart/FrequentContainerdRestart events to the node.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2025-08-19	1.0.5	Enhanced NPD's capability to directly report events to the corresponding Pod object when Pod OOM is detected.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2024-12-05	1.0.4	Upgrade the base image to TencentOS Server 4, fix Ubuntu 18.08 vulnerabilities, and improve image security. Enhanced monitoring capabilities: added Metrics API support to provide Prometheus-format monitoring metrics for easy integration into monitoring systems. Service check optimization: fix the Docker and Containerd service status check logic to eliminate redundant warning messages caused by misjudgment.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.
2023-07-26	1.0.3	Kernel compatibility fix: Resolved the issue of abnormal OOM (Out Of Memory) event information retrieval in the Linux 5.4 kernel environment.	This upgrade will not affect existing services. Components may be unavailable during the upgrade. It is recommended to upgrade during off-peak hours.