tke.cloud.tencent.com/qgpu.version.cat /proc/qgpu/version.rmmod qgpu). Then, delete the qgpu-manager Pod on that node. The automatically recreated Pod will install the new qGPU module upon startup.cat /proc/qgpu/version on the node to confirm that the qGPU module is installed successfully.Release Date | Version Number | Change Content | Limitations and Impact |
2026-08-13 | v1.1.4 | Supports GPU NUMA-aware scheduling to avoid cross-NUMA performance loss. Disabled by default. The upgrade has no impact. | After an upgrade to this version is performed, if NUMA-aware scheduling is enabled, there is a risk of GPU contention when existing pods using qgpu are present on the node. The upgrade changes the update policy of kube-system/qgpu-manager daemonset to OnDelete. You need to evict services node by node and restart the qgpu-manager Pod to complete the upgrade. |
2026-06-05 | v1.1.3 | Fixed the issue where the qgpu scheduler overwrites fields in the new version of PodSpec when the qgpu scheduler updates Pods. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2026-03-09 | v1.1.2 | Fix the spelling error in the name of the container_assigned_card label (container_name) for the qGPU metric. The gpu_card_index_mapping metric is added to query the mapping between the nvidia-smi index and the device minor index number. Fix the issue where the qgpu-manager garbage collection might delete the device of a newly launched pod that shares the same qGPU ID. Upgrade the qGPU kernel module to v3.0.7, which supports the 5880 and 5090D graphics cards. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2026-01-13 | v1.1.1 | The qGPU kernel module is upgraded to v3.0.5, supporting 550+ drivers, CUDA 12.3+, and the use of two shared cards by containers. qgpu-manager adds logic to check qGPU computing resources. If the resource quantity becomes zero due to certain reasons, it exits and restarts the registration process. Ignore the empty qGPU-related environment variables that come with the image to avoid startup failure. The collection of GPU Persistence Mode information is disabled by default to avoid OOM issues caused by driver bugs. The issue where qGPU metrics were not collected due to a logic error in qGPU node judgment is fixed. The GPU information allocated to Pods and containers is exposed. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2024-06-26 | v1.0.23 | Fixed a timing issue during garbage collection that could cause newly created qGPU devices to be immediately reclaimed and deleted, preventing business Pods from using GPUs normally. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2024-03-19 | v1.0.22 | Optimized the handling logic for Pods in the Completed state and for Update events to prevent backlog from causing slow cache release and affecting scheduling. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2024-03-01 | v1.0.21 | It supports drivers for minor versions such as 470/515 and higher versions such as 525/535. It supports the H800 graphics card. Mixed online and offline deployment is not supported. CUDA 12.0+ is recommended, as early versions may cause slow qGPU device initialization. qgpu-manager adds logic to check qGPU resources. If the resource quantity becomes zero due to certain reasons, it exits and restarts the registration process. The qgpu-exporter component is automatically installed to support basic GPU monitoring. During the upgrade, retain the CPU and memory resource settings for qgpu-scheduler. (The amount of memory occupied by the qgpu scheduler is related to the cluster scale and needs to be configured by the customer based on the actual situation.) | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2024-01-24 | v1.0.20 | Further converged the permissions of qgpu-manager, retaining only the minimum necessary permissions. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-12-18 | v1.0.19 | Fix the issue where the qgpu scheduler panics due to the required qGPU resource request not being set. Before scheduling, check whether the pod resources meet the constraints. If not, an error is returned, causing the scheduling to fail. This prevents issues such as panic and isolation failure that may arise from this situation. Modify the qgpu-manager updateStrategy and set maxUnavailable to 10% to accelerate the update process. Add resource limits to the qgpu-installer init container to ensure the qgpu-manager Pod is Guaranteed, preventing it from being evicted first when node resources are under high pressure. Fix the issue where the qgpu scheduler panics due to GPU quantity changes during node migration. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-11-28 | v1.0.18 | Converged the permissions of qgpu-manager to avoid privilege escalation risks. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-08-30 | v1.0.17 | Improved the handling logic during Pod deletion to speed up deletion and reduce the occurrence rate of scheduling failures for new Pods when a Deployment with the Recreate policy is updated. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-08-22 | v1.0.16 | Fix the issue where a panic occurs when a Pod is forcibly cast to a Pod after it is deleted and becomes a DeletedFinalStateUnknown object. After the label `tke.cloud.tencent.com/qgpu-mode=base` is set on the node, install the base version of the qGPU module. No kernel parameter modification is required. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-06-20 | v1.0.15 | Add the Tag `tke.cloud.tencent.com/qgpu.memory-unit`, which indicates the memory allocation unit, with a value of 1G or 128M. The qGPU hook is compatible with the nvidia solution. When the qGPU kernel module installation fails, the log prompts you to check whether the GPU type is supported. Supports the new TencentOS-3.1 5.4 kernel. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-05-09 | v1.0.14 | It supports native nodes, public images, 16 GPUs, and a memory unit of 128MB. Compatible with the nvidia 515 (515.65.01) driver. The qGPU scheduler adds the /metrics and /caches APIs. Fix the issue that causes qGPU device creation or termination to fail and leads to inconsistent scheduler cache information. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2023-02-08 | v1.0.13 | When a Pod with zero qGPU resources is processed, directly returns all nodes or scores all as zero to avoid excessive time consumption from processing a large number of nodes, which blocks the scheduling of qGPU pods. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-12-20 | v1.0.12 | Supports 16 cards. Optimized logs to avoid printing interference messages. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-10-28 | v1.0.11 | When a Pod is deleted, directly delete the metrics related to that Pod in batches, rather than deleting them container by container, to avoid the issue of metrics not being deleted under certain circumstances. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-10-24 | v1.0.10 | Fix the issue where, on some multi-GPU machines, the GPU card sequence number does not match the minor device number, causing qGPU to mount the wrong GPU and preventing services from running normally. In private deployments, installing the base edition run package that does not support mixed deployment is allowed. In this case, kernel parameter modification and node restart are not required. Fix the issue where the system prompts a missing qgpu-core even though qgpu-memory is not set. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-09-29 | v1.0.9 | Fix the regression issue introduced by the crd. This issue caused the qGPU scheduler to start scheduling new Pods after a restart without synchronizing Pod allocation information, resulting in inconsistencies. Before updating Pod container metrics, clean up the relevant data to avoid dirty data. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-09-21 | v1.0.8 | Add the /metrics API to expose GPU information allocated to Pod containers (node_gpu_core_request, node_gpu_memory_request_bytes; type: Gauge, Tags: "node", "gpu", "namespace", "pod", "container"). After a Pod is deleted, the corresponding container metrics are removed. To prevent interference with the creation of new qGPU devices, qGPU devices left behind for various reasons are cleaned up as a fallback during the garbage collection (gc) performed every minute. Fix the issue where creating a qGPU device may fail when a Pod with the same name is immediately created and scheduled to the same node after the previous Pod is deleted. It supports installing and using qGPU on native nodes. (qGPU on native nodes does not support the online-offline mixed deployment feature.) | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-09-06 | v1.0.7 | Fixed the issue where the qGPU scheduler did not clean up information of deleted nodes. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-08-31 | v1.0.6 | It supports accessing the apiserver using the node's kubeconfig. (In this case, you cannot enable the crd parameter.) It supports installing qGPU using the TencentOS 2.4(TK4) public image. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-07-20 | v1.0.5 | It supports installing and using qGPU with public images. It supports disabling the feature for labeling nodes (--enable-label=false). | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-06-08 | v1.0.3 | The qgpu-manager image is updated to tkeimages/elastic-gpu-agent:v1.0.2. The qgpu-scheduler image is updated to tkeimages/elastic-gpu-scheduler:v1.0.2. It supports managing GPU resources using GPU CRDs. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
2022-04-21 | v1.0.2 | Update the qgpu-manager image version to support automatically setting the GPU driver version and other information on the node. The clusterrole qgpu-manager is updated to add operational permissions for Nodes. | This upgrade does not affect existing services. During the upgrade process, the component may be unavailable. It is recommended to upgrade during off-peak hours. |
Esta página foi útil?
Você também pode entrar em contato com a Equipe de vendas ou Enviar um tíquete em caso de ajuda.
comentários