1.36 changes since 1.34
Major Updates
Version 1.35
The InPlacePodVerticalScaling feature gate has been upgraded to GA, allowing you to adjust container CPU and memory resources in place without rebuilding Pods. Vertical scaling tools can reduce restarts and service interruptions caused by resource adjustments.
The PodObservedGenerationTracking feature gate has been upgraded to GA. It helps controllers determine whether the status corresponds to the latest spec through the Pod's .status.observedGeneration and .status.conditions[].observedGeneration fields. This capability prevents controllers from processing updated Pod configurations based on stale status.
The JobManagedBy feature gate has been upgraded to GA, allowing Jobs to be managed by external controllers through spec.managedBy. Third-party batch processing systems can take over Job coordination and lifecycle management while continuing to use the Kubernetes Job API.
The PreferSameZone and PreferSameNode values of the Service trafficDistribution field have been upgraded to GA. This capability can prioritize routing traffic to endpoints in the same AZ or on the same node, reducing cross-AZ or cross-node communication.
The SupplementalGroupsPolicy feature gate has been upgraded to GA, allowing finer control over how supplementary groups for containers are calculated. With the Strict policy, supplementary groups are no longer merged with group memberships in the container image's /etc/group file, preventing supplementary groups that are not explicitly declared from entering the container.
The strict-cpu-reservation policy option of the CPU Manager static policy has been upgraded to GA, allowing stricter isolation of reserved CPUs. When this policy option is enabled, CPU Manager excludes reserved CPUs from the shared defaultCPUSet, preventing non-exclusive workloads from continuing to use these CPUs.
The max-allowable-numa-nodes policy option of Topology Manager has been upgraded to GA, allowing you to adjust the maximum number of NUMA nodes supported by Topology Manager. The default limit is 8, and you can raise it on servers with more NUMA nodes through this option.
The ControlPlaneKubeletLocalMode feature gate of kubeadm has been upgraded to GA and locked to enabled, making the kubelet on control plane nodes connect to the local API Server by default. As a result, control plane kubelets can reduce their dependency on external load balancers or remote API Server endpoints.
The CSIServiceAccountTokenSecrets feature gate has been upgraded to beta and enabled by default, allowing CSI drivers to receive service account tokens through a dedicated secrets field. After a CSI driver opts in to this method by setting CSIDriver.spec.serviceAccountTokenInSecrets: true, tokens are no longer passed through volume context, reducing the risk of tokens appearing in logs and other outputs.
The KubeletEnsureSecretPulledImages feature gate has been upgraded to beta and enabled by default, allowing kubelet to re-verify access credentials for existing images on the node based on the imagePullCredentialsVerificationPolicy. The default NeverVerifyPreloadedImages policy does not re-verify images preloaded outside of kubelet. Only when AlwaysVerify is used are all existing images on the node verified.
The ImageVolume feature is enabled by default in the beta stage, allowing Pods to mount OCI images as volumes. Applications can deliver configurations, models, or other read-only artifacts through the image distribution model without packaging these contents into container images.
The MutableCSINodeAllocatableCount feature is enabled by default in the beta stage, allowing CSI drivers to dynamically update the number of allocatable volumes on nodes. The scheduler can use fresher capacity information, reducing cases where stateful Pods fail to start for a long time due to outdated node volume quotas.
The ContainerRestartRules feature gate has been upgraded to beta and enabled by default, allowing you to configure exit-code-based restart rules for individual containers. Applications can decide whether to restart a container based on different failure causes, rather than relying entirely on the unified Pod-level restart policy.
The EnvFiles feature gate has been upgraded to beta and enabled by default, allowing containers to load environment variables from files in emptyDir volumes through FileKeyRef. Environment variable files use a restricted POSIX shell syntax, and variable values must be enclosed in single quotes.
The HostnameOverride feature gate has been upgraded to beta and enabled by default, allowing the Pod hostname to be specified through the hostnameOverride field in PodSpec. This field supports the use of a complete RFC 1123 DNS subdomain, making it suitable for workloads that depend on fixed hostnames or fully qualified domain names.
The HPAConfigurableTolerance feature gate has been upgraded to beta and enabled by default, allowing you to configure the scale-out and scale-in tolerances of HPA separately. Users can control the sensitivity of HPA to metric changes based on business fluctuation characteristics, reducing unnecessary replica flapping.
The MaxUnavailableStatefulSet feature gate has been upgraded to beta and enabled by default, allowing you to control the number of Pods that can be unavailable simultaneously during StatefulSet rolling updates. The default value was reverted to disabled in v1.36 due to a regression issue in Parallel Pod management mode.
The PodTopologyLabelsAdmission feature gate has been upgraded to beta and enabled by default, allowing Pods to automatically inherit the region and AZ labels of the nodes they run on. Workloads can directly read these labels to identify their own topology location without querying Node objects separately.
The DeploymentReplicaSetTerminatingReplicas feature gate has been upgraded to beta and enabled by default, allowing Deployments and ReplicaSets to track replicas that are still terminating through .status.terminatingReplicas. Users and controllers can more accurately understand the actual replica occupancy during rolling updates.
The KubeletCrashLoopBackOffMax feature gate has been upgraded to beta and enabled by default, allowing you to configure the node-level maximum restart delay for CrashLoopBackOff. Users can shorten the maximum wait time based on workload recovery characteristics to speed up retries for failed containers.
Pod Certificates has been upgraded to beta, allowing short-lived certificates to be provided to Pods through projected volumes with automatic certificate rotation. This capability still requires the certificates.k8s.io/v1beta1 API and the PodCertificateRequest feature gate, which is disabled by default, to be enabled.
The WatchListClient feature is enabled by default in the beta stage, allowing client-go to initialize its local cache through streaming watch-list requests. This approach reduces the API Server memory and network overhead caused by full LIST requests in large clusters.
A disabled-by-default alpha feature, MutablePVNodeAffinity, has been added. When MutablePVNodeAffinity is enabled, the .spec.nodeAffinity field of a PersistentVolume can be modified after the volume is created. Storage administrators can adjust the accessible scope of a volume after node or topology changes without recreating the PersistentVolume.
In-place resource adjustment for Pods only allows modifications to CPU and memory. Attempts to add other resource types in a resize are rejected by the API Server.
With the AuthorizePodWebsocketUpgradeCreatePermission feature gate enabled by default, kube-apiserver uniformly requires the create permission for SPDY and WebSocket requests to pods/exec, pods/attach, and pods/portforward. Related custom Roles and ClusterRoles are required to include this verb.
The kubelet no longer accepts the --pod-infra-container-image command-line flag, and Pod sandbox images are managed by the container runtime. kubeadm attempts to clean up this parameter in kubeadm-flags.env during upgrades. In non-kubeadm clusters, a leftover of this parameter prevents the kubelet from starting.
The StrictCostEnforcementForVAP, StrictCostEnforcementForWebhooks, WaitForAllControlPlaneComponents, UserNamespacesPodSecurityStandards, SizeMemoryBackedVolumes, and ComponentSLIs feature gates have been removed, and these gate names are no longer accepted for configuration.
Feature gate dependencies are now explicitly declared and uniformly validated to prevent incomplete feature combinations from running when their dependencies are not enabled. AllAlpha=true no longer automatically enables the disabled-by-default beta features it depends on, and configurations with incomplete dependencies fail startup validation.
The scheduler has fixed an issue where Pods using CSI volumes could be scheduled to nodes without the corresponding CSI driver installed, preventing Pods from failing to mount volumes due to missing drivers on nodes.
The kube-controller-manager has fixed an issue where upgrading the control plane from 1.33 to 1.34 could trigger unexpected rolling updates for existing StatefulSets. This fix is controlled by the StatefulSetSemanticRevisionComparison feature gate, which is enabled by default.
The kubelet has fixed an issue where the main container of a restartPolicy: Never Pod could remain in the Initializing state for a long time after the startup probe of a sidecar container failed for the first time.
Static Pods can now properly use container-level restart policies and restart rules, fixing the issue where admission validation previously rejected these configurations incorrectly.
Version 1.36
The StrictIPCIDRValidation feature gate has been upgraded to beta and enabled by default. IP address and CIDR fields in Kubernetes built-in APIs no longer accept IP addresses with extra leading zeros or CIDRs whose host bits are inconsistent with the mask. Stricter validation prevents different components from producing inconsistent parsing results for ambiguous IP address and CIDR expressions.
The MutatingAdmissionPolicy feature gate has been upgraded to GA and enabled by default, providing the admissionregistration.k8s.io/v1 API for defining mutating admission policies using CEL. For mutation logic that can be expressed with CEL, you can reduce the dependency on external mutating webhook services.
The UserNamespacesSupport feature gate has been upgraded to GA, providing stable user namespace isolation for stateless Pods. The root user inside a container can be mapped to an unprivileged user on the host, reducing the privilege risk after a container escape.
The ImageVolume feature gate has been upgraded to GA, allowing Pods to mount OCI images as read-only volumes. This capability integrates read-only artifact distribution into the standard image pull and cache process, simplifying the delivery of models, tools, and static data.
The DRAAdminAccess feature gate has been upgraded to GA, providing stable administrator-level access to DRA devices. Administrators can request access to devices that are already in use for scenarios such as device diagnostics, maintenance, or driver management.
The DRAPrioritizedList feature gate has been upgraded to GA, allowing DRA devices to be selected according to a priority list. When the preferred device cannot satisfy the request, the scheduler can continue trying subsequent options, improving the adaptability of device requests in heterogeneous clusters.
The MutableCSINodeAllocatableCount feature gate has been upgraded to GA and locked to enabled, allowing CSI drivers to dynamically update the number of allocatable volumes on nodes. The scheduler can select nodes based on the latest volume capacity, reducing cases where stateful Pods get stuck due to outdated capacity information.
The KubeletPSI feature gate has been upgraded to GA, and the kubelet Summary API can now expose CPU, memory, and I/O Pressure Stall Information metrics for Linux cgroups. These metrics reflect task stalls caused by resource contention, providing a basis for node capacity planning and performance diagnostics.
The CSIServiceAccountTokenSecrets feature gate has been upgraded to GA, allowing CSI drivers to receive service account tokens through a dedicated secrets field. After a CSI driver enables CSIDriver.spec.serviceAccountTokenInSecrets, tokens are separated from the regular volume context, reducing the risk of sensitive information being exposed in logs or diagnostic outputs.
The ProcMountType feature gate has been upgraded to GA, allowing you to configure the /proc mount type for containers. Users can control the process information visible to containers based on security isolation or compatibility requirements.
The NodeLogQuery feature gate has been upgraded to GA, providing a stable capability for querying node logs through the kubelet API. Ops systems can obtain node service logs through a unified interface, reducing the need to directly log in to nodes for troubleshooting.
The ExternalServiceAccountTokenSigner feature gate has been upgraded to GA, supporting the signing of ServiceAccount tokens using an external signer. Clusters can delegate Token signing and signing key management to services outside the control plane, reducing the need for kube-apiserver to directly hold the signing private key.
The KubeletFineGrainedAuthz feature gate has been upgraded to GA, providing more fine-grained authorization control for the kubelet API. Administrators can grant access permissions by specific subresources, reducing the security risks associated with using broad kubelet API permissions.
The KubeletPodResourcesDynamicResources feature gate has been upgraded to GA and locked to enabled, and the Pod Resources API can now return devices assigned to containers through Dynamic Resource Allocation (DRA). Device management and monitoring components can obtain the dynamic resource allocation results of containers through a unified interface.
The KubeletPodResourcesGet feature gate has been upgraded to GA and locked to enabled, providing a stable Get method for the Pod Resources API. Clients can query resource allocation information by Pod and container without obtaining the complete Pod resource list on the node.
The SELinuxChangePolicy feature gate has been upgraded to GA, allowing you to configure the SELinux label change policy for Pod volumes. For volumes that support SELinux mounting, you can use mount options instead of recursively modifying file labels, reducing the startup time of large-capacity volumes.
The SELinuxMountReadWriteOncePod feature gate has been upgraded to GA, providing SELinux label support based on mount options for ReadWriteOncePod volumes. This capability avoids relabeling a large number of files within a volume one by one, improving the startup efficiency of workloads that use SELinux.
The NodeLocalCRISocket feature gate of kubeadm has been upgraded to GA and locked to enabled, managing CRI socket information in a node-local manner. Different nodes can retain their own container runtime connection configurations, better supporting clusters with inconsistent runtime configurations.
The InPlacePodLevelResourcesVerticalScaling feature gate has been upgraded to beta and enabled by default, allowing you to adjust Pod-level CPU and memory resources in place. Applications can adjust the overall resource quota without rebuilding Pods, reducing service interruptions during scaling.
The InPlacePodVerticalScalingInitContainers feature is enabled by default in beta, supporting in-place resource adjustment for running non-sidecar init containers. Initialization tasks with high resource requirements can adjust resources during runtime without recreating the entire Pod.
The ExtendWebSocketsToKubelet feature gate has been upgraded to beta and enabled by default, allowing the API Server to proxy WebSocket requests for exec, attach, and port-forward directly to kubelets that support this capability. This approach reduces the protocol conversion workload on the API Server and improves the processing efficiency of streaming connections.
The DRADeviceBindingConditions feature gate has been upgraded to beta and enabled by default, allowing device drivers to control the device binding process of Pods through binding conditions. For devices that require initialization or reconfiguration, the scheduler can wait until the devices meet the conditions before completing Pod binding.
The DRADeviceTaints feature gate has been upgraded to beta and enabled by default, allowing device taints to be declared in ResourceSlice and device allocation to be controlled through tolerations. Device drivers can prevent new Pods from using devices that are faulty, under maintenance, or unsuitable for specific workloads.
The DRAConsumableCapacity feature gate has been upgraded to beta and enabled by default, allowing the same device to be allocated multiple times and its capacity to be consumed based on requests. This capability is suitable for accelerators or other devices that can be partitioned and shared by capacity, helping improve device utilization.
The DRAExtendedResource feature gate has been upgraded to beta and enabled by default, allowing devices requested as extended resources to be managed through DRA. Workloads can continue to use the extended resource request method, while the underlying device allocation is completed by DRA.
The DRAPartitionableDevices feature gate has been upgraded to beta and enabled by default, allowing drivers to describe partitionable devices through per-device node selection and shared counters. The scheduler can account for shared resource consumption when allocating logical devices, improving the utilization of partitionable hardware.
The DRAResourceClaimGranularStatusAuthorization feature gate has been upgraded to beta and enabled by default. ResourceClaim status update permissions are subdivided into binding and driver subresources, and authorization can be scoped by driver name. When using DRA drivers, schedulers, or related controllers, you need to adapt to the new fine-grained RBAC model.
The ResourceHealthStatus feature gate has been upgraded to beta and enabled by default, and kubelet can report the health status of Device Plugin and DRA devices in the .status.containerStatuses[].allocatedResourcesStatus field of a Pod. Applications and Ops systems can identify whether devices allocated to containers are Healthy, Unhealthy, or Unknown.
The ResourceHealthStatusMessage feature gate is enabled by default in beta, adding a human-readable message returned by DRA drivers to resource health status. This information can supplement the cause of device exceptions, facilitating fault diagnosis.
The ConstrainedImpersonation feature gate has been upgraded to beta and enabled by default, providing a more fine-grained authorization model than traditional impersonation through additional authorization checks. Platform components can obtain only the impersonation permissions required for specific scenarios, reducing the risks associated with granting broad impersonate permissions.
The RestartAllContainersOnContainerExits feature gate has been upgraded to beta and enabled by default, and container restart rules can trigger the restart of all containers in a Pod. This capability is suitable for Pods where the states of multiple containers depend on each other, allowing a consistent running state to be re-established after a critical container exits.
The RelaxedServiceNameValidation feature gate has been upgraded to beta and enabled by default, and the names of newly created Services are now validated using DNS Label rules. Service names can start with a digit, providing more flexible naming rules for reusing external system identifiers or auto-generated names.
The NodeDeclaredFeatures feature gate has been upgraded to beta and enabled by default. Nodes can declare their supported capabilities through Node.status.declaredFeatures for the scheduler to match nodes, and Pod updates are validated by the NodeDeclaredFeatureValidator admission plugin. Pods that depend on specific node capabilities can avoid being scheduled to nodes that do not support those features.
The MutablePodResourcesForSuspendedJobs feature gate has been upgraded to beta and enabled by default, allowing you to modify the resource requests and limits of containers and initContainers in the template of a suspended Job that has no active Pods. Users can reset resources based on current capacity or task requirements before resuming the Job.
The MutableSchedulingDirectivesForSuspendedJobs feature gate has been upgraded to beta and enabled by default, allowing you to modify the scheduling directives of a suspended Job that has no active Pods. Users can adjust scheduling conditions such as node selection, affinity, and tolerations before resuming execution.
The MaxUnavailableStatefulSet feature gate is disabled by default in v1.36 to avoid a regression issue in StatefulSet Parallel Pod management mode.
The scheduling.k8s.io/v1alpha2 Workload and PodGroup alpha APIs are introduced, and the related GangScheduling, GenericWorkload, TopologyAwareWorkloadScheduling, WorkloadAwarePreemption, and WorkloadWithJob feature gates are disabled by default. These capabilities integrate gang scheduling, topology-aware placement, and workload-level preemption into the scheduling process, making them suitable for distributed training and batch processing tasks.
The ManifestBasedAdmissionControlConfig alpha feature has been added, allowing the API Server to load Webhook and CEL admission policies from on-disk manifests. These policies take effect immediately when the API Server starts and continue to protect critical resources when etcd is unavailable.
The memoryReservationPolicy field has been added to the MemoryQoS alpha feature, which is disabled by default. When TieredReservation is selected, the kubelet sets cgroup v2 memory.min for Guaranteed Pods and memory.low for Burstable Pods based on memory requests.
The kube-controller-manager metric volume_operation_total_errors has been renamed to volume_operation_errors_total. Monitoring dashboards and alert rules based on the old name will no longer be able to obtain data.
The kube-apiserver metric etcd_bookmark_counts has been renamed to etcd_bookmark_total. Monitoring configurations based on the old name will no longer match this metric.
kubeadm no longer automatically configures flex-volumes, including the kube-controller-manager parameters and plugin directory mounts. To continue using flex-volumes, you need to use a custom non-distroless kube-controller-manager image, pass the --flex-volume-plugin-dir flag, and mount the plugin directory through kubeadm extraVolumes.
The in-tree Portworx volume plugin and the CSIMigrationPortworx and InTreePluginPortworxUnregister feature gates have been completely removed. All Portworx volume operations are now performed through CSI.
The kubectl kuberc command has been upgraded to beta, allowing you to manage kuberc files through the view and set subcommands. Users can use a unified command to view and modify kubectl client preferences, reducing the need to manually edit configuration files.
The credential plugin allowlist in kuberc now uses the AllowlistEntry.Command field to specify commands, replacing the original AllowlistEntry.Name field.
kube-scheduler now supports parallel execution of PreBind plugins, reducing latency during the Pod binding phase. Third-party PreBind plugins must implement PreBindPreFlight, return PreBindPreFlightResult, and then be recompiled. When nil is returned, serial execution is maintained. When AllowParallel: true is returned, parallel execution is allowed.
The HonorPVReclaimPolicy, ControlPlaneKubeletLocalMode, and WatchFromStorageWithoutResourceVersion feature gates have been removed, and the deprecated SeparateCacheWatchRPC has been locked to false.
The client-go informer enables AtomicFIFO by default. Objects from a list or relist are first updated to the store as a whole, and then event handlers are invoked item by item. This change allows handlers to read cached state at a consistent resource version, but it may alter the timing of callback triggers.
The DaemonSet, Job, ReplicaSet, and StatefulSet controllers now support stale cache awareness, deferring synchronization when the previous write has not yet been observed to avoid redundant reconciliation or resource creation caused by lagging informer caches. This behavior is controlled by the StaleControllerConsistencyDaemonSet, StaleControllerConsistencyJob, StaleControllerConsistencyReplicaSet, and StaleControllerConsistencyStatefulSet feature gates, which are enabled by default.
The kubelet has fixed an issue where, in a Pod that contains a sidecar and a startup probe, a regular container could not be restarted after crashing following a restart. Affected containers might have remained at RestartCount: 0 for a long time.
When the container runtime cannot return the initContainer status, a static Pod is no longer displayed as Init:0/1 for a long time. The kubelet now updates its status correctly.
The kube-controller-manager has fixed an issue where VolumeAttachments were left behind after the CSIDriver's attachRequired field was changed from true to false, preventing invalid attachment objects from persisting.
The kubelet has fixed a race condition where errors could be lost during concurrent unmounting of Pod volumes, improving the reliability of volume unmount results.
StatefulSet now updates .status.availableReplicas in a timely manner, reducing status reporting latency and accelerating the progress of rolling updates.
The default scheduler returns Pods preempted during the PreBind phase to the backoff queue, preventing them from being deleted from the API Server.
The default scheduler returns Pods preempted during the WaitOnPermit phase to the backoff queue, preventing them from being incorrectly marked as unschedulable.
kubectl logs -f waits for the container to start when it is in the ContainerCreating or PodInitializing state, instead of returning a failure immediately.
Deprecated APIs
cgroup v1 has been deprecated since v1.35, and kubelet and kubeadm treat running on cgroup v1 as a validation error by default.
The storagemigration.k8s.io/v1alpha1 API was removed from StorageVersionMigration in v1.35 and replaced by storagemigration.k8s.io/v1beta1.
The storage.k8s.io/v1alpha1 API was removed from VolumeAttributesClass in v1.35. Related manifests and clients must use storage.k8s.io/v1.
The ipvs mode of kube-proxy is deprecated as of v1.35 and will be removed in a future release. You are advised to migrate to the nftables mode.
The PreferClose value of the Service trafficDistribution field is deprecated as of v1.35. You are advised to use PreferSameZone instead, which has clearer semantics.
The git-repo volume plugin is disabled in v1.36 and cannot be re-enabled. Related workloads must be migrated to alternatives such as initContainer with emptyDir.
The scheduling.k8s.io/v1alpha1 Workload API was removed in v1.36. Related objects and clients must be migrated to scheduling.k8s.io/v1alpha2.
The cbt.storage.k8s.io/v1alpha1 API was removed from the SnapshotMetadataService CRD of external-snapshot-metadata in v1.36. Related CRDs, objects, and clients must use cbt.storage.k8s.io/v1beta1.
The apiserver.config.k8s.io/v1alpha1 WebhookAdmissionConfiguration was removed in v1.36. Related configurations must use apiserver.config.k8s.io/v1.
The Service .spec.externalIPs field is deprecated as of v1.36, and a warning is emitted when this field is used.
Changelogs
1.34 changes since 1.32
Major Updates
Version 1.33
The JobSuccessPolicy feature gate has been upgraded to GA, allowing you to configure a success policy for an Indexed Job. The policy defines when a Job is deemed as successfully completed.
The JobBackoffLimitPerIndex feature gate has been upgraded to GA, allowing you to configure a backoff limit for each index of an Indexed Job. The limit enables more fine-grained control over failure handling.
The MatchLabelKeys and MismatchLabelKeys features have been upgraded to GA. PodAffinity and PodAntiAffinity now support these fields, allowing more flexible affinity and anti-affinity configurations based on Pod tag keys and addressing the problem of distinguishing new and old Pods during Deployment rolling updates.
The NodeInclusionPolicyInPodTopologySpread feature has been upgraded to a stable version in version 1.33, allowing more precise control over node inclusion policies in Pod topology distribution constraints.
The Service trafficDistribution field (including the PreferClose option) has been upgraded to GA, allowing the configuration of a Service traffic distribution policy to optimize traffic routing.
Recursive read-only (RRO) mounting has been upgraded to GA. This feature allows setting an RRO property on volume mounting for Pods, meaning that volumes mounted this way and all subdirectories and files within them will be set to the read-only status.
The CPUManager policy option has been upgraded to GA, providing a more flexible configuration option for CPU management policies.
The AnyVolumeDataSource feature gate has been upgraded to GA, allowing any data source to be used in PersistentVolumeClaim and expanding the flexibility of volume data sources.
The way the API server responds to List requests whose response format is negotiated as Protobuf is improved. The List response in Protobuf marshals one element at a time, greatly reducing the memory required to serve large collections. The response method can be controlled through the StreamingCollectionEncodingToProtobuf feature gate.
The ListFromCacheSnapshot feature gate (alpha) has been added, allowing apiserver to provide List requests with accurate ResourceVersion and continuation tokens from the cache and improving the performance and consistency of List operations.
The v1 Endpoints API has now been officially deprecated. It will not be deleted, but all users should replace it with the EndpointSlice API.
The following general availability feature gates have been removed: CPUManager, PDBUnhealthyPodEvictionPolicy, PersistentVolumeLastPhaseTransitionTime, AppArmor, AppArmorFields, KubeProxyDrainingTerminatingNodes, and JobPodFailurePolicy.
Support for the v1alpha1 version of ValidatingAdmissionPolicy and ValidatingAdmissionPolicyBinding API types has been removed.
The inactive service code for the authentication.k8s.io/v1alpha1 API has been removed.
The SchedulerPopFromBackoffQ feature gate (beta, enabled by default) has been added. It improves scheduling queue behavior by popping Pods from the backoffQ when the activeQ is empty, allowing possibly schedulable Pods to be processed as soon as possible.
The ProcMountType feature has been promoted to beta (enabled by default), allowing the configuration of different /proc mount types in containers.
The ReduceDefaultCrashLoopBackOffDecay feature gate has been added to decrease the initial latency and maximum cumulative latency between container restarts on a node for containers in the CrashLoopBackOff status in the cluster. The recommended values are 1s for initial latency and 60s for maximum latency.
The EndpointSlice hints field has been upgraded to GA. The beta annotation service.kubernetes.io/topology-mode is now deemed as deprecated and will not be upgraded to GA. Users are encouraged to use the spec.trafficDistribution field in the Service API.
Version 1.34
The core feature of dynamic resource allocation (DRA) has been upgraded to GA. This new stable feature uses the structured parameter style of DRA, providing more flexible device resource management capabilities.
The StructuredAuthenticationConfiguration feature gate has been upgraded to GA. The AuthenticationConfiguration type has been upgraded to apiserver.config.k8s.io/v1, providing a more structured authentication configuration mode.
The VolumeAttributesClass feature gate has been upgraded to GA. VolumeAttributesClass and VolumeAttributesClassList APIs have been upgraded to storage.k8s.io/v1. This feature allows configuring volume property classes for PersistentVolumeClaim, providing more flexible volume configuration options.
The APIServerTracing feature gate has been upgraded to GA. The --tracing-config-file flag accepts TracingConfiguration in version apiserver.config.k8s.io/v1, providing full API server tracing capabilities.
The KubeletTracing feature gate has been upgraded to GA, providing full tracing support for kubelet.
The AuthorizeWithSelectors and AuthorizeNodeWithSelectors feature gates have been upgraded to a stable version and locked. When enabled, AuthorizeNodeWithSelectors allows the node authorizer to use more fine-grained selectors (such as fieldSelector and labelSelector) for authentication, improving the flexibility of the Kubernetes authorization system.
The RelaxedEnvironmentVariableValidation feature gate has been upgraded to GA and is locked in the enabled status by default. This feature loosens the validation rules for environment variable names, allowing the use of nearly all printable ASCII characters (except =) as environment variable names.
The PodLevelResources feature has been upgraded to beta and is enabled by default. This feature allows defining CPU and memory resources for the entire Pod in pod.spec.resources, providing Pod-level resource management capabilities.
The PodObservedGenerationTracking feature has been upgraded to beta and is enabled by default. This feature means that the status.observedGeneration and status.conditions[].observed Generation fields in a Pod are populated to reflect the metadata.generation of the Pod specification when the status or condition is reported, providing better status tracking capabilities.
The scheduler QueueingHint has been upgraded to GA. Each scheduler plugin integrates a QueueingHint callback function that helps the scheduler obtain suggestions for Pod re-enqueueing. This allows the scheduler to quickly determine whether a Pod needs to be re-enqueued, reducing unnecessary scheduling retries and thereby improving scheduling throughput.
StreamingCollectionEncodingToJSON and StreamingCollectionEncodingToProtobuf have been upgraded to GA. These features allow the API server to encode collection responses in a streaming method, significantly reducing the memory required for large collections.
ResilientWatchCacheInitialization has been upgraded to GA, providing a healthier cache monitoring initialization mechanism.
ListFromCacheSnapshot has been upgraded to beta, allowing apiserver to provide List requests with accurate ResourceVersion and continuation tokens from the cache.
JobPodReplacementPolicy has been upgraded to GA. The JobPodReplacementPolicy feature gate is locked to true and will be removed in future Kubernetes versions. This feature allows configuring replacement policies for Job Pods.
KubeletCgroupDriverFromCRI has been upgraded to GA and added metrics to track unsupported CRI implementations. This feature allows kubelet to obtain cgroup driver information from the CRI.
The OrderedNamespaceDeletion feature has been upgraded to GA. When it is enabled, Pod resources will be deleted prior to all other resources during namespace deletion, providing a more ordered process for namespace deletion.
The WatchList feature gate has been upgraded to beta, and WatchListClient is enabled for kube-controller-manager. Normally, a large number of List operations can cause a significant increase in the resource consumption of the control plane. After the WatchListClient feature is enabled, client-go clients can use streaming requests to replace full List operations, reducing the resource consumption of the control plane.
The KubeletServiceAccountTokenForCredentialProviders feature is in beta version and enabled by default. This feature allows kubelet to use service account tokens for credential provider authentication.
The MutableCSINodeAllocatableCount feature has been upgraded to beta. When this feature gate is enabled, the CSINode.Spec.Drivers[*].Allocatable.Count field becomes variable, allowing periodic updates to the allocatable volume capacity reported by nodes and preventing stateful Pods from getting stuck due to outdated information that kube-scheduler depends on.
The loose DNS search string validation has been upgraded to GA. For Pod APIs, .spec.dnsConfig.searches now allows using underscores (_) in places where hyphens (-) are allowed, and allows the search string to be a single dot.
The SizeBasedListCostEstimate feature gate has been added (enabled by default). It alters how APF seats are assigned to List requests. With this feature, one seat is assigned for every 100 KB of data loaded into memory during a List operation.
The DetectCacheInconsistency feature gate has been added, allowing the API server to regularly verify consistencies between its cache and etcd. Detected inconsistencies are reported via the apiserver_storage_consistency_checks_total metric.
The SchedulerAsyncAPICalls feature gate has been added. If it is enabled, API calls assigned during Pod scheduling are executed asynchronously to enhance scheduling performance.
The ContainerRestartRules alpha feature has been added, allowing the configuration of container-level restart rules for more fine-grained control over container restarts.
The EnvFiles alpha feature has been added. It introduces a new FileKeyRef field for containers, allowing them to load variables from files by setting this field.
The ResourceHealthStatus alpha feature has been added, adding support in kubelet for monitoring device health of dynamic resources assigned by DRA.
The HostnameOverride alpha feature has been added, allowing the hostnameOverride field to be set in PodSpec to specify any RFC 1123 DNS subdomain as the hostname of the Pod.
The MutatingAdmissionPolicy alpha feature has been added, introducing the admissionregistration.k8s.io/v1beta1/MutatingAdmissionPolicy API type.
The DRA kubelet gRPC API has been upgraded to v1, and v1beta1 is deprecated starting from version 1.34. Version 1.34 supports both API versions for updating the DRA driver to the k8s.io/dynamic-resource-allocation/kubeletplugin helper.
Support for the v1alpha4 kubelet gRPC API has been removed (it was added in version 1.31 and replaced in version 1.32).
The following error has been resolved: DaemonSet updates unnecessarily trigger repeated verification due to overlapping calls to ValidateDaemonSet and ValidateDaemonSetUpdate.
When the PodMaxBackoffDuration kube-scheduler option is set to zero, and the SchedulerPopFromBackoffQ feature gate is enabled, Pod backoff is skipped entirely.
Expansion for PVCs with the node-expand-not-required annotation has been stopped.
If controller expansion has been completed, the scale-out of volumes on nodes is stopped.
Recording error events while waiting for expansion on kubelet is stopped.
Deleting CSI JSON files during subsequent errors is stopped if the volume is already mounted.
The baseline and restricted Pod security access levels prevent setting the host field in probes and lifecycle handlers.
The garbage collection controller no longer competes with ownerReferences changes during the deletion of orphaned objects.
The local podresources API endpoint of the node has been modified to only consider active Pods. Since this changes long-term behavior, the KubeletPodResourcesListUseActivePods feature gate (enabled by default) can be disabled to restore the previous behavior.
Mixed use of user-namespaces (hostUsers: false) and volumeDevices is not allowed for Pods. Kubernetes returns an error in such cases.
kubelet detects a terminal CSI volume mounting failure due to exceeding attachment limits on the node and marks the stateful Pod as failed, allowing its controller to recreate it. This prevents the Pod from being stuck indefinitely in the ContainerCreating status.
Static Pods referencing API objects are denied access by kubelet, preventing silent operation after mirror Pod creation failures.
The scheduling framework exposes NodeInfos to PreFilter plugins. PreFilter plugins now accept a NodeInfo list from parameters.
Deprecated APIs
The v1 Endpoints API is officially deprecated in version 1.33 (though still fully supported). It will not be deleted, but all users should replace it with the EndpointSlice API.
The resource.k8s.io/v1beta1 API is deprecated in version 1.33 and will be deleted in version 1.36. Replace it with v1beta2.
DRA kubelet gRPC API v1beta1 is deprecated starting from version 1.34, and v1 becomes the stable version. Version 1.34 supports both API versions for updating the DRA driver to the k8s.io/dynamic-resource-allocation/kubeletplugin helper.
The StreamingConnectionIdleTimeout field configured in kubelet is deprecated in version 1.34.
The apiserver_storage_objects metric is deprecated in version 1.34 and replaced by the apiserver_resource_objects metric, which uses labels consistent with other metrics.
Apimachinery MessageCountMap and CreateAggregateFromMessageCountMap are deprecated in version 1.34.
The beta annotation service.kubernetes.io/topology-mode is deprecated in version 1.33 and will not be upgraded to GA. Users are encouraged to use the spec.trafficDistribution field in the Service API.
The WatchFromStorageWithoutResourceVersion feature is deprecated in version 1.33 and can no longer be enabled.
The Pod status.resize field is deprecated in version 1.33 and will no longer be set. The Pod resize status will be exposed under two new conditions: PodResizeInProgress and PodResizePending.
The StorageNamespaceIndex feature gate is flipped to false and deprecated in version 1.33.
The SeparateCacheWatchRPC feature gate is deprecated and disabled by default in version 1.33.
The InPlacePodVerticalScalingAllocatedStatus feature gate is deprecated and no longer used in version 1.33.
The scheduler_scheduler_cache_size metric is deprecated in version 1.33, which supports scheduler_cache_size. It will be removed in version 1.34.
The pod_scheduling_duration_seconds metric is deprecated in version 1.33, and users need to migrate to pod_scheduling_sli_duration_seconds.
The AppArmor annotation (prefix container.apparmor.security.beta.kubernetes.io/) is deprecated in version 1.34. The AppArmor configuration file specified in the Pod or container SecurityContext will no longer be copied to the deprecated annotation. Any checks for the deprecated annotation content must be migrated to using the SecurityContext field.
Changelogs
1.32 changes since 1.30
Major Updates
Version 1.31
If the caBundle field specified in a CustomResourceDefinition (CRD) is non-empty but contains invalid content or no CA certificates, the CRD will not provide services. Once a valid caBundle is configured, updates to an invalid or empty status are prohibited to avoid interrupting CRD services.
The MatchLabelKeysInPodAffinity feature gate graduates to Beta and is enabled by default. podAffinity and podAntiAffinity support more fine-grained configuration fields matchLabelKeys and mismatchLabelKeys by default. This resolves the issue where the scheduler cannot distinguish between new and existing Pods during Deployment rolling updates, ensuring that scheduling outcomes align with affinity and anti-affinity expectations. JobSuccessPolicy graduates to Beta and is enabled by default. It allows the configuration of success policies for indexed Jobs. For details, see Job Success Policy. DisableNodeKubeProxyVersion graduates to Beta and is enabled by default. The kube-proxy version is no longer displayed in the status.nodeInfo.kubeProxyVersion field of a node. The value previously displayed in this field was inaccurate and failed to provide the actual version information of kube-proxy.
ServiceAccountTokenNodeBinding graduates to Beta and is enabled by default. It allows the creation of a ServiceAccount token that is directly bound to a node. The token will become invalid when it expires or when the associated node or ServiceAccount is deleted.
RecursiveReadOnlyMounts graduates to Beta and is enabled by default. It allows recursive read-only attributes to be set on Pod volume mounts, meaning that the mounted volumes, along with all internal subdirectories and files, will be set to read-only. For details, see Recursive read-only mounts. When the Pod spec changes without modifications to the image field, kubelet will not restart the container, avoiding unnecessary Pod restarts caused by non-functional configuration updates.
HonorPVReclaimPolicy graduates to Beta and is enabled by default. It supports adding a finalizer to a PersistentVolume, ensuring that the PersistentVolume with the Delete reclaim policy is deleted only after the corresponding underlying storage resources are deleted. For details, see PersistentVolume deletion protection finalizer. The streaming transport protocol in Kubernetes clients (such as kubectl) for kubectl cp, kubectl attach, kubectl exec, and kubectl port-forward commands has been updated to a more modern and flexible WebSocket streaming protocol.
The API server enables consistent reads from the cache, reducing pass-through requests to etcd and improving the efficiency of List requests. For details, see Consistent Reads from Cache. The built-in CephFS volume plugin (kubernetes.io/cephfs) is removed and replaced with the CephFS CSI driver. If you are using the CephFS volume plugin, switch to the new solution and redeploy your applications with the new driver before upgrading your cluster to v1.31. The built-in CephRBD volume plugin (kubernetes.io/rbd) is removed and replaced with the RBD CSI driver. If you are using the CephRBD volume plugin, switch to the new solution and redeploy your applications with the new driver before upgrading your cluster to v1.31. The CSIMigrationPortworx feature gate is enabled by default. It supports the migration of volume operations from the existing Portworx embedded plugin to the Portworx CSI plugin. If you are using Portworx as a storage solution, install and configure the Portworx CSI plugin in your cluster before upgrading the cluster to v1.31.
Version 1.32
The WatchList feature gate graduates to Beta and is enabled by default. In most cases, a large number of List operations can cause a significant increase in control plane resource consumption. After the WatchListClient feature is enabled, the client-go client can use streaming requests to replace full List requests, reducing control plane resource consumption. The feature is enabled for the kube-controller-manager component by default. For details, see Enhancing Kubernetes API Server Efficiency with API Streaming. The RelaxedEnvironmentVariableValidation feature gate graduates to Beta and is enabled by default. The feature loosens the validation rules for environment variable names, allowing the use of nearly all printable ASCII characters (except =) in environment variable names.
The Pod status display is optimized. When an image pull failure occurs, the status.containerStatuses[*].state.waiting field records the cause of the failure (ImagePullBackOff) and detailed failure information.
The CustomResourceFieldSelectors feature gate graduates to General Availability (GA) in v1.32 and is enabled by default. The feature supports enabling selectableFields, including list, watch, and deletecollection in a CRD for more efficient and precise filtering of CRD resources.
The StatefulSetAutoDeletePVC feature gate graduates to GA in v1.32 and is enabled by default. The feature provides fine-grained control over PersistentVolumeClaim (PVC) deletion in StatefulSet, allowing automatic cleanup of PVCs when the StatefulSet explicitly no longer needs them, reducing the impact of orphaned PVCs on StatefulSet. StatefulSet updates and node maintenance processes do not affect PVCs or trigger auto cleanup.
The JobManagedBy feature gate graduates to Beta and is enabled by default. The feature allows external controllers to take over the reconciliation of Jobs by configuring the spec.managedBy field, making Job scheduling and management more flexible. The managedBy field cannot be set to kubernetes.io/job-controller, which is a reserved value for the built-in Kubernetes controller.
Each scheduler plugin integrates a QueueingHint callback function that helps the scheduler obtain suggestions for Pod re-enqueueing. This allows the scheduler to quickly determine whether a Pod needs to be re-enqueued, reducing unnecessary scheduling retries and thereby improving scheduling throughput. For details, see QueueingHint Brings a New Possibility to Optimize Pod Scheduling. The RecoverVolumeExpansionFailure feature gate graduates to Beta and is enabled by default. When a PVC volume expansion fails, administrators usually need to perform complex manual recovery operations. With this feature, users can manually reduce the value of .spec.resources on a PVC to quickly recover the PVC without data loss. For details, see Recovering from Failure when Expanding Volumes. The AuthorizeWithSelectors and AuthorizeNodeWithSelectors feature gates graduate to Beta and need to be enabled together. With these features enabled, AuthorizeNodeWithSelectors allows the node authorizer to use more fine-grained selectors (such as fieldSelector and labelSelector) for authentication, improving the flexibility of the Kubernetes authorization system. This ensures that kubelet has only minimum permissions and can only read its own node objects and Pods bound to its node. For details, see Using Node Authorization. The Stream field is added to PodLogOptions, which is a parameter for the client to retrieve Pod logs. This field enables streaming requests for container logs, meaning that logs are transmitted in real time to the client without waiting for the entire log file download to complete. Note that Stream and TailLines cannot be used together.
JobController is optimized to significantly improve Job update and deletion efficiency, particularly in scenarios with extensive use of Jobs. For details, see pr126567, pr127228, and pr127378. kube-proxy is optimized. fieldSelector: clusterIP!=None is used during Service updates to avoid listening to Headless Services, thereby reducing unnecessary bandwidth usage. For details, see pr126769. The DRAControlPlaneController feature gate graduates to Alpha in v1.26 and disabled by default, is removed in v1.32. If you have enabled and used this feature, delete the related resources before upgrading the cluster to avoid workload exceptions after the upgrade. For details, see pr128003. Deprecated APIs
The FlowSchema and PriorityLevelConfiguration in the flowcontrol.apiserver.k8s.io/v1beta3 version are deprecated in v1.32. It is recommended to migrate to the flowcontrol.apiserver.k8s.io/v1 API version (available since v1.29). In the flowcontrol.apiserver.k8s.io/v1 version, the spec.limited.nominalConcurrencyShares field of PriorityLevelConfiguration is 30 by default if not specified. However, if this field is explicitly set to 0, it will not be changed to 30.
Notes on TKE Control Plane Parameter Changes
During a major version upgrade of the cluster, TKE adjusts the default parameters of control plane components (such as kube-apiserver and kube-controller-manager) based on Kubernetes version features. These adjustments may include changes to Admission Plugins and Feature Gates. Before the upgrade, you are advised to check the current cluster parameters by referring to Customizing Kubernetes Component Startup Parameters, and confirm that the parameter changes meet your expectations after the upgrade. Parameter Changes from 1.30 to 1.32
kube-apiserver admission plugins (--enable-admission-plugins)
|
Remove | PersistentVolumeLabel | This admission plugin has been deprecated since Kubernetes 1.11, and its functionality is assumed by cloud-controller-manager. TKE removed it from the default admission plugin list during the upgrade to version 1.32. If your business relies on the label injection capability of this plugin, confirm that cloud-controller-manager is correctly configured. |
kube-apiserver feature gates (--feature-gates)
|
Added (explicitly disabled) | WatchList=false | In Kubernetes 1.32, the WatchList feature is upgraded to Beta and enabled by default (see the major update note above). When TKE is upgraded to 1.32, WatchList=false is explicitly set to maintain consistency with the behavior of earlier versions and avoid control plane compatibility issues that may arise from enabling streaming Watch. This limitation will be gradually lifted after evaluation in subsequent versions. |
Note:
If you view the cluster parameters through the Customizing Kubernetes Component Startup Parameters feature after the upgrade, you will see the parameter changes described above. This is expected behavior and part of the normal TKE version upgrade process. Changelogs
1.30 changes since 1.28
Major Updates
Version 1.29
PreStop Hook now supports a new sleep mode, allowing a specified period of pause before the container is actually terminated. This helps wait for pending processing or network requests to complete. For details, see KEP-3960: Introducing Sleep Action for PreStop Hook. SidecarContainers graduates to Beta and is enabled by default. The feature allows setting the restartPolicy of init containers to Always, transforming them into sidecar containers that can be independently started, stopped, or restarted without affecting the main application containers or other init containers. For details, see Sidecar Containers. The ServiceCIDR resource type is added to allow dynamically configuring the allocation range of Service ClusterIP addresses in clusters. The feature is in Alpha and disabled by default. For details about dynamically increasing the number of available IP addresses for Services, see KEP-1880: Multiple Service CIDRs. The APIs of PVCs and containers originally shared the same ResourceRequirements struct to define resource requests and limits. As a result, any change to the resources structure of containers, such as the addition of the claims field, would affect PVC APIs. Therefore, PVCs now use a dedicated VolumeResourceRequirements struct that includes only requests and limits and excludes claims. For details, see Volume resource requirements. The PodReadyToStartContainers feature graduates to Beta and is enabled by default. The feature indicates that the Pod's sandbox environment has been successfully created (with the container runtime environment is ready) and network configuration is complete, allowing kubelet to understand the Pod status. For details, see Pod conditions. PodAffinity and PodAntiAffinity now support matchLabelKeys and mismatchLabelKeys. This addresses the issue where the scheduler cannot distinguish between old and new Pods during Deployment rolling updates, which may lead to scheduling results that do not meet affinity and anti-affinity expectations. When matchLabelKeys is configured for PodAffinity, the Deployment adds the pod-template-hash label to ReplicaSets, so that each Pod in the Deployment carries the corresponding hash string. This informs the scheduler to evaluate only Pods with the same pod-template-hash value, making it easier to distinguish Pods from the same update batch. For details, see KEP-3633. ValidatingAdmissionPolicy type checking now supports CRD and API Extension types in addition to core Kubernetes API resources, ensuring policy reliability and cluster configuration accuracy. For details, see type-checking. The UserNamespacesPodSecurityStandards feature gate is added to integrate user namespaces into Pod security standards. When this feature gate is enabled, non-root users or specified users can run containers in the Pod security context. The feature gate is currently in the Alpha stage and disabled by default. It may remain disabled by default in the future. For details, see KEP-127: Update PSS based on feature gate. For Node objects, the DisableNodeKubeProxyVersion feature gate is added and the status.nodeInfo.kubeProxyVersion field is deprecated. This means that setting the kubeProxyVersion field on Nodes is prohibited in Kubernetes. kubelet may not accurately identify the version of kube-proxy, making this field potentially unreliable. The feature gate is currently in the Alpha stage and disabled by default.
Version 1.30
ImageMaximumGCAge allows kubelet to configure the maximum lifetime of unused images before they are garbage collected. After the specified time elapses, if an image remains unused, it can be cleaned up by the garbage collection mechanism. The default value is "0s", which means no time limit is set. The feature gate entered Alpha in v1.29 and advanced to Beta in v1.30.
The monitoring metric image_pull_duration_seconds is added for kubelet to track the image pull time. For details, see List of Alpha Kubernetes Metrics for the Alpha stage. The LegacyServiceAccountTokenCleanUp feature gate graduates to GA and is enabled by default. If an auto-generated Secret associated with a ServiceAccount remains unused for a specified period (one year by default) and is not mounted by any Pod, kube-controller-manager adds the kubernetes.io/legacy-token-invalid-since label to the Secret with the current date as the value, marking the Secret as invalid. If the Secret remains unused for a specified period (one year by default) after it is marked as invalid, kube-controller-manager automatically cleans it up. For Secrets that have been labeled as invalid but not automatically removed, you can remove the kubernetes.io/legacy-token-invalid-since label to make them valid again. For details, see Auto-generated legacy ServiceAccount token clean up and Legacy ServiceAccount token cleaner. In v1.30, when kube-proxy's --nodeport-addresses is not set (unset by default), updates to NodePort Services will only update the Primary IP address of the node, rather than all IP addresses of the node. For details, see #122724. To prevent configuration conflicts and security issues, the OIDC Issuer URL must not be set to the same as the API Server ServiceAccount Issuer URL. For details, see #123561. The LoadBalancerIPMode feature gate allows adding the .status.loadBalancer.ingress.ipMode field to Services of the LoadBalancer type to specify the forwarding behavior of the load balancer IP address. This field can be specified only when the .status.loadBalancer.ingress.ip field is specified. The LoadBalancerIPMode feature gate graduates to Beta. For details, see Specifying IPMode of load balancer status and Load Balancer IP Mode for Services. Pod horizontal autoscaling (HorizontalPodAutoscaler, HPA) based on ContainerResource metrics is upgraded to Stable in v1.30. This new behavior allows HPA to configure autoscaling based on the resource usage of individual containers in a Pod, rather than only the overall resource usage of the Pod, making it easier to configure scaling thresholds for the most important containers in a Pod. For details, see Container resource metrics. The AdmissionWebhookMatchConditions feature gate graduates to GA. It is enabled by default and cannot be disabled. The feature gate supports admission webhook matching based on specific conditions to achieve finer-grained control over webhook trigger conditions. For details, see Dynamic Admission Control. The JobSuccessPolicy feature gate is added, which supports claiming that a job of a set of successful Pods has been successfully completed based on this set of Pods. In a success policy, you can specify certain indexed Pods (for example, indexed Pods x, y, and z) or the number of indexed Pods (for example, 3 Pods in a set of indexed Pods) to declare whether the job has been successfully completed. For details, see Job success/completion policy. The RelaxedEnvironmentVariableValidation feature gate is added, allowing the use of most printable ASCII characters (ASCII codes 32-126, excluding=) in environmental variables. The feature gate is in Alpha and disabled by default. For details, see #123385. The CustomResourceFieldSelectors feature gate is added, supporting the configuration of selectableFields for CRDs to filter List, Watch, and DeleteCollection requests using Field Selectors. This facilitates locating or managing CRD resources that meet specific criteria. The feature gate is in Alpha and disabled by default. For details, see Custom Resource Field Selectors. The CRDValidationRatcheting feature gate is updated: When new validations are added to CRDs, the API server will not block updates to existing resources that become invalid after the update, provided that the invalid portions themselves remain unchanged. Existing resources and users are not affected. This helps perform validity checks when OpenAPI v3 schemas are used during CRD-based migration and securely update validation rules. The feature gate graduates to Beta and is enabled by default. For details, see CRD Validation ratcheting. The Downward API uses the status.hostIPs field to support dual protocol stack, that is, to support IPv4 and IPv6 at the same time. The first IP address in the list of status.hostIPs is always the same as status.hostIP. For details, see Downward API. The NodeLogQuery feature gate allows you to use the /logs endpoint to query node service logs. The feature gate graduates to Beta and is set to false by default. For details, see Log Query. Deprecated Features
In v1.29, CronJob does not support configuring CRON_TZ or TZ in .spec.schedule. Use the .spec.timeZone field instead, which is available since v1.25. For details, see CronJob limitations. In v1.29, the networking/v1alpha1 API ClusterCIDR that is in Alpha and controversial, is removed. For details, see Cluster CIDR V1alpha1. In v1.30, the --prune-whitelist parameter is removed from the kubectl apply command. It is recommended to use --prune-allowlist instead. For details, see --prune. In v1.30, the SecurityContextDeny admission plugin has been deprecated since v1.27 and removed from the code in v1.30. It is recommended to use the PodSecurity admission plugin instead, which graduates to Stable in v1.25 and is enabled by default. For details, see PodSecurity. Deprecated APIs
The FlowSchema and PriorityLevelConfiguration in the flowcontrol.apiserver.k8s.io/v1beta2 API version are deprecated in v1.29. It is recommended to use the flowcontrol.apiserver.k8s.io/v1 API version (available since v1.29) or the flowcontrol.apiserver.k8s.io/v1beta3 API version (available since v1.26).
Significant changes in flowcontrol.apiserver.k8s.io/v1 include: the spec.limited.assuredConcurrencyShares field of PriorityLevelConfiguration is renamed to spec.limited.nominalConcurrencyShares, which defaults to 30 only when unspecified, and an explicit value of 0 will not be changed to 30.
Significant changes in flowcontrol.apiserver.k8s.io/v1beta3 include: the spec.limited.assuredConcurrencyShares field of PriorityLevelConfiguration is renamed to spec.limited.nominalConcurrencyShares.
Changelogs
1.28 changes since 1.26
Major Updates
The IPv6DualStack feature gate is removed. It graduates to GA in v1.23 and is completely removed in v1.27. If this parameter is configured in your cluster, remove it before an upgrade.
Set terminal phase for all Pods to be restarted:
Set terminal phase to failed for deleted Pods in pending status and to succeeded or failed for deleted Pods in running status (depending on the container exit status). This fixes the issue that Job Pods could get stuck in pending when being deleted while a failure policy is in use.
For Pods with RestartPolicy set to Always, the Pod status will end with succeeded during deletion or graceful node shutdown. For this reason, your controllers need to be developed to support this scenario.
The ReadWriteOncePod access mode of PersistentVolumes (PVs) graduates to Beta and is supported by default. Only one Pod is allowed to write data to volumes.
The Pod topology spread constraint controls the reasonable distribution of Pods within a cluster across multiple availability zones (AZs). Multiple enhanced features graduate to Beta, including minDomains (specifying the minimum number of domains in which Pods are spread), nodeTaintsPolicy (specifying how node taints are treated during Pod scheduling), nodeAffinityPolicy (specifying how node affinity is treated during Pod scheduling), and whenUnsatisfiable (specifying how a Pod is treated if it does not comply with the constraints).
The server-side field validation feature, which validates resources sent to the API server, graduates to GA. kubectl skips client-side validation and automatically uses server-side field validation in Strict mode, returning errors if validation fails.
OpenAPI v3 is the latest standard of OpenAPI. It is introduced in Kubernetes v1.23 and graduates to GA in v1.27.
HPA allows you to configure container resource metrics for containers in a Pod to perform automatic scaling based on the resource usage of each container. The feature graduates to Beta in v1.27. Compared with resource metrics that indicate the average resource usage of Pods, container resource metrics indicate the resource usage of each container. This helps solve the issue that the average resource usage of a Pod cannot trigger scale-out because the resource usage of sidecar containers in the Pod is low but the resource usage of application containers is high.
Multiple StatefulSet features graduate to Beta, including the features for assigning serial numbers to Pods from a number other than zero, deleting specified PVCs, and automatically deleting created PVCs during scale-in.
A new feature is added to support adjusting CPU and memory resources of containers in a Pod specified by the resources field without restarting the Pod and containers. Nodes allocate resources for the Pod based on requests and restrict resource usage based on limits. Some fields are added to Pods to support this feature. The feature is in Alpha in v1.27 and disabled by default.
kubelet supports setting serializeImagePulls to false to pull images concurrently instead of serially by default. In v1.27, the maxParallelImagePulls field is added to limit the number of concurrent image pulls, preventing image pulls from consuming excessive network bandwidth or disk I/O.
Regarding the PV snapshot capability, in addition to the existing Volume Snapshot API, a new Volume Group Snapshot capability is added for consistent snapshots, allowing multiple volumes to be snapshotted at the same point in time.
The non-graceful node shutdown feature graduates to GA. When a node shuts down abnormally due to power outage or other failures, stateful applications need to be promptly restarted on running nodes to avoid impacting the normal business operation.
The NodeOutOfServiceVolumeDetach feature gate graduates to GA. It allows immediate volume separation of terminated Pods on abnormal nodes to support quick recovery of Pods on other nodes.
The traceable default StorageClass assignment feature graduates to GA. Previously, if a PVC without storageClassName configured was created before the default StorageClass existed, the PVC stayed pending indefinitely. With this feature, once the default StorageClass exists, existing PVCs without storageClassName configured will be automatically updated to use the default StorageClass.
Two features are introduced to avoid Job failures:
The JobPodReplacementPolicy feature gate (Alpha) controls that Pods are replaced only when they reach the Failed phase (status.phase set to Failed) rather than when they have a deletionTimestamp and are in the process of terminating. This prevents two Pods from simultaneously occupying the same index and node resources.
The JobBackoffLimitPerIndex feature gate (Alpha) limits the number of retries upon failure for some indexes of an indexed Job by configuring .spec.backoffLimitPerIndex. This prevents a single persistently failing index from reaching the .spec.backoffLimit and causing the entire Job to fail.
For an indexed Job, if completion exceeds 100,000 and parallelism exceeds 10,000, tracking the Job termination status may fail when a large number of Pods fail. Therefore, a warning will be issued during Job creation if these parameters are set too high.
The reason and fieldPath fields are added to CRD validation rules, allowing the specified reason and field path to be returned when validation fails.
Webhook request filtering now supports CEL expressions, allowing for the definition of up to 64 match conditions.
The SidecarContainers feature gate is added to support sidecar containers and optimize the container startup order.
The .status.resizeStatus field of PVCs are replaced by the .status.allocatedResourceStatusMap structured field, which indicates the status during PVC resizing.
Pod indexes (sequence numbers) are added as labels to Pods created by indexed Jobs and StatefulSets.
ValidatingAdmissionPolicy (Beta) provides a declarative way to validate resource requests, replacing the deployment of validating admission webhooks and supporting CEL expressions for complex validation rules. The API server validates requests against the CEL expressions; only after successful evaluation are the requests considered approved.
Kube Controller Manager (KCM) adds --concurrent-cron-job-syncs to configure the concurrency of the CronJob controller and --concurrent-job-syncs to configure the concurrency of the Job controller.
API server optimizations include:
Memory consumption of GetList execution on cache is reduced.
This fixes the issue where the Kubernetes Service endpoint is not removed when only one API server replica remains. It ensures that endpoints are promptly removed during graceful termination.
Set the OpenAPI v2 controller to lazily aggregate CRD information and significantly reduce the OpenAPI v2 specification. When no clients request OpenAPI v2, this reduces API Server CPU and memory consumption and improves performance when many CRDs are installed, although the first client request will be slower. It is recommended to upgrade clients to support OpenAPI v3.
The Consistent Reads from Cache feature gate is added to enable consistent List reads by using the watch cache.
A variety of monitoring metrics are provided and can be collected by calling the metrics API.
Deprecated Features
The spec.externalID field for nodes is deprecated. Updates to this field will now trigger a warning.
For the Service API, a warning will be triggered if IPv4 addresses with leading zeros or non-canonical IPv6 addresses are used.
The SecurityContextDeny admission plugin is deprecated in v1.27 and will be removed in later versions.
The AWS in-tree cloud provider is removed and replaced with an external CSI plugin.
The Alpha-stage annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io are deprecated in v1.19 and completely deleted in v1.27. It is recommended to replace them with the securityContext.seccompProfile field of Pods or containers.
The startup commands --pod-eviction-timeout (grace period for deleting failed nodes) and --enable-taint-manager (taint-based eviction, which is enabled by default) are removed from KCM.
The startup commands --container-runtime, --container-runtime-endpoint, and --image-service-endpoint are removed from kubelet. Among them, --container-runtime uses the default value remote after dockershim is removed. It is deprecated starting from v1.24 and removed in v1.27. --container-runtime-endpoint and --image-service-endpoint are no longer supported and can be configured using a configuration file instead.
The code of the CephFS volume plugin is removed from the tree. The CephFS CSI driver is recommended.
The migration of Ceph RBD volumes to the out-of-tree CSI is no longer supported. The feature will be completely removed in later versions. It is recommended to complete the migration before removing the code from the tree.
The RBD volume plugin (kubernetes.io/rbd) is deprecated and will be removed in later versions. The CephFS CSI driver is recommended.
KMSv1 is deprecated. If you still need to use KMSv1 in a compatible way, set --feature-gates=KMSv1 to true.
Startup commands --volume-host-cidr-denylist and --volume-host-allow-local-loopback of KCM are deprecated.
The command --azure-container-registry-config of kubelet is deprecated. It is recommended to use image-credential-provider-config and --image-credential-provider-bin-dir.
Windows node pools can no longer be created.
The CSIStorageCapacity API can expose the current available storage capacity, which ensures that Pods are scheduled to nodes with sufficient storage capacity. The storage.k8s.io/v1beta1 API version of CSIStorageCapacity is deprecated in v1.24 and removed in v1.27.
It is recommended to use the storage.k8s.io/v1 API, which has been available since v1.24.
Feature Gates
The NodeLogQuery feature gate (Alpha) is added. kubelet can be used to query node logs after enableSystemLogHandler and enableSystemLogQuery are set to true.
StatefulSetStartOrdinal graduates to Beta and is enabled by default. It allows that the sequence number of Pods configured with StatefulSets can start from a non-zero number.
StatefulSetAutoDeletePVC graduates to Beta. A new StatefulSet PVC retention policy is used to control whether and when to delete PVCs created by using the volumeClaimTemplate of a StatefulSet.
IPv6DualStack is enabled by default since it graduates to GA in v1.23. It is completely removed from the code of each component in v1.27. If it is manually configured in a cluster, it needs to be deleted before a cluster upgrade.
The ServiceNodePortStaticSubrange feature gate (Alpha) is added to reduce conflicts in assigning ports to NodePort Services. The feature gate divides the port range. Dynamic port assignment of NodePort Services uses the upper band, and the lower band with a low risk of port conflicts can be used to statically assign ports.
The InPlacePodVerticalScaling feature gate (Alpha) is added to support adjusting CPU and memory resources of Pods without restarting Pods and containers.
The volume expansion feature graduates to GA and is enabled by default. The corresponding three feature gates are removed: ExpandCSIVolumes (enables CSI volume expansion), ExpandInUsePersistentVolumes (enables expansion of in-use PVs), and ExpandPersistentVolumes (enables PV expansion).
CSIMigration moves in-tree storage plugins to external CSI drivers and is enabled by default. The feature gate is removed.
CSIInlineVolume graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
Ephemeral Containers graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
LocalStorageCapacityIsolation supports capacity isolation for local temporary storage of emptyDir volumes. It can set hard limit on the consumption of local storage resources by Pods. When the local storage usage of a Pod exceeds the limit, kubelet will evict the Pod. It graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
NetworkPolicyEndPort supports the use of the endPort field in NetworkPolicy to specify a port range, replacing the previous limitation where only a single port could be specified. The feature graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
StatefulSetMinReadySeconds supports configuring minReadySeconds for StatefulSets. The feature graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
DaemonSetUpdateSurge supports configuring maxSurge for Daemonsets. It graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
IdentifyPodOS supports specifying an operating system for a Pod. It graduates to GA in v1.25 and is enabled by default. The feature gate is removed.
ReadWriteOncePod gradates to Beta and is enabled by default. It allows access to PVs in ReadWriteOncePod mode.
When NodeOutOfServiceVolumeDetach marks a node as out-of-service by using the node.kubernetes.io/out-of-service taint, Pods on the node that cannot tolerate this taint will be forcibly deleted, and the volume separation operation will be immediately performed. It graduates to GA in v1.28 and is enabled by default.
AdmissionWebhookMatchCondition is enabled by default. It allows webhooks to use CEL expressions to write match conditions.
UnknownVersionInteroperabilityProxy enters Alpha. It supports the proxying of requests to the correct API server when multiple API server versions exist.
IPTablesOwnershipCleanup graduates to GA. KUBE-MARK-DROP and KUBE-MARK-MASQ iptables chains are no longer created.
ConsistentListFromCache enters Alpha. It allows the API server to achieve consistent List reads by using the watch cache.
Some feature gates that have graduated to GA are removed, including DelegateFSGroupToCSIDriver, DevicePlugins, KubeletCredentialProviders, MixedProtocolLBService, ServiceInternalTrafficPolicy, ServiceIPStaticSubrange, and EndpointSliceTerminatingCondition.
Changelogs
1.26 changes since 1.24
Major Updates
PodSecurityPolicy is removed and Pod Security Admission graduates to Stable: PodSecurityPolicy is deprecated in v1.21 and completely removed in v1.25. Updates required to improve its usability lead to breaking changes; therefore, it is necessary to remove it and replace it with the more user-friendly Pod Security Admission. If you are currently using PodSecurityPolicy, see Migrate from PodSecurityPolicy to the Built-In PodSecurity Admission Controller. Ephemeral Containers graduates to Stable: Ephemeral Containers graduates to Beta in v1.23 and to Stable in v1.25. Ephemeral containers can be used in a Pod for inspection and troubleshooting when you cannot use kubectl exec effectively due to a crash or a lack of debugging tools in the image.
cgroup v2 graduates to Stable in v1.25: cgroups v2 provides numerous improvements over cgroups v1. While support for cgroups v1 continues, this cgroups v2 enhancement prepares Kubernetes for the eventual deprecation the replacement with v2.
Windows support is further optimized in v1.25, including support for unit tests, conformance tests, and the creation of a new repository for Windows Operational Readiness. The image repository is migrated from k8s.gcr.io to registry.k8s.io.
SeccompDefault graduates to Beta in v1.25.
The endPort field of NetworkPolicy graduates to stable in v1.25. Previously, each NetworkPolicy could only target a single port, while the endPort field can specify a port range.
Local Ephemeral Storage Capacity Isolation graduates to Stable in v1.25, providing capacity isolation for local ephemeral container storage (such as EmptyDir) between Pods. If a Pod's consumption of local ephemeral storage exceeds the limit, it can be evicted to limit its consumption of shared resources.
CSI Migration graduates to Stable in v1.25. As a continuous effort by SIG Storage across multiple versions, CSI Migration aims to migrate in-tree volume plugins to out-of-tree CSI drivers.
CSI Ephemeral Volume graduates to Stable in v1.25. The feature allows CSI volumes to be specified directly in the Pod specifications for ephemeral use cases. They can be used to inject arbitrary states, such as configurations, secrets, identities, and variables, directly into Pods via mounted volumes. Initially introduced as an Alpha feature in v1.15, it is in Stable now. Some CSI drivers (such as the secrets-store CSI driver), uses this feature.
CRD Validation Expression Language graduates to Beta in v1.25.
Server Side Unknown Field Validation graduates to Beta in v1.25 and is enabled by default. The API server supports validation of unknown fields, allowing the planned and phased removal of client-side validation functionality in kubectl.
KMS v2alpha1 API is introduced in v1.25 to enhance performance, enable rotation, and improve observability. It replaces AES-CBC with AES-GCM for static data encryption via DEK without requiring additional user operations, while continuing to allow reads using both AES-GCM and AES-CBC.
CRI v1alpha2 API is removed in v1.26. CRI v1 is recommended. containerd v1.5 and earlier are no longer supported in v1.26. If you use containerd, upgrade it to v1.6.0 or later before upgrading the node to Kubernetes v1.26.
In v1.26, the flowcontrol.apiserver.k8s.io/v1beta1 API version of FlowSchema and PriorityLevelConfiguration in the v1beta1 flow control API group is no longer served. The flowcontrol.apiserver.k8s.io/v1beta2 API version is available since v1.23.
The HPA in autoscaling/v2beta1 API version is no longer served starting from v1.25. The HPA in autoscaling/v2beta2 API version is no longer served in v1.26. The autoscaling/v2 API version is recommended, which is available starting from v1.23.
The in-tree credential management code, specifically the vendor-specific identity authentication code for Azure and Google Cloud that was previously part of Kubernetes, is removed from client-go and kubectl. As an alternative, Kubernetes provides a vendor-neutral authentication plugin mechanism before the v1.26 release.
Deprecation and Removal
Parameters
kubectl command-line parameter: As part of the implementation of the Inclusive Naming Initiative, the --prune-whitelist flag is deprecated in v1.26 and replaced with --prune-allowlist.
kube-apiserver command-line parameter: The --master-service-namespace command-line parameter has no effect on kube-apiserver and has been informally deprecated. This parameter will be formally marked as deprecated in v1.26, with removal planned in a future release. The Kubernetes project expects no impact from this deprecation and removal.
kubectl run command-line parameters: In v1.26, several unused options for the kubectl run subcommand are marked as deprecated, including: cascade, filename, force, grace-period, kustomize, recursive, timeout, and wait.
The kube-proxy userspace mode is removed in v1.26. Linux users must use the iptables or ipvs mode, and Windows users must use the kernelspace mode. Using --proxy-mode userspace results in failures.
API Versions
1.25
batch/v1beta1 is removed and replaced with batch/v1.
discovery.k8s.io/v1beta1 is removed and replaced with discovery.k8s.io/v1.
events.k8s.io/v1beta1 is removed and replaced with events.k8s.io/v1.
policy/v1beta1 is removed and replaced with policy/v1.
node.k8s.io/v1beta1 is removed and replaced with node.k8s.io/v1.
autoscaling/v2beta1 is removed and replaced with autoscaling/v2.
1.26
autoscaling/v2beta2 is removed and replaced with autoscaling/v2.
flowcontrol.apiserver.k8s.io/v1beta1 is removed and replaced with flowcontrol.apiserver.k8s.io/v1beta2 or flowcontrol.apiserver.k8s.io/v1beta3.
Others
In v1.25, the GlusterFS and Portworx in-tree volume plugins are deprecated. Although a CSI driver exists for GlusterFS, it is not maintained. The community has discussed the possibility of migrating to a compatible CSI driver, but ultimately decides to start deprecating the GlusterFS plugin from the in-tree drivers.
In v1.25, PodSecurityPolicy, Flocker, Quobyte, and StorageOS are removed.
In v1.25, the change to vSphere version support: The in-tree vSphere volume driver no longer supports any vSphere versions earlier than 7.0u2.
In v1.25, the ownership of iptables chains is cleaned up. Kubelet will gradually migrate to no longer creating the following iptables chains in the nat table: KUBE-MARK-DROP, KUBE-MARK-MASQ, and KUBE-POSTROUTING. This change will be rolled out in phases through the IPTablesCleanup feature gate.
In v1.26, the GlusterFS in-tree driver is removed, and the deprecated OpenStack in-tree storage integration (Cinder volume type) is removed. Changelogs
1.24 changes since 1.22
Attention:
1. Support for docker via dockershim is removed in Kubernetes 1.24 and the container runtime for new TKE nodes only supports containerd 1.6.9.
2. For the scenario of upgrading clusters from v1.22 to v1.24, nodes using docker or containerd of a version earlier than 1.4 as the runtime only support node upgrades via the reinstallation rolling upgrade mode.
Major Updates
Removing dockershim from kubelet
The built-in dockershim in kubelet is deprecated in v1.20 and removed starting from v1.24. From v1.24 onwards, nodes using the docker runtime need to migrate to other runtimes such as containerd or CRI-O. For details, see Is Your Cluster Ready for v1.24. Graduating Ephemeral Containers to Beta
The Ephemeral Containers feature graduates to Beta in v1.23. Ephemeral containers are containers that exist for a limited time within an existing Pod. They are particularly useful for troubleshooting when you need to inspect another container but cannot use kubectl exec because the container has crashed or its image lacks debugging tools.
Disabling Beta APIs by Default
New Beta APIs are disabled by default starting from v1.24. Existing Beta APIs and those graduating to Beta remain unaffected.
Graduating Storage Capacity Tracking and Volume Expansion to GA
Graduating NonPreemptingPriority to GA
The NonPreemptingPriority feature graduates to GA in v1.24. It is enabled by default and cannot be disabled.
Graduating GRPCContainerProbe to Beta
The GRPCContainerProbe feature graduates to Beta in v1.24 and is enabled by default.
Graduating IdentifyPodOS to Beta
The IdentifyPodOS feature is introduced in v1.23 and graduates to Beta in v1.24. Once enabled, the podSpec can specify an os field to indicate the desired OS type (Linux/Windows) for the Pod. During Pod creation, other settings, such as the SecurityContext field, are validated against this field. Note that you still need to use nodeSelector kubernetes.io/os to select nodes with the expected OS type. If the OS of the scheduled node does not match the OS specified in this field, kubelet refuses to run the Pod.
Graduating JobReadyPods to Beta
The JobReadyPods feature is introduced in v1.23 and graduates to Beta in v1.24. Once enabled, the Job status displays the number of Ready Pods.
Graduating KubeletCredentialProviders to Beta
The feature graduates to Beta in v1.24 and is enabled by default, allowing kubelet to dynamically obtain credentials for container image repositories using exec plugins, rather than storing credentials on the node's file system.
ServiceIPStaticSubrange Alpha Feature
In v1.24: ServiceIPStaticSubrange is introduced. Once enabled, it effectively prevents conflicts between ClusterIPs assigned to Services.
Removing Dynamic kubelet Configuration
Dynamic kubelet configuration is deprecated in v1.22 and removed in v1.24.
Incompatible Changes Related to CNI Versions
The following runtime versions are validated as compatible with v1.24:
containerd v1.6.4+,v1.5.11+
CRI-O 1.24+
For containerd versions v1.6.0 to v1.6.3, you need to upgrade the CNI plugin or declare the CNI configuration version; otherwise, you encounter errors such as Incompatible CNI versions and Failed to destroy network for sandbox.
Removing VolumeSnapshot v1beta1
VolumeSnapshot v1beta1 is deprecated in v1.20 and removed in v1.24.
Deprecateing FlexVolume
FlexVolume is deprecated starting from v1.23. CSI is recommended for volume driver development.
Deprecating klog-related Parameters
Starting from v1.23, all klog parameters except --v and --vmodule are deprecated: --log-dir, --log-file, --log-flush-frequency, --logtostderr, --alsologtostderr, --one-output, --stderrthreshold, --log-file-max-size, --skip-log-headers, --add-dir-header, --skip-headers, --log-backtrace-at.
Graduating IPv4/IPv6 Dual-Stack Support to Stable
Kubernetes enables dual-stack support by default starting from v1.21, graduates to GA in v1.23, and removes the IPv6DualStack feature. Although clusters support dual-stack networking by default, Pods and Services still use a single network stack by default. To use dual-stack, nodes must have routable IPv4/IPv6 network interfaces and use a CNI network plugin that supports dual-stack. Additionally, Services must be configured to use dual-stack: the .spec.ipFamilyPolicy field must be set to PreferDualStack or RequireDualStack. For more information, see IPv4/IPv6 dual-stack. Graduating HPA v2 to GA
The autoscaling/v2 API graduates to GA in v1.23, and the autoscaling/v2beta2 API is deprecated.
Graduating Generic Ephemeral Volume to GA
The Generic Ephemeral Volume feature graduates to GA in v1.23, allowing all storage plugins that support dynamic provisioning to provide ephemeral volumes for Pods.
Graduating Skip Volume Ownership Change to GA
This feature graduates to GA in v1.23, allowing the .securityContext.fsGroupChangePolicy field to be set to OnRootMismatch. Only when the Owner and permissions of the root directory do not match the expected permissions of the volume are the Owner and permissions of its contents changed. Otherwise, recursive permission setting can be skipped when the volume is mounted, accelerating Pod startup.
Graduating PodSecurity to Beta
PodSecurity replaces the deprecated PodSecurityPolicy admission controller and graduates to Beta in v1.23.
Using CRI v1 by Default
Starting from v1.23, kubelet supports and uses the CRI v1 API by default. If the runtime does not support v1, it falls back to v1alpha2.
Graduating Structured Logging to Beta
Structured Logging graduates to Beta in v1.23, with most logs of kubelet and kube-scheduler being recorded in structured JSON format.
Simplifying Scheduler Plugin Configuration
Starting from v1.23, the MultiPoint field is introduced in the scheduler plugin configuration. Plugins configured using this field automatically register to all extension point plugin sets they support, simplifying plugin configuration.
CustomResourceValidationExpressions Alpha Feature
The CustomResourceValidationExpressions feature is introduced in v1.23. Once enabled, CRDs can be validated using rules written in Common Expression Language (CEL). ServerSideFieldValidation Alpha Feature
The ServerSideFieldValidation feature is introduced in v1.23. Once enabled, the API server returns warning messages when a request contains unknown or duplicate fields. You can specify the fieldValidation parameter in the request:
Ignore: Default behavior prior to v1.23 and when the ServerSideFieldValidation feature is enabled. Unknown or duplicate fields (except the last duplicate field) are ignored.
Warn: Default behavior when the ServerSideFieldValidation feature is enabled.
Strict: If the request fails, an Invalid Request error is returned.
Graduating OpenAPIV3 to Beta
The OpenAPIV3 feature is introduced in v1.23 and graduates to Beta in v1.24. You can obtain the OpenAPI v3.0 specifications for all Kubernetes types:
$cluster/openapi/v3/apis/<group>/<version>: a specific type
$cluster/openapi/v3: all types
PodAndContainerStatsFromCRI Alpha Feature
The PodAndContainerStatsFromCRI Alpha feature is introduced in v1.23. Once enabled, Pod metrics are obtained via the CRI API rather than from cAdvisor.
Other Updates
Features Graduating to GA
1.23:TTLAfterFinished,CSIVolumeFSGroupPolicy,IngressClassNamespacedParams,
Features Graduating to Beta
1.23:JobTrackingWithFinalizers,StatefulSetMinReadySeconds
1.24:AnyVolumeDataSource,MixedProtocolLBService,GracefulNodeShutdownBasedOnPodPriority
Deprecation and Removal
Parameters
kube-apiserver: In v1.24, the --master-count parameter and the --endpoint-reconciler-type=master-count option are deprecated. By default, the Lease objects created by the API server are used to synchronize API server Service Endpoints. The following parameters are removed in v1.24: --address, --insecure-bind-address, --port, and --insecure-port.
In v1.24, the following kube-controller-manager parameters are removed: --address, --port, and --deployment-controller-sync-period.
The --port and --address parameters for kube-controller-manager and kube-scheduler are deprecated and become non-functional in v1.23, and are removed in v1.24.
Legacy scheduler configuration methods and the related parameters policy-config-file, policy-configmap, policy-configmap-namespace, and use-legacy-policy-config are removed in v1.23. The KubeSchedulerConfiguration configuration file must be used.
In v1.23, the kubelet experimental-bootstrap-kubeconfig parameter is removed and replaced with the --bootstrap-kubeconfig parameter.
In v1.23, the kubelet --seccomp-profile-root parameter and the seccompProfileRoot configuration item are removed.
In v1.24, the kubelet --pod-infra-container-image parameter is deprecated. The sandbox image specified by this parameter is not garbage collected. The image garbage collection process obtains sandbox image information via the CRI interface.
API Versions
1.23
rbac.authorization.k8s.io/v1alpha1 and scheduling.k8s.io/v1alpha1 are removed; use rbac.authorization.k8s.io/v1 and scheduling.k8s.io/v1 instead.
The v1beta1 scheduler configuration version is removed; please use v1beta2 or v1beta3 configuration files.
1.24
client.authentication.k8s.io/v1alpha1 ExecCredential is removed; use client.authentication.k8s.io/v1 instead.
node.k8s.io/v1alpha1 RuntimeClass is removed; use node.k8s.io/v1 instead.
audit.k8s.io/v1alpha1 and audit.k8s.io/v1beta1 are removed; use audit.k8s.io/v1 instead.
CSIStorageCapacity storage.k8s.io/v1beta1 is deprecated; use storage.k8s.io/v1 instead.
Others
In v1.24, the Service tolerate-unready-endpoints annotation, deprecated since v1.11, is removed. Use Service.spec.publishNotReadyAddresses instead.
In v1.24, the following deprecated features are removed: ValidateProxyRedirects and StreamingProxyRedirects.
In v1.24, the metadata.clusterName field is deprecated. This read-only field is always empty and can be misleading.
Changelogs
1.22 changes since 1.20
Major Updates
Deprecating PodSecurityPolicy
PodSecurityPolicy is deprecated in v1.21 and is planned to be removed in v1.25. Evaluate and migrate to Pod Security Admission or third-party admission plugins.
Graduating Immutable Secrets and ConfigMaps to GA
Once a Secret or ConfigMap is set as immutable (immutable: true), kubelet stops watching for changes to these objects and remounting them into containers, which reduces the load on the API server. The feature graduates to GA in v1.21.
CronJobs GA
CronJobs graduates to GA in v1.21 (batch/v1), and the higher-performance CronJobControllerV2 is enabled by default.
Graduating IPv4/IPv6 Dual-Stack Support to Beta
Dual-stack networking allows Pods, services, and nodes to obtain both IPv4 and IPv6 addresses. In v1.21, this feature graduates from Alpha to Beta and is enabled by default.
Graduating Graceful Node Shutdown to Beta
The feature graduates to Beta in v1.21, allowing for the graceful termination of Pods on a node by notifying kubelet when the node is shut down.
Introducing PV Health Monitoring to Alpha
The feature is introduced as an Alpha feature in v1.21, allowing for PV monitoring to track volume health. When volumes become unhealthy, they are marked, which enables workloads to react and prevent data from being written to or read from unhealthy volumes.
Server-side Apply GA
Server-side Apply helps both users and controllers manage resources through declarative configuration, including the declarative creation or modification of objects. Server-side Apply graduates to GA in v1.22.
Graduating External Credentials to GA
Starting from v1.22, the External Credentials feature graduates to GA, offering improved support for interactive login flow plugins. For details, see sample-exec-plugin. Updating etcd to v3.5.0
In v1.22, etcd v3.5.0 is used by default. This version improves security, performance, monitoring, and developer experience, fixes multiple bugs, and introduces important new features such as structured logging and built-in log rotation.
MemoryQoS
Starting from v1.22, the alpha version of the MemoryQoS feature is supported. When enabled, it uses the Cgroups v2 API to manage and control memory allocation and isolation, ensuring workload memory usage and improving workload and node availability during memory resource contention. This feature was proposed by Tencent Cloud and contributed to the community.
Default seccomp Configuration for Clusters
In v1.22, the SeccompDefault Alpha feature is introduced for kubelet. Combined with the --seccomp-default parameter and configuration, kubelet uses the RuntimeDefault seccomp configuration instead of Unconfined, thereby enhancing workload security.
Other Updates
Features Graduating to GA:
1.21: EndpointSlice,Sysctls,PodDisruptionBudget
1.22: CSIServiceAccountToken
Features Graduating to Beta:
1.21: TTLAfterFinished
1.22: SuspendJob,PodDeletionCost,NetworkPolicyEndPort
In v1.22, a new scheduler scoring plugin, NodeResourcesFit, is introduced to replace NodeResourcesLeastAllocated, NodeResourcesMostAllocated, and RequestedToCapacityRatio.
Starting from v1.22, when the APIServerTracing Alpha feature is enabled, the API server supports distributed tracing. It also supports setting multiple issuers via the --service-account-issuer parameter, providing uninterrupted service during issuer changes.
Deprecation and Removal
Removed Parameters and Features
1. Service TopologyKeys is deprecated; Topology Aware Hints can be used instead.
2. kube-proxy
Starting from v1.21, net.ipv4.conf.all.route_localnet=1 is no longer automatically set in IPVS mode. Upgraded nodes retain net.ipv4.conf.all.route_localnet=1, but new nodes inherit the system default value (generally 0).
The --cleanup-ipvs parameter is deleted; the --cleanup parameter can be used instead.
3. kube-controller-manager
Starting from 1.22, the --horizontal-pod-autoscaler-use-rest-clients parameter is removed.
The --port and --address parameters no longer function and will be removed in v1.24.
4. kube-scheduler: The --hard-pod-affinity-symmetric-weight and --scheduler-name parameters are removed starting from v1.22; the config file can be used for configuration.
5. kubelet: The DynamicKubeletConfig feature is deprecated and is disabled by default. When kubelet is started, If the --dynamic-config-dir parameter is configured, an alarm is issued during kubelet startup.
Removed or Deprecated Versions
1. CronJob batch/v2alpha1 is removed starting from v1.21.
rbac.authorization.k8s.io/v1beta1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1beta1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1beta1
certificates.k8s.io/v1beta1
coordination.k8s.io/v1beta1
extensions/v1beta1 and networking.k8s.io/v1beta1 ingress APIs
Changelogs
1.20 changes since 1.18
Major Updates
New CronJob Controller
1.20 introduced a new CronJob controller that uses the informer mechanism instead of the original polling to improve performance. You can enable it by specifying --feature-gates="CronJobControllerV2=true" in kube-controller-manager. In later versions, the new controller will be used by default.
Deprecating dockershim
Dockershim has been officially deprecated. kubernetes' support for Docker is deprecated and will be removed in a future release. Images built by Docker that comply with the OCI specification can continue to run in CRI-compatible runtimes.
Structured Logging
The structures of log messages and k8s object references have been standardized, making log parsing, processing, storage, querying, and analysis simpler. klog has added two methods to support structured logging: InfoS and ErrorS.
Add the --logging-format parameter to all components. The default value is text, which maintains the previous format. Set it to json to support structured logging. In this case, the following parameters no longer take effect: --add_dir_header, --alsologtostderr, --log_backtrace_at, --log_dir, --log_file, --log_file_max_size, --logtostderr, --skip_headers, --skip_log_headers, --stderrthreshold, --vmodule, --log-flush-frequency.
Timeout Handling for Exec Probes
A long-standing bug related to Exec probe timeouts has been fixed, which may affect existing Pod definitions. Before this fix, the timeout specified by the timeoutSeconds field was not honored. Instead, the probe would run indefinitely, even beyond its configured deadline, until a result was returned. After this change, if no value is specified, the probe applies a default timeout of 1 second. If the probe takes longer than 1 second to execute, existing Pod definitions may need to be modified to explicitly specify the timeoutSeconds field. This fix also adds a feature gate named ExecProbeTimeout that allows the previous behavior to be retained (this feature will be locked and removed in a later release). To retain the previous behavior, set ExecProbeTimeout to false.
Volume Snapshot Operations GA
This feature provides a standard method for triggering volume snapshot operations and allows users to perform combined snapshot operations in a portable manner across any Kubernetes environment and supported storage providers.
Additionally, these Kubernetes snapshot primitives can serve as a foundation to unlock the ability to develop advanced enterprise-grade storage management features for Kubernetes, including application-level or cluster-level backup solutions.
Note that snapshot support requires the Kubernetes cluster to deploy the snapshot controller, snapshot CRDs, and validating Webhook, as well as a CSI driver that supports snapshot functionality.
kubectl debug Enters beta
The kubectl alpha debug command has entered the beta stage and has been replaced by kubectl debug. This feature supports common debugging tasks directly from kubectl, including:
Create a copy of a Pod using a different container image or command to troubleshoot a workload that crashes at startup.
Troubleshoot containers that do not contain debugging tools, such as distroless images, by adding a new container with debugging tools in a new Pod copy or in an ephemeral container. (Ephemeral containers EphemeralContainers are an Alpha feature and are not enabled by default.)
You can troubleshoot issues on a node by creating a container that runs in the host namespace and accessing the host's filesystem.
Note that as a new built-in command, kubectl debug takes precedence over any kubectl plugin named "debug". You must rename the affected plugins.
kubectl alpha debug is deprecated and will be removed in a subsequent version. You need to replace it with kubectl debug. For more information, see Debug Running Pods.API Priority and Fairness Enters beta
The API Priority and Fairness feature introduced in version 1.18 will be enabled by default in version 1.20, allowing kube-apiserver to classify incoming requests by priority.
PID Resource Limit Feature GA
SupportNodePidsLimit (PID isolation from node to Pod) and SupportPodPidsLimit (the ability to limit PIDs per Pod) have both reached GA.
alpha Feature: Graceful Node Shutdown
Users and cluster administrators expect Pods to follow the expected Pod lifecycle, including Pod termination. However, when a node shuts down, Pods do not follow the expected Pod termination lifecycle and are not terminated gracefully, which may cause issues for workloads. Kubernetes 1.20 introduced the alpha GracefulNodeShutdown feature, which enables the kubelet to detect node system shutdown events and gracefully terminate Pods during system shutdown.
CSIVolumeFSGroupPolicy Enters beta
CSIDrivers can use the fsGroupPolicy field to control whether ownership and permissions are modified at mount time. (ReadWriteOnceWithFSType, File, None)
ConfigurableFSGroupPolicy Enters beta
Supports non-recursive fsgroup setting - PodFSGroupChangePolicy = OnRootMismatch
Other Updates
Add the cloud controller manager component.
Features Graduating to GA:
RuntimeClassnode.k8s.io/v1beta1 is deprecated. Use node.k8s.io/v1 instead. SCTPSupport
Windows containerd support
Ingress
networking.k8s.io/v1beta1 is deprecated and will be removed in version 1.22. Use networking.k8s.io/v1 instead.
seccomp
The seccomp-related annotations seccomp.security.alpha.kubernetes.io/pod and container.seccomp.security.alpha.kubernetes.io/... are deprecated and will be removed in version 1.22. You can specify the following fields directly in the Pod and container specs:
securityContext:
seccompProfile:
type: RuntimeDefault|Localhost|Unconfined
localhostProfile: my-profiles/profile-allow.json
K8s automatically converts annotations and fields, so no additional action is required for the conversion.
All features related to Node authentication modes are implemented.
The Event API was redesigned to mitigate the impact of Events on system performance and to add more fields for providing more useful information. This work was completed in version 1.19.
In addition to the previous certificates.k8s.io/v1beta1, a new version certificates.k8s.io/v1 is added for CertificateSigningRequest. When certificates.k8s.io/v1 is used,
You must specify spec.signerName and must no longer use kubernetes.io/legacy-unknown.
You must specify spec.usages. It cannot contain duplicate values, and its values must be known usages only.
You must specify status.conditions[*].status.
status.certificate must be PEM-encoded and contain only CERTIFICATE blocks.
Features Graduating to Beta:
The following features enter the Beta stage and are enabled by default.
EndpointSliceProxying
kube-proxy reads information from EndpointSlices instead of Endpoints, which greatly improves scalability for large clusters and facilitates adding new features in the future, such as topology-aware routing.
KubeSchedulerConfiguration
HugePageStorageMediumSize
ImmutableEphemeralVolumes
Secret and ConfigMap volumes can be marked as immutable, which greatly reduces the pressure on the API server when there are a large number of Secret and ConfigMap volumes.
NodeDisruptionExclusion
NonPreemptingPriority
ServiceNodeExclusion
ServiceAccountIssuerDiscovery
Deprecation and Removal
Deprecated Versions
|
apiextensions.k8s.io/v1beta1 | apiextensions.k8s.io/v1 |
apiregistration.k8s.io/v1beta1 | apiregistration.k8s.io/v1 |
authentication.k8s.io/v1beta1 | authentication.k8s.io/v1 |
authorization.k8s.io/v1beta1 | authorization.k8s.io/v1 |
autoscaling/v2beta1 | autoscaling/v2beta2 |
coordination.k8s.io/v1beta1 | coordination.k8s.io/v1 |
storage.k8s.io/v1beta1 | storage.k8s.io/v1 |
kube-apiserver
1. The componentstatus API is deprecated. This API is used to provide the health status of etcd, kube-scheduler, and kube-controller-manager, but it only works under the following conditions: these components run on the same node as the API server, and kube-scheduler and kube-controller-manager expose insecure health check ports.
After this API is deprecated, the health check of etcd is included in the health check of kube-apiserver, and kube-scheduler/kube-controller-manager can check their respective health check endpoints.
2. The apiserver no longer listens on non-secure ports.
The --address and --insecure-bind-address parameters can be set but have no effect. The --port and --insecure-port parameters can only be set to 0. These parameters will be removed in version 1.24.
3. TokenRequest and TokenRequestProjection are now GA, and kube-apiserver requires the following parameters to be set:
--service-account-issuer identifies the fixed URL of the cluster API Server.
--service-account-key-file specifies one or more public keys for verifying tokens.
--service-account-signing-key-file specifies the private key for signing service accounts, which can use the same file as the --service-account-private-key-file parameter of kube-controller-manager.
kubelet
1. The following parameters are removed:
--seccomp-profile-root
Use config to replace --cloud-provider and --cloud-config.
--really-crash-for-testing , --chaos-chance
2. The deprecated metrics/resource/v1alpha1 endpoint has been removed. Use metrics/resource instead.
Other Removals
The failure-domain.beta.kubernetes.io/zone and failure-domain.beta.kubernetes.io/region labels are deprecated. Use topology.kubernetes.io/zone and topology.kubernetes.io/region instead. All labels with the failure-domain.beta... prefix must be replaced with the corresponding labels starting with topology....
PodPreset is removed. You can use a webhook to implement this feature.
basic auth authentication is no longer supported.
Directly mounting Tencent Cloud CBS storage (cbs inline) in workloads is no longer supported.
Note:
Successful mounting of CSI ephemeral volumes (csi inline) cannot be guaranteed during the upgrade from version 1.18 to version 1.20. If your business uses CSI ephemeral volumes, convert them to persistent volume storage before upgrading. Changelogs
1.18 changes since 1.16
Major Updates
cloud provider Tag Feature Reaches General Availability (GA)
The deprecated labels and their replacements are listed in the following table:
|
beta.kubernetes.io/instance-type
| node.kubernetes.io/instance-type
|
failure-domain.beta.kubernetes.io/region
| topology.kubernetes.io/region
|
failure-domain.beta.kubernetes.io/zone
| topology.kubernetes.io/zone
|
Volume Snapshot Feature Enters Beta
CSI Migration Enters Beta
Kubernetes Topology Manager Reaches Beta
The TopologyManager feature went to Beta in version 1.18. It enables NUMA alignment between CPUs and other devices such as SR-IOV-VF, allowing workloads to support low-latency scenarios.
Before the introduction of TopologyManager, the CPU manager and device manager could only make resource allocation decisions independently. This could lead to suboptimal resource allocation on multi-socket CPU systems, affecting the performance of latency-sensitive applications.
Serverside Apply Enters Beta 2
Server-side Apply was promoted to Beta in Kubernetes 1.16. Version 1.18 introduced a second Beta version (ServerSideApply), which records and manages changes to all new Kubernetes object fields, ensuring that users are aware of resource dynamics.
IngressClass Resource
The IngressClass resource describes a type of Ingress controller within a Kubernetes cluster. The Ingress resource introduces the ingressClassName field, which specifies the name of the controller that uses IngressClass, replacing the deprecated kubernetes.io/ingress.class annotation.
Other Updates
Node Local DNSCache has reached GA.
IPv6 has entered the Beta stage.
kubectl debug is an Alpha feature.
Windows CSI support is an Alpha feature.
ImmutableEphemeralVolumes is an Alpha feature that supports immutable ConfigMaps and Secrets without refreshing the corresponding volumes.
The following features have reached GA:
ScheduleDaemonSetPods
TaintNodesByCondition
WatchBookmark
NodeLease
CSINodeInfo
VolumeSubpathEnvExpansion
AttachVolumeLimit
ResourceQuotaScopeSelectors
VolumePVCDataSource
TaintBasedEvictions
BlockVolume,CSIBlockVolume
Windows RunAsUserName
Features Graduating to Beta:
EndpointSlices: disabled by default
CSIMigrationAWS: disabled by default
StartupProbe
EvenPodsSpread
Deprecation and Removal
Removed Features
The following features are removed. They were enabled by default and could not be configured:
GCERegionalPersistentDisk
EnableAggregatedDiscoveryTimeout
PersistentLocalVolumes
CustomResourceValidation
CustomResourceSubresources
CustomResourceWebhookConversion
CustomResourcePublishOpenAPI
CustomResourceDefaulting
Other Removals
Remove the built-in cluster role:
system:csi-external-provisioner
system:csi-external-attacher
Deprecated Feature Gates and Parameters
The default service IP address CIDR (10.0.0.0/24) is deprecated. You must specify a CIDR by using the --service-cluster-ip-range parameter of kube-apiserver.
The API groups rbac.authorization.k8s.io/v1alpha1 and rbac.authorization.k8s.io/v1beta1 are deprecated and will be removed in version 1.20. Migrate to rbac.authorization.k8s.io/v1.
The CSINodeInfo feature is deprecated. It has reached GA and is enabled by default.
Parameter and Other Changes
kube-apiserver
--encryption-provider-config: If cacheSize: 0 is specified in the configuration file, versions before 1.18 automatically set the cache to 1000 keys, while version 1.18 reports a configuration validation error. You can disable the cache by setting cacheSize to a negative value.
--feature-gates: The following features are enabled by default and can no longer be configured through the command line.
PodPriority
TaintNodesByCondition
ResourceQuotaScopeSelectors
ScheduleDaemonSetPods
The following resource versions (group versions) are no longer supported:
Use apps/v1 instead of apps/v1beta1 and apps/v1beta2.
Among them, under extensions/v1beta1:
For daemonsets, deployments, and replicasets, use apps/v1.
For networkpolicies, use networking.k8s.io/v1.
For podsecuritypolicies, use policy/v1beta1.
kubelet
--enable-cadvisor-endpoints: This parameter is disabled by default. To access the cAdvisor v1 JSON API, you must explicitly enable it.
The --redirect-container-streaming parameter is deprecated and will be removed in a future release. In v1.18, only the default behavior is supported, which proxies streaming requests through the kubelet. If --redirect-container-streaming=true is set, you must remove it.
The metrics endpoint /metrics/resource/v1alpha1 is deprecated. Use /metrics/resource instead.
kube-proxy
The following parameters are deprecated:
The --healthz-port parameter is deprecated. Use --healthz-bind-address instead.
The --metrics-port parameter is deprecated. Use --metrics-bind-address instead.
Add the feature gate EndpointSliceProxying (disabled by default) to control whether EndpointSlices are enabled in kube-proxy. The EndpointSlice feature gate no longer affects kube-proxy behavior.
Add the following parameters to configure the timeout for ipvs connections:
--ipvs-tcp-timeout
--ipvs-tcpfin-timeout
--ipvs-udp-timeout
The iptables mode now supports IPv4/IPv6 dual-stack.
kube-scheduler
The scheduling_duration_seconds metric is deprecated:
The scheduling_algorithm_predicate_evaluation_seconds metric is deprecated. Use framework_extension_point_duration_seconds[extension_point="Filter"] instead.
The scheduling_algorithm_priority_evaluation_seconds metric is deprecated. Use framework_extension_point_duration_seconds[extension_point="Score"] instead.
Deprecate AlwaysCheckAllPredicates in the scheduler Policy API.
-enable-profiling Parameter
profiling is enabled by default to align with kube-apiserver, kube-controller-manager, and kube-scheduler. To disable profiling, specify the --enable-profiling=false parameter.
kubectl
Remove the deprecated --include-uninitialized parameter.
kubectl and k8s.io/client-go no longer use http://localhost:8080 as the default apiserver address.
kubectl run supports creating Pods, but no longer supports using the previously deprecated generator to create other types of resources.
Remove the deprecated kubectl rolling-update command. Use the rollout command instead.
--dry-run supports three parameter values: client, server, and none.
--dry-run=server supports the following commands: apply, patch, create, run, annotate, label, set, autoscale, drain, rollout undo, and expose.
The new kubectl alpha debug command can attach temporary containers to a Pod for debugging and troubleshooting, which requires enabling the EphemeralContainers feature introduced in 1.16.
hyperkube
Hyperkube has been changed from Go code to a bash script.
Changelogs
1.16 changes since 1.14
Major Updates
Cluster Stability and Availability Improvements
Production-ready features such as bare metal cluster tooling and HA have been improved and strengthened.
kubeadm support for HA has entered the beta stage. Users can use the kubeadm init and kubeadm join commands to deploy a highly available control plane. Certificate management is more stable and robust, and kubeadm can seamlessly renew all certificates before they expire when updating a cluster. For details, see pr357 and pr970. CSI Continuous Improvements
The Storage SIG continues to work on migrating in-tree storage plugins to CSI interfaces, supporting features such as resizing of in-tree storage plugins and inline volumes. It also introduces some alpha features that did not previously exist in the Kubernetes storage subsystem, such as volume cloning.
Volume cloning allows users to specify another PVC as the "DataSource" when a new volume is provisioned. If the underlying storage system supports this feature and implements the "CLONE_VOLUME" capability in its CSI driver, the new volume becomes a clone of the source volume. For details, see pr625. Feature
Features Graduating to GA:
CRD
Admission Webhook
GCERegionalPersistentDisk
CustomResourcePublishOpenAPI
CustomResourceSubresources
CustomResourceValidation
CustomResourceWebhookConversion
The feature in the CSI specification that supports volume resizing has entered Beta.
General Updates
The Kubernetes core code supports Go modules.
Continue preparations for extracting and organizing the cloud-provider code. The cloud-provider code has been moved to kubernetes/legacy-cloud-providers to facilitate subsequent deletion and external use.
Release a new alpha scheduling framework for developing and managing plugins to extend scheduler functionality. For details, see pr624. Continue deprecating the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 APIs. These extensions will be completely removed in version 1.16.
Kubelet introduces the Topology Manager component to coordinate resource allocation decisions and optimize resource allocation.
Supports IPv4/IPv6 dual-stack networking and can assign both IPv4 and IPv6 addresses to Pods and services simultaneously.
alpha feature: API Server network proxy.
Provide more extension options for cloud controller manager migration.
Deprecate the extensions/v1beta1, apps/v1beta1, and apps/v1beta2 APIs.
Known Issues
In version 1.15, using the --log-file parameter causes logs to be written to the same file multiple times. For details, see pr78734. Update Notes
Cluster
The following tags are no longer set on nodes: beta.kubernetes.io/metadata-proxy-ready, beta.kubernetes.io/metadata-proxy-ready, and beta.kubernetes.io/kube-proxy-ds-ready.
ip-mask-agent uses node.kubernetes.io/masq-agent-ds-ready as the node selector instead of beta.kubernetes.io/masq-agent-ds-ready.
kube-proxy uses node.kubernetes.io/kube-proxy-ds-ready as the node selector instead of beta.kubernetes.io/kube-proxy-ds-ready.
metadata-proxy uses cloud.google.com/metadata-proxy-ready as the node selector instead of beta.kubernetes.io/metadata-proxy-ready.
API Machinery
k8s.io/kubernetes and other published components, including k8s.io/client-go and k8s.io/api, now include Go module files that contain version information for dependent libraries. When using k8s.io/client-go as a Go module, refer to go-modules and pr74877. The Appshyperkube short aliases have been removed from the code. These aliases are created when the hyperkube docker image is built. For details, see pr76953. Lifecycle
All deprecated kubeadm v1alpha3 configurations have been removed.
kube-up.sh no longer supports centos and local.
Storage
The Node.Status.Volumes.Attached.DevicePath field is no longer set for CSI volumes, and external controllers that update this field need to be updated.
alpha CRDs have been removed.
The StorageObjectInUseProtection admission plugin is enabled by default. If this plugin was not enabled previously, cluster behavior may change. After a CSI driver enables PodInfoOnMount, a new parameter csi.storage.k8s.io/ephemeral is added to the volume context, allowing the driver to determine whether each volume is ephemeral or persistent when the driver implements NodePublishVolume. For details, see pr79983. VolumePVCDataSource (the volume cloning feature) has entered beta. For details, see pr81792. Combine the limits of built-in and CSI volumes into a single scheduler predicate. For details, see pr77595. kube-apiserver
The --enable-logs-handler parameter is deprecated and is planned for removal in v1.19.
The --basic-auth-file parameter and its corresponding authentication mode are deprecated and are planned for removal in the future.
The default service IP address CIDR (10.0.0.0/24) is deprecated and is planned for removal in six months or after two releases. You must use the --service-cluster-ip-range parameter to specify the service IP address range.
kube-scheduler
Use the v1beta1 Events API. Tools that consume scheduler events need to use the v1beta1 Event API.
kube-proxy
Remove the --conntrack-max parameter. You can use --conntrack-min and --conntrack-max-per-core instead.
Remove the --cleanup-iptables parameter.
Remove the --resource-container parameter.
kubelet
Remove the --allow-privileged, --host-ipc-sources, --host-pid-sources, and --host-network-sources parameters. You can use the PodSecurityPolicy admission controller instead.
Deprecate the cAdvisor JSON interface.
Remove the --containerized parameter.
Setting disallowed labels with the kubernetes.io- or k8s.io- prefix through the --node-labels parameter is no longer supported.
kubectl
Remove kubectl scale job.
Remove the --pod/-p parameter from the kubectl exec command.
Remove the kubectl convert command.
Remove the --include-uninitialized parameter.
kubectl cp no longer supports copying symbolic links from containers. You can use the following command instead:
local to pod:tar cf - /tmp/foo | kubectl exec -i -n <some-namespace> <some-pod> -- tar xf - -C /tmp/bar
pod to local:kubectl exec -n <some-namespace> <some-pod> -- tar cf - /tmp/foo | tar xf - -C /tmp/bar
kubeadm
Deprecate the kubeadm upgrade node config and kubeadm upgrade node experimental-control-plane commands. Use kubeadm upgrade node instead.
Deprecate the --experimental-control-plane parameter. Use --control-plane instead.
Deprecate the --experimental-upload-certs parameter. Use --upload-certs instead.
Deprecate the kubeadm config upload command. Use kubeadm init phase upload-config instead.
CoreDNS uses the ready plugin to perform readiness checks.
Deprecate the proxy plugin. Use the forward plugin instead.
The kubernetes plugin removes the resyncperiod option.
The upstream option is deprecated and will be ignored if specified.
Changelogs
1.14 changes since 1.12
Major Updates
Use kubeadm to simplify cluster management.
General Updates
dry-run goes beta. It allows users to simulate real API requests without actually changing the cluster state.
kubectl diff goes beta.
kubectl plugin registration goes stable.
The kubelet plugin mechanism goes beta.
CSIPersistentVolume goes GA.
TaintBasedEviction goes beta.
kube-scheduler can perceive volume topology and goes stable.
Support for out-of-tree CSI Volume plugins goes stable.
Support third-party device monitoring plugins.
kube-scheduler subnet feasibility goes beta.
Pod Ready supports custom probe conditions.
Node memory supports HugePages.
RuntimeClass goes beta.
Node OS/Arch labels have reached GA.
node-leases goes beta.
The kubelet resource metrics endpoint goes alpha and supports data collection through prometheus.
runAsGroup goes beta.
kubectl apply server-side goes alpha, allowing you to perform apply operations on the server side.
kubectl supports kustomize.
You can configure resolv.conf in a Pod.
resizing of CSI volumes is supported.
CSI supports topology.
volume mounts support configuring subpath parameters.
CSI supports raw block devices.
CSI supports local ephemeral volumes.
Update Notes
kube-apiserver
etcd2 is no longer supported. --storage-backend=etcd3 is used by default.
Deprecate the --etcd-quorum-read parameter.
Deprecate the --storage-versions parameter.
Deprecate the --repair-malformed-updates parameter.
kube-controller-manager
Deprecate the --insecure-experimental-approve-all-kubelet-csrs-for-group parameter.
kubelet
Deprecate the --google-json-key parameter.
Deprecate the --experimental-fail-swap-on parameter.
kube-scheduler
componentconfig/v1alpha1 is no longer supported.
kubectl
The run-container command is no longer supported.
taints
node.alpha.kubernetes.io/notReady and node.alpha.kubernetes.io/unreachable are no longer supported. Use node.kubernetes.io/not-ready and node.kubernetes.io/unreachable instead.
Changelogs
1.12 changes since 1.10
Major Updates
API
The CustomResources subresource is now in beta and enabled by default. You can update fields other than .status in the /status subresource, whereas previously only .spec and .metadata could be updated. When the /status subresource is enabled, required and description can be used in the CRD OpenAPI validation schema. In addition, users can create multiple versions of CustomResourceDefinitions without automatic conversion. You can use the spec.additionalPrinterColumns field of CustomResourceDefinitions to include additional columns in the output of kubectl get.
The dry run feature is supported, allowing users to see the execution results of certain commands without actually committing the related changes.
Authentication authorization
RBAC aggregated ClusterRoles graduate to GA, and the client-go credentials plugin graduates to Beta, supporting the retrieval of TLS authentication information from external plugins.
The following annotations are added to audit events, allowing users to understand the audit decision process more clearly:
The Authorization component sets authorization.k8s.io/decision, which indicates whether the authorization decision is allow or forbid, and authorization.k8s.io/reason, which indicates the reason for this decision.
The PodSecurityPolicy admission controller sets podsecuritypolicy.admission.k8s.io/admit-policy and podsecuritypolicy.admission.k8s.io/validate-policy, which contain the names of policies related to Pod admission. PodSecurityPolicy can also restrict hostPath mounts to read-only mode.
The NodeRestriction admission controller prevents nodes from modifying the taint information of their corresponding Node objects, making it easier for users to control and track taint settings on nodes.
CLI Command Line
The CLI implements a new plugin mechanism and provides a development library containing common CLI tools to facilitate plugin development for plugin developers.
Network
The ipvs mode has reached GA.
CoreDNS has reached GA, replacing kube-dns.
Node
DynamicKubeletConfig enters the Beta stage.
cri-tools GA.
PodShareProcessNamespace enters the Beta stage.
New Alpha features: RuntimeClass and CustomCFSQuotaPeriod.
Scheduler
Pod Priority and Preemption enter the Beta stage.
The scheduling of DaemonSet Pods is no longer managed by the DaemonSet controller but by the default scheduler.
TaintNodeByCondition enters the Beta stage.
The local image preference feature is enabled by default. When Pods are scheduled, nodes that have already pulled all or part of the required images locally are assigned a higher priority, which accelerates Pod startup.
General Updates
Features that have reached GA: ClusterRole and StorageObjectInUseProtection.
Features that have entered Beta: external Cloud Provider.
Update Notes
kube-apiserver
The --storage-version parameter is removed and replaced by --storage-versions. Meanwhile, --storage-versions is also deprecated.
The default value of --endpoint-reconciler-type is changed to lease.
It is included by default when --enable-admission-plugins is used. It must be explicitly specified when the --admission-control parameter is used.
kubelet
Deprecate the --rotate-certificates parameter. Use the .RotateCertificates field in the configuration file instead.
kubectl
Except for run-pod/v1, other generators of kubectl run are deprecated.
Remove the --interactive parameter from kubectl logs.
--use-openapi-print-columns is deprecated and replaced by --server-print.
Changelogs