릴리스 노트
제품 릴리스 기록
Using SecurityGroupPolicy for Pods on TKE
포커스 모드
폰트 크기
Description
This article describes how users can use the SecurityGroupPolicy (abbreviation SGP) on TKE general nodes and native nodes.
In TKE Serverless clusters, SGP support is already provided. For details, see Pod Security Group. However, due to network capability limitations (Multiple pods with shared ENI mode), this feature was previously unavailable for general nodes and native nodes in TKE. This time, the TKE network component has undergone a comprehensive upgrade. By leveraging Trunking ENI feature provided by the VPC, Pods are allocated separate ENIs from the relay network. These network interfaces are individually bound to security groups, thereby achieving the SGP capability for Pods running on TKE.
Solution Strengths
Support security group binding at the Pod level.
Pods occupy IPs individually, static IPs can be scheduled freely across AZ nodes which further enhancing scheduling flexibility.
Trunking ENI do not consume the quota of secondary IP resources and exclusive ENI resources.
Use Cases
Reduce exposed surface, thereby decreasing the likelihood of network attacks.
By using security groups, Pods can configure their minimum exposed surface, thereby reducing possible attack surface and enhancing security.
Network security policy control with the smallest granularity.
Pod binding to security groups implements network security policy control with the smallest granularity, enabling more precise control of network security policies for each application, thereby enhancing overall cluster security.
Migrate from virtual machine architecture to container architecture and reuse existing security groups of applications.
In traditional virtual machine architecture, applications are deployed on virtual machines, and security groups are bound to the VMs. After migrating to container architecture, with the SGP feature, these security groups can be directly reused.
Features
Support coexistence with the shared ENI mode.
When the eniipamd component enables the security group feature, the security group setting of the Pod has a higher priority, followed by the security group logic of eniipamd, inheriting the security group settings of the node (from the primary ENI or according to the node settings).
By default, Pods on the same node do not go through security groups to communicate with each other.
Limitations
The number of Pods using Trunking ENIs on a node cannot exceed the node's Trunking ENI quota, which is currently 100 by default, and a maximum of 256 is supported for 64C models.
Currently, only VPC-CNI shared ENI + static IP mode clusters are supported.
Currently only the following models are supported by default: "ITA5", "M8", "MA4", "MA5", "S8", "S9", "SA4", "SA5". For other models added to the cluster, the Trunking ENI quota will be assigned to 0, prohibiting the use of standalone security groups for Pods. If you need to use other models, please submit a ticket to request support from the VPC.
Usage
Enable the Client-Token Allowlist Feature on the VPC Side
The creation of a Trunking ENI strongly depends on the client-token feature of the VPC. Currently, this feature is in the allowlist management stage. Please submit a ticket to the VPC side to apply for adding the user account to this allowlist.
Note:
If the account is not added to the VPC client-token allowlist, there may be a risk of resource leakage during the use of Trunking ENIs.
Trunking ENI Related API Authorization
1. On the CAM Console > Policies page, click Create a custom policy.
2. Select Create according to the policy syntax method:
2.1 Select Blank Template, click Next.
2.2 Policy Name can be "SubENIForIPAMD". Copy and create the following policies:
{"statement": [{"action": ["vpc:CreateSubNetworkInterface","vpc:DeleteSubNetworkInterface","cvm:CheckNetworkInterfaceSecurityGroupReady"],"effect": "allow","resource": ["*"]}],"version": "2.0"}
3. Click Complete.
4. After creation, in the CAM Console > Roles, search for "IPAMDofTKE_QCSRole". Click the role name, select Associate Policies in Permissions, check the created policy "SubENIForIPAMD", and click OK.
5. In the CAM Console > Roles, search for "TKE_QCSRole". Click the role name, select Associate Policies in Permissions, check the created policy "SubENIForIPAMD", and click OK.
Through this authorization, assign TKE and related tke-eni-ipamd components the permission to create and delete Trunking ENIs.
Enable Trunking ENI Feature By Default For Incremental Clusters
1. Submit a ticket to TKE to enable this feature by default for incremental clusters.
2. When creating a cluster, select vpc-cni shared ENI mode and enable static Pod IP to auto turn on the Trunking ENI feature.
Enabling Trunking ENI Feature For Existing Clusters
Conditions:
Cluster restrictions: VPC-CNI network mode, shared ENI, and static Pod IP enabled.
Version limits: 3.5.6 and above versions.
Directions are as follows:
1. Log in to the TKE console and select Cluster in the left sidebar.
2. On the Cluster page, select the target cluster name to enter the cluster details page.
3. Select Add-on management in the left sidebar, choose the eniipamd on the component page, click Modify Global Configuration on the right, change the "vpc-cni.trunking-eni.enable-sub-eni" configuration item to
true, then save and exit after modification.
Enable SGP Feature In The Cluster
After enabling the Trunking ENI for the cluster, you can use SGP on both general and native nodes. Directions:
1. Log in to the TKE console and select Cluster in the left sidebar.
2. On the Cluster page, select the target cluster name to enter the cluster details page.
3. Select Add-on management in the left sidebar, then click Create on the component page.
4. On the Create Add-on management page, select SecurityGroupPolicy. As shown below:
5. Click Done.
6. Please create the required security group for the Pod on the security group page, such as sg-xxxxyyyy.
7. Deploy the SecurityGroupPolicy CR as follows in the yaml file. If you prefer to select Pods based on service account labels, replace
podSelector with serviceAccountSelector. You must specify one type of selector. If multiple security groups are assigned, all rules in all security groups will apply to the selected Pods.apiVersion: vpcresources.tke.cloud.tencent.com/v1beta1kind: SecurityGroupPolicymetadata:name: my-security-group-policynamespace: my-namespacespec:podSelector:matchLabels:app: my-appsecurityGroups:groupIds:- sg-xxxxyyyy
All subsequently created Pods with the label
app: my-app will use Trunking ENI network capability and bind to the security group sg-xxxxyyyy.8. Enter the TKE console, deploy an application that needs to bind to a security group in the cluster workload. See the following yaml to deploy an nginx deployment:
apiVersion: apps/v1kind: Deploymentmetadata:name: nginxlabels:app: my-appspec:replicas: 1selector:matchLabels:app: my-apptemplate:metadata:annotations:labels:app: my-appspec:terminationGracePeriodSeconds: 0containers:- name: nginximage: nginx:1.7.9ports:- containerPort: 80
After successful deployment, you can check Pod information through
kubectl get pod -oyaml to confirm the Trunking ENI feature is already enabled and the corresponding security group is bound:
The key information in the red box is as follows:
# tke-sub-eni means the pod already used the Trunking ENItke.cloud.tencent.com/networks: "tke-sub-eni"# means the pod has been bound to a Pod-level security grouptke.cloud.tencent.com/security-group-id: "sg-xx1,sg-xx2"# extension resource, means the pod currently uses Trunking ENI resourcetke.cloud.tencent.com/sub-eni: "1"
Check whether the security group associated with the ENI used by the Pod meets expectations. First, obtain the ENI id currently used by the Pod with the following command:
kubectl -n <Pod namespace> get vipc <Pod name> -ojsonpath='{.metadata.labels.tke\\.cloud\\.tencent\\.com/eni-id}'
Enter the ENI console, search for the ENI id, enter its details page, and view its associated security groups:
피드백