NetworkPolicy 是 K8S 提供的一种资源，用于定义基于 Pod 的网络隔离策略。它描述了一组 Pod 能否与其它组 Pod 及其它 Endpoints 进行通信。本文主要演示如何使用开源工具 felix 或 kube-router 在 CCE 上实现 NetworkPolicy 功能。
用户可以根据集群的容器网络模式，选择对应的组件进行部署。
felix
注意： felix 仅能搭配 veth 网络模式使用（详见 “VPC 网络”模式高级选项）
felix 是开源容器网络方案 Calico 的一个组件，运行在每个节点上负责配置路由及ACL等信息。
- 官网: https://docs.projectcalico.org/reference/felix/
- 项目: https://github.com/projectcalico/felix
CCE 基于 felix 进行修改和适配，实现了容器网络策略功能。
部署 felix
在 CCE K8S 集群上部署 felix，YAML 如下：
---
# Source: calico-felix/templates/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: cce-calico-felix
namespace: kube-system
---
# Source: calico-felix/templates/cce-reserved.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: calico-felix-cce-reserved
namespace: kube-system
labels:
heritage: Helm
release: RELEASE-NAME
chart: calico-felix-1.0.0
app: cce-calico-felix
data:
hash: "22ec24f7bfe36fe18917ff07659f9e6e3dfd725af4c3371d3e60c7195744bea4"
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: felixconfigurations.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: FelixConfiguration
plural: felixconfigurations
singular: felixconfiguration
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: bgpconfigurations.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: BGPConfiguration
plural: bgpconfigurations
singular: bgpconfiguration
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ippools.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: IPPool
plural: ippools
singular: ippool
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: hostendpoints.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: HostEndpoint
plural: hostendpoints
singular: hostendpoint
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: clusterinformations.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: ClusterInformation
plural: clusterinformations
singular: clusterinformation
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: globalnetworkpolicies.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: GlobalNetworkPolicy
plural: globalnetworkpolicies
singular: globalnetworkpolicy
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: globalnetworksets.crd.projectcalico.org
spec:
scope: Cluster
group: crd.projectcalico.org
version: v1
names:
kind: GlobalNetworkSet
plural: globalnetworksets
singular: globalnetworkset
---
# Source: calico-felix/templates/crd.yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: networkpolicies.crd.projectcalico.org
spec:
scope: Namespaced
group: crd.projectcalico.org
version: v1
names:
kind: NetworkPolicy
plural: networkpolicies
singular: networkpolicy
---
# Source: calico-felix/templates/rbac.yaml
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cce-calico-felix
rules:
- apiGroups: [""]
resources: ["pods", "nodes", "namespaces", "configmaps", "serviceaccounts"]
verbs: ["get", "watch", "list", "update"]
- apiGroups: ["networking.k8s.io"]
resources:
- networkpolicies
verbs:
- get
- list
- watch
- apiGroups: ["extensions"]
resources:
- networkpolicies
verbs:
- get
- list
- watch
- apiGroups: [""]
resources:
- pods/status
verbs:
- update
- apiGroups: ["crd.projectcalico.org"]
resources: ["*"]
verbs: ["*"]
---
# Source: calico-felix/templates/rbac.yaml
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: cce-calico-felix
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: cce-calico-felix
subjects:
- kind: ServiceAccount
name: cce-calico-felix
namespace: kube-system
---
# Source: calico-felix/templates/daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: cce-calico-felix
namespace: kube-system
spec:
selector:
matchLabels:
app: cce-calico-felix
template:
metadata:
labels:
app: cce-calico-felix
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
spec:
hostPID: true
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- key: node.cloudprovider.kubernetes.io/uninitialized
value: "true"
effect: NoSchedule
- key: node-role.kubernetes.io/master
effect: NoSchedule
- key: "CriticalAddonsOnly"
operator: "Exists"
terminationGracePeriodSeconds: 0
serviceAccountName: cce-calico-felix
hostNetwork: true
containers:
- name: policy
image: registry.baidubce.com/cce-plugin-pro/cce-calico-felix:v3.5.8
command: ["/bin/policyinit.sh"]
imagePullPolicy: Always
env:
- name: NODENAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: FELIX_INTERFACEPREFIX
value: veth
securityContext:
privileged: true
resources:
requests:
cpu: 250m
livenessProbe:
httpGet:
path: /liveness
port: 9099
host: localhost
periodSeconds: 10
initialDelaySeconds: 10
failureThreshold: 6
readinessProbe:
httpGet:
path: /readiness
port: 9099
host: localhost
periodSeconds: 10
volumeMounts:
- mountPath: /lib/modules
name: lib-modules
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-proxy-config
namespace: kube-system
labels:
app: kube-proxy-config
spec:
selector:
matchLabels:
app: kube-proxy-config
template:
metadata:
labels:
app: kube-proxy-config
spec:
nodeSelector:
beta.kubernetes.io/arch: amd64
tolerations:
- operator: "Exists"
restartPolicy: Always
hostNetwork: true
containers:
- name: busybox
image: busybox
command:
- sh
- /tmp/update-proxy-yaml.sh
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
imagePullPolicy: IfNotPresent
volumeMounts:
- name: etc-k8s
mountPath: /etc/kubernetes/
- name: shell
mountPath: /tmp/
terminationGracePeriodSeconds: 0
volumes:
- name: etc-k8s
hostPath:
path: /etc/kubernetes/
type: "DirectoryOrCreate"
- name: shell
configMap:
name: update-proxy-yaml-shell
optional: true
items:
- key: update-proxy-yaml.sh
path: update-proxy-yaml.sh
---
apiVersion: v1
kind: ConfigMap
metadata:
labels:
addonmanager.kubernetes.io/mode: EnsureExists
name: update-proxy-yaml-shell
namespace: kube-system
data:
update-proxy-yaml.sh: |-
#!/bin/sh
if [[ -e /etc/kubernetes/proxy.yaml ]]; then
sed -i 's/masqueradeAll: true/masqueradeAll: false/g' /etc/kubernetes/proxy.yaml
if grep -q "masqueradeAll: false" /etc/kubernetes/proxy.yaml; then
echo "update config successfully"
else
exit 1
fi
else
echo "/etc/kubernetes/proxy.yaml not exists"
exit 1
fi
sleep infinity
kube-router
注意： kube-router 仅能搭配 kubenet 网络模式使用（详见 “VPC 网络”模式高级选项）
kube-router 是一个 kubernetes 的容器网络解决方案，它的官网和代码地址如下：
- 官网：https://www.kube-router.io
- 项目：https://github.com/cloudnativelabs/kube-router
kube-router 有三大功能：
- Pod Networking；
- IPVS/LVS based service proxy；
- Network Policy Controller.
CCE 有自己的容器网络实现方案，本文主要使用 kube-router 的 Network Policy Controller 的功能.
部署 kube-router
在 CCE K8S 集群上部署 kube-router ，YAML 如下：
apiVersion: v1
kind: ServiceAccount
metadata:
name: kube-router
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kube-router
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- namespaces
- pods
- services
- nodes
- endpoints
verbs:
- list
- get
- watch
- apiGroups:
- "networking.k8s.io"
resources:
- networkpolicies
verbs:
- list
- get
- watch
- apiGroups:
- extensions
resources:
- networkpolicies
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: kube-router
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kube-router
subjects:
- kind: ServiceAccount
name: kube-router
namespace: kube-system
---
apiVersion: v1
kind: ConfigMap
metadata:
name: kube-router-cfg
namespace: kube-system
labels:
tier: node
k8s-app: kube-router
data:
cni-conf.json: |
{
"name":"kubernetes",
"type":"bridge",
"bridge":"kube-bridge",
"isDefaultGateway":true,
"ipam": {
"type":"host-local"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-router
namespace: kube-system
labels:
k8s-app: kube-router
spec:
selector:
matchLabels:
k8s-app: kube-router
template:
metadata:
labels:
k8s-app: kube-router
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
serviceAccountName: kube-router
containers:
- name: kube-router
image: registry.baidubce.com/cce-plugin-pro/kube-router:latest
args: ["--run-router=false", "--run-firewall=true", "--run-service-proxy=false"]
securityContext:
privileged: true
imagePullPolicy: Always
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
livenessProbe:
httpGet:
path: /healthz
port: 20244
initialDelaySeconds: 10
periodSeconds: 3
volumeMounts:
- name: lib-modules
mountPath: /lib/modules
readOnly: true
- name: cni-conf-dir
mountPath: /etc/cni/net.d
initContainers:
- name: install-cni
image: registry.baidubce.com/cce-plugin-pro/kube-router-busybox:latest
imagePullPolicy: Always
command:
- /bin/sh
- -c
- set -e -x;
if [ ! -f /etc/cni/net.d/10-kuberouter.conf ]; then
TMP=/etc/cni/net.d/.tmp-kuberouter-cfg;
cp /etc/kube-router/cni-conf.json ${TMP};
mv ${TMP} /etc/cni/net.d/10-kuberouter.conf;
fi
volumeMounts:
- name: cni-conf-dir
mountPath: /etc/cni/net.d
- name: kube-router-cfg
mountPath: /etc/kube-router
hostNetwork: true
tolerations:
- key: CriticalAddonsOnly
operator: Exists
- effect: NoSchedule
key: node-role.kubernetes.io/master
operator: Exists
- effect: NoSchedule
key: node.kubernetes.io/not-ready
operator: Exists
volumes:
- name: lib-modules
hostPath:
path: /lib/modules
- name: cni-conf-dir
hostPath:
path: /etc/cni/net.d
- name: kube-router-cfg
configMap:
name: kube-router-cfg
例子说明
1 创建namespaces
$kubectl create namespace production
$kubectl create namespace staging
2 启动 nginx 服务
在不同的 namespace 中创建 nginx deployment.
$kubectl apply -f nginx.yaml --namespace=production
$kubectl apply -f nginx.yaml --namespace=staging
nginx.yaml 的 YAML 如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: hub.baidubce.com/cce/nginx-alpine-go:latest
ports:
- containerPort: 80
验证 Pod 启动成功：
# staging 环境
$kubectl get pods -n staging
NAME READY STATUS RESTARTS AGE
nginx-deployment-7fbd5f4c55-2xgd4 1/1 Running 0 45s
nginx-deployment-7fbd5f4c55-5xr75 1/1 Running 0 45s
nginx-deployment-7fbd5f4c55-fn6lr 1/1 Running 0 20m
# productionn 环境
$kubectl get pods -n production
NAME READY STATUS RESTARTS AGE
nginx-deployment-7fbd5f4c55-m764f 1/1 Running 0 10s
nginx-deployment-7fbd5f4c55-pdhhz 1/1 Running 0 10s
nginx-deployment-7fbd5f4c55-r98w5 1/1 Running 0 20m
没有设置 NetworkPolicy 的时候，所有的 Pod 是可以相互访问的，可以直接 ping PodIP.
Network Policy 策略测试
1. Default deny all ingress traffic
禁止 namespace=staging 中 pod 被访问.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: staging
spec:
podSelector: {}
policyTypes:
- Ingress
各个字段含义说明：
- PodSelector：选中需要隔离的 Pod；
- policyTypes： 策略类型，NetworkPolicy 将流量分为 ingress 和 egress，即入方向和出方向。如果没有指定则表示不闲置；
- ingress：入方向，白名单，需要指定 from、ports，即来源、目的端口号，from有三种类型，ipBlock/namespaceSelector/podSelector；
- egress：出方向，白名单，类似 ingress，egress 需要指定 to、ports，即目的、目的端口号。
上述 NetworkPolicy 创建完成后，可以在任意 Pod 中访问 namespace=staging 下的 PodIP，发现是无法访问，比如从 production 中的 pod 进行访问 ：
$kubectl exec -it nginx-deployment-7fbd5f4c55-m764f /bin/sh -n production
/ # ping 172.16.0.92
PING 172.16.0.92 (172.16.0.92): 56 data bytes
2. Default allow all ingress traffic
允许 namespace=staging 中 pod 被访问.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all
namespace: staging
spec:
podSelector: {}
ingress:
- {}
policyTypes:
- Ingress
3. Default deny all egress traffic
禁止 namespace=production 中 pod 对外访问.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: production
spec:
podSelector: {}
policyTypes:
- Egress
4. Default allow all egress traffic
允许 namespace=production 中 pod 对外访问.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: allow-all
namespace: production
spec:
podSelector: {}
egress:
- {}
policyTypes:
- Egress
5. Default deny all ingress and all egress traffic
禁止所有 pod 的入和出的流量：
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
