通过 Multus 为容器添加附加网络

默认情况下，Kubernetes 管理 Pod 网络和 Service 网络，Pod 网络为每个 Pod 提供网络接口，用于 Pod 之间的网络通信。Service 网络是为 Pod 提供稳定的寻址。

Multus CNI 插件可以将 Pod 附加到自定义的网络，来提供特定流量的性能豁满足安全需要。

Multus 网络示意图：

在 Pod 中默认的网络接口名称为 eth0 ，通过 Multus 添加的网络接口名称为 netX 。

安装 Multus

如果使用现有的自定义网络，首先必须确保在集群节点上使网络可用。

安装 Multus 的难易程度取决于所使用的 K8S 发行版，例如 OpenShift 可以方便的从 Operator Hub 里面安装，Rancher 可以在创建集群的时候选择使用 Muluts 网络。

在这里使用的方式是通过 Rancher 部署 RKE2 集群时，选择了 mulus,calico 网络。

验证相关 Pod 是否运行正常：

[vagrant@master01 ~]$ kubectl get pods --all-namespaces | grep -i multus
kube-system           helm-install-rke2-multus-c5hbb                                   0/1     Completed   0              2d12h
kube-system           rke2-multus-bn9kn                                                1/1     Running     5 (53m ago)    2d12h
kube-system           rke2-multus-h54z4                                                1/1     Running     7 (52m ago)    2d4h
kube-system           rke2-multus-mklpk                                                1/1     Running     13 (52m ago)   2d4h

在每个节点上都自动生成了 /etc/cni/net.d/00-multus.conf 配置文件：

[vagrant@master01 ~]$ sudo cat /etc/cni/net.d/00-multus.conf
{
        "cniVersion": "0.3.1",
        "name": "multus-cni-network",
        "type": "multus",
        "capabilities": {"bandwidth":true,"portMappings":true},
        "kubeconfig": "/etc/cni/net.d/multus.d/multus.kubeconfig",
        "delegates": [
                {"cniVersion":"0.3.1","name":"k8s-pod-network","plugins":[{"container_settings":{"allow_ip_forwarding":false},"datastore_type":"kubernetes","ipam":{"assign_ipv4":"true","assign_ipv6":"false","type":"calico-ipam"},"kubernetes":{"k8s_api_root":"https://10.43.0.1:443","kubeconfig":"/etc/cni/net.d/calico-kubeconfig"},"log_file_max_age":30,"log_file_max_count":10,"log_file_max_size":100,"log_file_path":"/var/log/calico/cni/cni.log","log_level":"Info","mtu":0,"nodename_file_optional":false,"policy":{"type":"k8s"},"type":"calico"},{"capabilities":{"bandwidth":true},"type":"bandwidth"},{"capabilities":{"portMappings":true},"snat":true,"type":"portmap"}]}
        ]
}

创建 NetworkAttachmenDefinition

要创建附加接口，需要创建自定义资源用于定义针对接口的 CNI 配置。
这种资源称为 NetworkAttachmentDefinition 。

创建一个名为 example-nad 的 NetworkAttachmentDefinition 资源：

apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
  name: example-nad
  namespace: example-multus
spec:
  config: |-
    {
         "cniVersion": "0.3.1",
         "type": "ipvlan",
         "master": "eth1",
         "ipam": {
             "type": "static",
             "routes": [
               { "dst": "192.168.122.0/24", "gw": "192.168.122.1" }
             ]
         }
    }

中创建该资源并查看：

kubectl apply -f example-nad.yml
networkattachmentdefinition.k8s.cni.cncf.io/example-nad created

[vagrant@master01 ~]$ kubectl describe net-attach-def example-nad -n example-multus
Name:         example-nad
Namespace:    example-multus
Labels:       <none>
Annotations:  <none>
API Version:  k8s.cni.cncf.io/v1
Kind:         NetworkAttachmentDefinition
Metadata:
  Creation Timestamp:  2024-05-05T11:45:32Z
  Generation:          1
  Resource Version:    481874
  UID:                 413fcfb7-fbc9-44b0-83fe-730fb72c1e31
Spec:
  Config:  {
     "cniVersion": "0.3.1",
     "type": "ipvlan",
     "master": "eth1",
     "ipam": {
         "type": "static",
         "routes": [
           { "dst": "192.168.122.0/24", "gw": "192.168.122.1" }
         ]
     }
}
Events:  <none>

创建一个测试 Pod 来使用 example-nad 这个附加网络：

apiVersion: v1
kind: Pod
metadata:
  name: example-nad-pod
  namespace: example-multus
  annotations:
    k8s.v1.cni.cncf.io/networks: |
      [{
        "name": "example-nad",
        "namespace": "example-multus",
        "ips": ["192.168.122.188/24"]
      }]
spec:
  containers:
  - name: samplepod
    command: ["/bin/ash", "-c", "trap : TERM INT; sleep infinity & wait"]
    image: alpine

创建并查看状态：

[vagrant@master01 ~]$ kubectl create -f example-nad-pod.yml
pod/example-nad-pod created

[vagrant@master01 ~]$ k get pod -n example-multus
NAME              READY   STATUS    RESTARTS   AGE
example-nad-pod   1/1     Running   0          10s

[vagrant@master01 ~]$ kubectl describe pod example-nad-pod -n example-multus
Name:             example-nad-pod
Namespace:        example-multus
Priority:         0
Service Account:  default
Node:             worker01.example.com/192.168.121.27
Start Time:       Sun, 05 May 2024 13:05:38 +0000
Labels:           <none>
Annotations:      cni.projectcalico.org/containerID: 81dcd5467f0d8be6275f61c0b22329086c5bc18a7fa595031696bd200b6708a0
                  cni.projectcalico.org/podIP: 10.42.95.152/32
                  cni.projectcalico.org/podIPs: 10.42.95.152/32
                  k8s.v1.cni.cncf.io/network-status:
                    [{
                        "name": "k8s-pod-network",
                        "ips": [
                            "10.42.95.152"
                        ],
                        "default": true,
                        "dns": {}
                    },{
                        "name": "example-multus/example-nad",
                        "interface": "net1",
                        "ips": [
                            "192.168.122.188"
                        ],
                        "mac": "52:54:00:26:5a:9f",
                        "dns": {}
                    }]
                  k8s.v1.cni.cncf.io/networks:
                    [{
                      "name": "example-nad",
                      "namespace": "example-multus",
                      "ips": ["192.168.122.188/24"]
                    }]
Status:           Running
IP:               10.42.95.152
IPs:
  IP:  10.42.95.152
Containers:
  samplepod:
    Container ID:  containerd://ff28ad8ac418342ca212763184600e313b287bc8e92b386ef3e81de52f6f953c
    Image:         alpine
    Image ID:      docker.io/library/alpine@sha256:c5b1261d6d3e43071626931fc004f70149baeba2c8ec672bd4f27761f8e1ad6b
    Port:          <none>
    Host Port:     <none>
    Command:
      /bin/ash
      -c
      trap : TERM INT; sleep infinity & wait
    State:          Running
      Started:      Sun, 05 May 2024 13:05:41 +0000
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-d44h6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  kube-api-access-d44h6:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type    Reason          Age   From               Message
  ----    ------          ----  ----               -------
  Normal  Scheduled       76s   default-scheduler  Successfully assigned example-multus/example-nad-pod to worker01.example.com
  Normal  AddedInterface  75s   multus             Add eth0 [10.42.95.152/32] from k8s-pod-network
  Normal  AddedInterface  75s   multus             Add net1 [192.168.122.188/24] from example-multus/example-nad
  Normal  Pulling         75s   kubelet            Pulling image "alpine"
  Normal  Pulled          73s   kubelet            Successfully pulled image "alpine" in 2.218s (2.218s including waiting)
  Normal  Created         73s   kubelet            Created container samplepod
  Normal  Started         73s   kubelet            Started container samplepod

在 Events 中可以看到已经将 192.168.122.188 分配给 Pod 。

在 Pod 中查看网络信息：

[vagrant@master01 ~]$ kubectl -n example-multus exec example-nad-pod -- ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0@if24: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP qlen 1000
    link/ether 96:fa:00:7d:38:8a brd ff:ff:ff:ff:ff:ff
    inet 10.42.95.152/32 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::94fa:ff:fe7d:388a/64 scope link
       valid_lft forever preferred_lft forever
3: net1@net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 52:54:00:26:5a:9f brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.188/24 brd 192.168.122.255 scope global net1
       valid_lft forever preferred_lft forever
    inet6 fe80::5254:0:126:5a9f/64 scope link
       valid_lft forever preferred_lft forever

其中 eth0 是默认的 Pod 网络，net1 是通过 Multus 添加的附加网络。
此时可以从集群外部与 Pod 进行通信：

hcai@P1-Gen4:~$ ip a s virbr0
5: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:e8:39:b6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
hcai@P1-Gen4:~$ ping -c 4 192.168.122.188
PING 192.168.122.188 (192.168.122.188) 56(84) bytes of data.
64 bytes from 192.168.122.188: icmp_seq=1 ttl=64 time=0.244 ms
64 bytes from 192.168.122.188: icmp_seq=2 ttl=64 time=0.242 ms
64 bytes from 192.168.122.188: icmp_seq=3 ttl=64 time=0.175 ms
64 bytes from 192.168.122.188: icmp_seq=4 ttl=64 time=0.144 ms

--- 192.168.122.188 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3049ms
rtt min/avg/max/mdev = 0.144/0.201/0.244/0.043 ms

NetworkAttachDefiniton 类型

可以创建以下 NAD 类型：

host-device ：将网络接口附加到单个 Pod
bridge：在节点上使用现有的网络接口，或者配置一个新的网桥，连接到该网络的 Pod 可以通过 bridge 相互通信，也可以与连接到该 bridge 的任何其它网络通信
IPVLAN：创建基于 IPVLAN 并附加到网络接口的网络
MACVLAN：创建基于 MACVLAN 并附加网络接口的网络

其中 bridge 非常适合虚拟化环境，IPVLAN 和 MACVLAN 是专为容器环境设计的 Linux 网络驱动程序。

IPVLAN 和 MACVLAN 是两种不同的虚拟网络技术它们的主要区别在于 MAC 地址和 IP 地址的分配以及数据包的路由方式

IPVLAN允许一个物理网卡拥有多个IP地址，而所有的虚拟接口使用同一个MAC地址，IPVLAN支持两种模式，L2 模式和 L3 模式，在 L2 模式下，每个端点获得相同的MAC地址但不同的IP地址，而在 L3 模式下，数据包在端点之间路由，提供更好的可伸缩性。
MACVLAN 允许同一个网卡拥有多个 MAC 地址，而虚拟出的网卡可以没有IP地址，在 MACVLAN 中，通过MAC地址查找设备。
此外，IPVLAN 通过IP查找设备，而 MACVLAN 通过MAC地址查找设备。

通过 Multus 为容器添加附加网络

安装 Multus

创建 NetworkAttachmenDefinition

NetworkAttachDefiniton 类型

评论· · · · · ·

评论