Linux Team 聚合链

发布时间： 更新时间： 总字数：1276 阅读时间：3m 作者： IP上海 分享 网址

bond 和 team 均是通过 多张网卡 绑定为一个 逻辑网卡，实现本地网卡的 冗余，带宽扩容 和 负载均衡 的功能

术语

  • LA: Link aggregation（链路聚合）
  • LAG: Link aggregation group
  • LACP: Link aggregation control protocol

环境

  • CentOS7
  • 两块网卡
    • ens32 192.168.179.20/24
    • ens33 192.168.179.21/24
  • bond/team IP 192.168.179.22/24

安装与使用

  • teamd 工具集
yum install -y teamd

bond 和 team 配置使用 NetworkManager 提供的 nmctl 命令行。

  • help
nmcli --help
Usage: nmcli [OPTIONS] OBJECT { COMMAND | help }

OPTIONS
  -a, --ask                                ask for missing parameters
  -c, --colors auto|yes|no                 whether to use colors in output
  -e, --escape yes|no                      escape columns separators in values
  -f, --fields <field,...>|all|common      specify fields to output
  -g, --get-values <field,...>|all|common  shortcut for -m tabular -t -f
  -h, --help                               print this help
  -m, --mode tabular|multiline             output mode
  -o, --overview                           overview mode
  -p, --pretty                             pretty output
  -s, --show-secrets                       allow displaying passwords
  -t, --terse                              terse output
  -v, --version                            show program version
  -w, --wait <seconds>                     set timeout waiting for finishing operations

OBJECT
  g[eneral]       NetworkManager's general status and operations
  n[etworking]    overall networking control
  r[adio]         NetworkManager radio switches
  c[onnection]    NetworkManager's connections
  d[evice]        devices managed by NetworkManager
  a[gent]         NetworkManager secret agent or polkit agent
  m[onitor]       monitor NetworkManager changes

示意图

bond

临时创建

  • 创建 bond 链路
$ nmcli connection add con-name bond0 ifname bond0 type bond mode active-backup ip4 192.168.179.22/24
Connection 'bond0' (eabf814c-486f-47bc-81f7-7ccafa56d483) successfully added.
$ nmcli connection show
NAME   UUID                                  TYPE      DEVICE
bond0  eabf814c-486f-47bc-81f7-7ccafa56d483  bond      bond0
...
$ ip addr show bond0
5: bond0: <NO-CARRIER,BROADCAST,MULTICAST,MASTER,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000  # NO-CARRIER 表示当前网卡没有载波讯号
    link/ether 8e:3d:97:79:73:22 brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.22/24 brd 192.168.179.255 scope global noprefixroute bond0
       valid_lft forever preferred_lft forever
$ cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None  # 没有物理网卡工作，不能与外界通信
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
  • 绑定物理网卡

请注意：可能会断网

$ ip link set dev ens32 down
$ ip link set dev ens33 down
$ nmcli connection add con-name ens32 ifname ens32 type bond-slave master bond0
Connection 'ens32' (97a2b698-1032-4475-8487-a9ca55140149) successfully added.
$ nmcli connection add con-name ens33 ifname ens33 type bond-slave master bond0
Connection 'ens33' (ae4bce4a-64be-4d6c-a021-4656258c70f8) successfully added.
$ watch -n 1 cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: ens32
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: ens32
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:29:0e:6b:bc
Slave queue ID: 0

Slave Interface: ens33
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:50:56:3b:49:58
Slave queue ID: 0
  • 验证
$ ip a show bond0
4: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0c:29:0e:6b:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.22/24 brd 192.168.179.255 scope global bond0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe0e:6bbc/64 scope link
       valid_lft forever preferred_lft forever
$ ping -I bond0 1.1.1.1
$ tcpdump -vv -nnt -i bond0 icmp
tcpdump: listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
IP (tos 0x0, ttl 64, id 3908, offset 0, flags [DF], proto ICMP (1), length 84)
    192.168.179.22 > 1.1.1.1: ICMP echo request, id 3322, seq 23, length 64
IP (tos 0x0, ttl 253, id 41388, offset 0, flags [none], proto ICMP (1), length 84)
    1.1.1.1 > 192.168.179.22: ICMP echo reply, id 3322, seq 23, length 64

可以停止 ip link set ens32 down 后，可以观察到 ens33 自动变成主。

  • 删除
nmcli connection delete bond0 ens33 ens32

文件配置

  • ens32
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-ens32
TYPE=Ethernet
NAME=ens32
DEVICE=ens32
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
EOF
  • ens33
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-ens33
TYPE=Ethernet
NAME=ens33
DEVICE=ens33
ONBOOT=yes
BOOTPROTO=none
MASTER=bond0
SLAVE=yes
USERCTL=no
EOF
  • bond0
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-bond0
NAME=bond0
DEVICE=bond0
TYPE=Bond
BONDING_MASTER=yes
IPADDR=192.168.179.22
BROADCAST=192.168.179.255
GATEWAY=192.168.179.1
ONBOOT=yes
BOOTPROTO=none
BONDING_OPTS="mode=active-backup"
USERCTL=no
EOF
  • 生效
nmcli con reload
systemctl restart network.service

说明：

  • bond0/ens32/ens33 最终的 mac 地址是相同的

team

命令创建

$ nmcli connection add con-name team0 ifname team0 type team config '{"runner":{"name":"activebackup"}}' ip4 192.168.179.23/24
Connection 'team0' (70185ab8-4b91-4b65-819c-cd55ed63259b) successfully added.
$ ip link set dev ens32 down
$ ip link set dev ens33 down
$ nmcli connection add con-name ens32 ifname ens32 type team-slave master team0
$ nmcli connection add con-name ens33 ifname ens33 type team-slave master team0
$ ip link set dev ens32 up
$ ip link set dev ens33 up
$ systemctl restart network.service

监控命令

$ teamdctl team0 stat
setup:
  runner: activebackup
ports:
  ens32
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
  ens33
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
runner:
  active port: ens32
  • 切换验证
$ ip link set ens32 down
$ teamdctl team0 stat
setup:
  runner: activebackup
ports:
  ens32
    link watches:
      link summary: up
      instance[link_watch_0]:
        name: ethtool
        link: up
        down count: 0
  ens33
    link watches:
      link summary: down
      instance[link_watch_0]:
        name: ethtool
        link: down
        down count: 1
runner:
  active port: ens33    # ens32 -> ens33

PS：

  • team0、ens32、ens33 MAC 地址也相同

文件创建

  • ens32
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-ens32
NAME=ens32
DEVICE=ens32
ONBOOT=yes
BOOTPROTO=none
TEAM_MASTER=team0
DEVICETYPE=TeamPort
EOF
  • ens33
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-ens33
NAME=ens33
DEVICE=ens33
ONBOOT=yes
BOOTPROTO=none
TEAM_MASTER=team0
DEVICETYPE=TeamPort
EOF
  • team0
cat << EOF >> /etc/sysconfig/network-scripts/ifcfg-team0
DEVICE=team0
NAME=team0
DEVICETYPE=Team
TEAM_CONFIG="{\"runner\":{\"name\":\"activebackup\"}}"
IPADDR=192.168.179.22
BROADCAST=192.168.179.255
GATEWAY=192.168.179.1
ONBOOT=yes
BOOTPROTO=none
EOF
  • 生效
nmcli con reload
systemctl restart network.service
  • 生效后信息
$ ip a
2: ens32: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
    link/ether 00:0c:29:4b:90:5b brd ff:ff:ff:ff:ff:ff
3: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master team0 state UP group default qlen 1000
    link/ether 00:0c:29:4b:90:5b brd ff:ff:ff:ff:ff:ff
6: team0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:0c:29:4b:90:5b brd ff:ff:ff:ff:ff:ff
    inet 192.168.179.22/24 brd 192.168.179.255 scope global noprefixroute team0
       valid_lft forever preferred_lft forever
    inet6 fe80::20c:29ff:fe4b:905b/64 scope link
       valid_lft forever preferred_lft forever

绑定多地址

  • activebackup
$ cat ifcfg-eno1
DEVICE=eno1
DEVICETYPE=TeamPort
TEAM_MASTER=team1
TEAM_PORT_CONFIG='{"prio": 100}'
ETHTOOL_OPTS="-K ${DEVICE} lro off; -N ${DEVICE} rx-flow-hash udp4 sdfn"
NM_CONTROLLED="no"
ONBOOT=yes
MTU=9000

$ cat ifcfg-eno2
DEVICE=eno2
DEVICETYPE=TeamPort
TEAM_MASTER=team1
TEAM_PORT_CONFIG='{"prio": 100}'
ETHTOOL_OPTS="-K ${DEVICE} lro off; -N ${DEVICE} rx-flow-hash udp4 sdfn"
NM_CONTROLLED="no"
ONBOOT=yes
MTU=9000

$ cat ifcfg-team1
DEVICE=team1
DEVICETYPE=Team
TEAM_CONFIG='{"runner": {"name": "activebackup"}, "link_watch": {"name": "ethtool"}}'
NM_CONTROLLED="no"
BOOTPROTO=none
ONBOOT=yes
MTU=9000

$ cat ifcfg-team1.1
DEVICE=team1.1
DEVICETYPE=Team
NM_CONTROLLED="no"
BOOTPROTO=static
IPADDR=192.168.179.22
PREFIX=24
MTU=9000
VLAN=yes
ONBOOT=yes
GATEWAY=192.168.179.1
  • lacp
$ cat ifcfg-enp175s0f0
DEVICE=enp175s0f0
DEVICETYPE=TeamPort
TEAM_MASTER=team2
TEAM_PORT_CONFIG='{"prio": 100}'
ETHTOOL_OPTS="-K ${DEVICE} lro off"
NM_CONTROLLED="no"
ONBOOT=yes
MTU=9000

$ cat ifcfg-enp175s0f1
DEVICE=enp175s0f1
DEVICETYPE=TeamPort
TEAM_MASTER=team2
TEAM_PORT_CONFIG='{"prio": 100}'
ETHTOOL_OPTS="-K ${DEVICE} lro off"
NM_CONTROLLED="no"
ONBOOT=yes
MTU=9000

$ cat ifcfg-team2
DEVICE=team2
DEVICETYPE=Team
TEAM_CONFIG='{ "runner" : { "name" : "lacp", "active" : true , "agg_select_policy" : "lacp_prio", "fast_rate" : true , "tx_hash" : [ "vlan", "eth", "ipv4" ] , "tx_balancer" : { "name" : "basic"} }, "link_watch": { "name" : "ethtool" } }'
NM_CONTROLLED="no"
BOOTPROTO=none
ONBOOT=yes
MTU=9000

$ cat ifcfg-team2.1
DEVICE=team2.1
DEVICETYPE=Team
NM_CONTROLLED="no"
BOOTPROTO=static
IPADDR=192.168.179.22
PREFIX=24
MTU=9000
VLAN=yes
ONBOOT=yes
GATEWAY=192.168.179.1

参考

  1. comparison_teaming_and_bonding
最近更新
相关文章
最新评论
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数 人