Neutron 基于 iptables 实现安全组功能
OpenStack安全组（Security Group）默认是通过Linux iptables实现的，本文介绍Neutron基于iptables实现的安全组。
启用 iptables 安全组
修改 /etc/neutron/plugins/ml2/ml2_conf.ini 配置文件
[securitygroup]
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver重启服务
sudo systemctl restart devstack@q-*默认的 iptables 规则
通过 iptables-save 命令可以看到的流被转发到 neutron-openvswi-x 的链中：
$ sudo iptables-save
...
-A INPUT -j neutron-openvswi-INPUT # 将 INPUT 转到 neutron-openvswi-INPUT 链
-A FORWARD -j neutron-filter-top
-A FORWARD -j neutron-openvswi-FORWARD # 将 FORWARD 转到 neutron-openvswi-FORWARD 链
-A OUTPUT -j neutron-filter-top
-A OUTPUT -j neutron-openvswi-OUTPUT # 将 OUTPUT 转到 neutron-openvswi-OUTPUT 链
-A neutron-filter-top -j neutron-openvswi-local
...查看 Chain 信息如下：
默认初始化的 iptables Chain 链如下：
- neutron-filter-top
- neutron-openvswi-FORWARD # neutron 定义的 FORWARD 链
- neutron-openvswi-INPUT # neutron 定义的 INPUT 链
- neutron-openvswi-OUTPUT # neutron 定义的 OUTPUT 链
- neutron-openvswi-local
- neutron-openvswi-sg-chain
- neutron-openvswi-sg-fallback
创建网络和子网
- 创建网络
- 创建子网，并启用
--dhcp
可以发现已经分配了一个ip地址 10.0.0.2，该地址为 dhcp 的地址，接下来我们从 netns、OVS 等方面查看他的信息。
netns 信息
创建子网（并启用dhcp）时，创建的 netns 信息如下，可以看到 tap190045e5-44 已经在该 netns 中，对应的地址为： 10.0.0.2/24
OVS 信息
tap190045e5-44 绑定在 br-int 网桥上，tag 为 1，为该网络提供 dhcp 功能。
iptables 信息
通过 iptables 规则，默认把所有流量转发到 tap190045e5-44，iptables 信息如下：
stack@xiexianbin-vm:~$ sudo iptables -nvL
Chain neutron-openvswi-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-out tap190045e5-44 --physdev-is-bridged /* Accept all packets when port is trusted. */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 PHYSDEV match --physdev-in tap190045e5-44 --physdev-is-bridged /* Accept all packets when port is trusted. */neutron-dhcp-agent 通过该 tap 设备为该 network 提供 dhcp 功能。TODO(xiexianbin)
安全组信息
默认情况下，对所有 ipv4/6 出的流量和本安全组内的流量不限制：
创建 VM
发现：
- 每个
port均和security_group_ids绑定，并且一个port上可以绑定多个安全组
查看 OVS 信息
发现 br-int 上新增了一个 qvo33d3e592-c6，该网桥名称为 qvo<port-id-prefix-11> 组成，信息如下：
stack@xiexianbin-vm:~$ sudo ovs-vsctl show
0976470f-8041-45d1-ad1a-34cd061684d3
Manager "ptcp:6640:127.0.0.1"
is_connected: true
Bridge br-ex
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: system
Port br-ex
Interface br-ex
type: internal
Port phy-br-ex
Interface phy-br-ex
type: patch
options: {peer=int-br-ex}
Bridge br-tun
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: system
Port br-tun
Interface br-tun
type: internal
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Bridge br-int
Controller "tcp:127.0.0.1:6633"
is_connected: true
fail_mode: secure
datapath_type: system
Port br-int
Interface br-int
type: internal
Port qvo33d3e592-c6
tag: 1
Interface qvo33d3e592-c6
Port int-br-ex
Interface int-br-ex
type: patch
options: {peer=phy-br-ex}
Port tap190045e5-44
tag: 1
Interface tap190045e5-44
type: internal
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
ovs_version: "2.13.1"
stack@xiexianbin-vm:~$ sudo ovs-ofctl show br-int
OFPT_FEATURES_REPLY (xid=0x2): dpid:0000a62b2fabc24d
n_tables:254, n_buffers:0
capabilities: FLOW_STATS TABLE_STATS PORT_STATS QUEUE_STATS ARP_MATCH_IP
actions: output enqueue set_vlan_vid set_vlan_pcp strip_vlan mod_dl_src mod_dl_dst mod_nw_src mod_nw_dst mod_nw_tos mod_tp_src mod_tp_dst
1(int-br-ex): addr:3e:e4:b9:12:01:57
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
2(patch-tun): addr:96:5e:3a:be:41:08
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
3(tap190045e5-44): addr:fa:16:3e:a3:28:f3
config: 0
state: 0
speed: 0 Mbps now, 0 Mbps max
4(qvo33d3e592-c6): addr:0a:d6:fb:38:47:33
config: 0
state: 0
current: 10GB-FD COPPER
speed: 10000 Mbps now, 0 Mbps max
LOCAL(br-int): addr:a6:2b:2f:ab:c2:4d
config: PORT_DOWN
state: LINK_DOWN
speed: 0 Mbps now, 0 Mbps max
OFPT_GET_CONFIG_REPLY (xid=0x4): frags=normal miss_send_len=0veth-pair 信息
通过 ip addr show 命令，我们发现 qvo33d3e592-c6 是 veth-pair的一个，另一个为 qvb33d3e592-c6 绑定在 Linux bridge qbr33d3e592-c6 上
Linux bridge 信息
发现新增了一个 qbr33d3e592-c6，该网桥名称为 qbr<port-id-prefix-11> 组成，基于 iptables 实现的安全组规则将基于此网桥实现。
stack@xiexianbin-vm:~$ sudo ip a show type bridge
11: qbr33d3e592-c6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default qlen 1000
link/ether 3a:6d:32:cc:34:4a brd ff:ff:ff:ff:ff:fftap 信息
同样的，通过 ip addr show 命令，我们发现网桥 qbr33d3e592-c6 还绑定这一个 tap 设备TODO(xiexianbin)，名称为 tap33d3e592-c6
stack@xiexianbin-vm:~$ ip addr show
...
14: tap33d3e592-c6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc fq_codel master qbr33d3e592-c6 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:57:1c:cd brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe57:1ccd/64 scope link
valid_lft forever preferred_lft forever
...云主机使用 tap 设备
通过 ps -ef|grep qemu 查看当前使用qemuTODO(xiexianbin)虚拟化的主机进程为 5369 -netdev tap,fd=32,id=hostnet0，（TODO(xiexianbin)什么是fd）通过如下命令发现，该主机使用的就是 tap33d3e592-c6
stack@xiexianbin-vm:~$ sudo cat /proc/5369/fdinfo/32
pos: 82
flags: 0104002
mnt_id: 27
iff: tap33d3e592-c6至此，我们已经将云主机怎么与br-int连接介绍清楚。
iptables 信息
创建 vm 后，iptables 规则变化如下：
与该主机端口相关的，主要的 iptables Chain 如下：
neutron-openvswi-s33d3e592-c：命令neutron-openvswi-s<port-id-prefix-10>，处理出该端口的网络包的防IP欺骗neutron-openvswi-i33d3e592-c：命令neutron-openvswi-i<port-id-prefix-10>，处理进该端口的网络包的防IP欺骗neutron-openvswi-o33d3e592-c：命令neutron-openvswi-o<port-id-prefix-10>，处理出该端口的网络包
以进入端口的链 neutron-openvswi-i33d3e592-c 为例，说明如下：
stack@xiexianbin-vm:~$ sudo iptables -nvL neutron-openvswi-i33d3e592-c
Chain neutron-openvswi-i33d3e592-c (1 references)
pkts bytes target prot opt in out source destination
5 420 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */ # 放行已知会话相关联的信息包
2 696 RETURN udp -- * * 0.0.0.0/0 10.0.0.29 udp spt:67 dpt:68 # 放行到 10.0.0.29 的 utp 包
0 0 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68 # 放行到 255.255.255.255 的 utp 包
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv422b33e63-0f84-4bbc-a797- src # 放行到 0.0.0.0/0 匹配 ipset NIPv422b33e63-0f84-4bbc-a797- 的包
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */ # 丢弃 INVALID 的包
0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */ # 将不匹配的通信发送到回退链
stack@xiexianbin-vm:~$ sudo ipset list NIPv422b33e63-0f84-4bbc-a797-
Name: NIPv422b33e63-0f84-4bbc-a797-
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 512
References: 1
Number of entries: 1
Members:
10.0.0.29 # ipset 匹配 10.0.0.29新建防火墙规则对 iptables 的影响
新建允许从 10.1.0.0/24 ping 主机的规则到 default 安全组，命令如下：
stack@xiexianbin-vm:~$ openstack security group rule create 22b33e63-0f84-4bbc-a797-7c3d7ae107aa --remote-ip 10.1.0.0/24 --protocol icmp --ingress
+-------------------+--------------------------------------+
| Field | Value |
+-------------------+--------------------------------------+
| created_at | 2021-01-03T03:48:27Z |
| description | |
| direction | ingress |
| ether_type | IPv4 |
| id | 3588e58b-1890-4929-ab85-d5519ec1b7a7 |
| name | None |
| port_range_max | None |
| port_range_min | None |
| project_id | 45837e6267c44d7788a919d6e342e64c |
| protocol | icmp |
| remote_group_id | None |
| remote_ip_prefix | 10.1.0.0/24 |
| revision_number | 0 |
| security_group_id | 22b33e63-0f84-4bbc-a797-7c3d7ae107aa |
| tags | [] |
| updated_at | 2021-01-03T03:48:27Z |
+-------------------+--------------------------------------+查看 iptables 规则，多了一条：
$ iptables -nvL
Chain neutron-openvswi-i33d3e592-c (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN icmp -- * * 10.1.0.0/24 0.0.0.0/0 # 放行从 `10.1.0.0/24` 发出的 icmp 包至此，我们可以从 10.1.0.0/24 ping 通云主机，安全组规则生效。类似的出站规则也是这样工作。
allow_address_key_pairs
为 port 10.0.0.29 添加 allowed-address 0.0.0.0/0，操作如下：
stack@xiexianbin-vm:~$ openstack port set 33d3e592-c6eb-4cb3-8266-33c8e2e1f646 --allowed-address ip-address=0.0.0.0/0
stack@xiexianbin-vm:~$ openstack port show 33d3e592-c6eb-4cb3-8266-33c8e2e1f646
+-------------------------+-------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-------------------------+-------------------------------------------------------------------------------------------------------------+
| admin_state_up | UP |
| allowed_address_pairs | ip_address='0.0.0.0/0', mac_address='fa:16:3e:57:1c:cd' |
| binding_host_id | xiexianbin-vm |
| binding_profile | |
| binding_vif_details | bridge_name='br-int', connectivity='l2', datapath_type='system', ovs_hybrid_plug='True', port_filter='True' |
| binding_vif_type | ovs |
| binding_vnic_type | normal |
| created_at | 2021-01-03T01:30:54Z |
| data_plane_status | None |
| description | |
| device_id | a9afb91b-8700-4901-9565-5bf6a63f0143 |
| device_owner | compute:nova |
| dns_assignment | None |
| dns_domain | None |
| dns_name | None |
| extra_dhcp_opts | |
| fixed_ips | ip_address='10.0.0.29', subnet_id='3768efcb-e21e-4293-a1f9-596e28a21cb5' |
| id | 33d3e592-c6eb-4cb3-8266-33c8e2e1f646 |
| ip_allocation | None |
| mac_address | fa:16:3e:57:1c:cd |
| name | |
| network_id | 557bc377-cece-41dd-8b77-1348355ff66c |
| numa_affinity_policy | None |
| port_security_enabled | True |
| project_id | 45837e6267c44d7788a919d6e342e64c |
| propagate_uplink_status | None |
| qos_network_policy_id | None |
| qos_policy_id | None |
| resource_request | None |
| revision_number | 5 |
| security_group_ids | 22b33e63-0f84-4bbc-a797-7c3d7ae107aa |
| status | ACTIVE |
| tags | |
| trunk_details | None |
| updated_at | 2021-01-03T03:56:17Z |
+-------------------------+-------------------------------------------------------------------------------------------------------------+Chain neutron-openvswi-s33d3e592-c 增加一条对所有 0.0.0.0/0 FA:16:3E:57:1C:CD 地址来的流量放行的规则，如下：
Chain neutron-openvswi-s33d3e592-c (1 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 MAC FA:16:3E:57:1C:CD /* Allow traffic from defined IP/MAC pairs. */安全组规则没有发生变化（其他场景也可能新增 match-set，本示例已经存在），我们查看 ipset 信息如下：
stack@xiexianbin-vm:~$ sudo ipset list NIPv422b33e63-0f84-4bbc-a797-
Name: NIPv422b33e63-0f84-4bbc-a797-
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 640
References: 1
Number of entries: 3
Members:
128.0.0.0/1
0.0.0.0/1 # 放行所有流量
10.0.0.29从 ipset 看，当添加 --allowed-address ip-address=0.0.0.0/0 时，会放行所有流量，导致安全组失效。
因此，我们建议在生产环境，添加 --allowed-address 时，尽量指定地址范围，避免安全组失效的问题。指定 --allowed-address ip-address=10.0.0.128/26 时，ipset 信息如下：
stack@xiexianbin-vm:~$ sudo ipset list NIPv422b33e63-0f84-4bbc-a797-
Name: NIPv422b33e63-0f84-4bbc-a797-
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 640
References: 1
Number of entries: 3
Members:
128.0.0.0/1
10.0.0.128/26 # 放行指定网段流量
10.0.0.29导致安全组失效，主要是自身添加了到自身安全组放行导致的，我们删除对应的规则，如下：
stack@xiexianbin-vm:~$ openstack security group list
+--------------------------------------+---------+------------------------+----------------------------------+------+
| ID | Name | Description | Project | Tags |
+--------------------------------------+---------+------------------------+----------------------------------+------+
| 22b33e63-0f84-4bbc-a797-7c3d7ae107aa | default | Default security group | 45837e6267c44d7788a919d6e342e64c | [] |
+--------------------------------------+---------+------------------------+----------------------------------+------+
stack@xiexianbin-vm:~$ openstack security group rule list 22b33e63-0f84-4bbc-a797-7c3d7ae107aa
+--------------------------------------+-------------+-----------+-------------+------------+--------------------------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-------------+------------+--------------------------------------+
| 09c623fc-bf1f-46bf-9149-facdc6148831 | None | IPv6 | ::/0 | | None |
| 0f709d8c-7400-40dd-a5b7-59beaee93c74 | None | IPv6 | ::/0 | | 22b33e63-0f84-4bbc-a797-7c3d7ae107aa |
| 3588e58b-1890-4929-ab85-d5519ec1b7a7 | icmp | IPv4 | 10.1.0.0/24 | | None |
| 366c1188-988b-4c5a-8bf4-dfcec6fb902f | None | IPv4 | 0.0.0.0/0 | | 22b33e63-0f84-4bbc-a797-7c3d7ae107aa |
| 6ff01e6d-9b34-433c-968f-c04495098145 | None | IPv4 | 0.0.0.0/0 | | None |
+--------------------------------------+-------------+-----------+-------------+------------+--------------------------------------+
stack@xiexianbin-vm:~$ openstack security group rule delete 0f709d8c-7400-40dd-a5b7-59beaee93c74 366c1188-988b-4c5a-8bf4-dfcec6fb902f
stack@xiexianbin-vm:~$ openstack security group rule list 22b33e63-0f84-4bbc-a797-7c3d7ae107aa
+--------------------------------------+-------------+-----------+-------------+------------+-----------------------+
| ID | IP Protocol | Ethertype | IP Range | Port Range | Remote Security Group |
+--------------------------------------+-------------+-----------+-------------+------------+-----------------------+
| 09c623fc-bf1f-46bf-9149-facdc6148831 | None | IPv6 | ::/0 | | None |
| 3588e58b-1890-4929-ab85-d5519ec1b7a7 | icmp | IPv4 | 10.1.0.0/24 | | None |
| 6ff01e6d-9b34-433c-968f-c04495098145 | None | IPv4 | 0.0.0.0/0 | | None |
+--------------------------------------+-------------+-----------+-------------+------------+-----------------------+iptables Chain neutron-openvswi-i33d3e592-c 对应的 ipset NIPv422b33e63-0f84-4bbc-a797- 也被删除，因此安全组可以正常工作，如下：
stack@xiexianbin-vm:~$ sudo iptables -nvL neutron-openvswi-i33d3e592-c
Chain neutron-openvswi-i33d3e592-c (1 references)
pkts bytes target prot opt in out source destination
5 420 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
2 696 RETURN udp -- * * 0.0.0.0/0 10.0.0.29 udp spt:67 dpt:68
0 0 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68
0 0 RETURN icmp -- * * 10.1.0.0/24 0.0.0.0/0
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */在管理其他安全组，观察ipset变化，如下：
stack@xiexianbin-vm:~$ openstack security group create demo-1
+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+
| created_at | 2021-01-03T05:29:25Z |
| description | demo-1 |
| id | 178a2ad5-f758-4af0-a920-1e54cc8260e5 |
| name | demo-1 |
| project_id | 45837e6267c44d7788a919d6e342e64c |
| revision_number | 1 |
| rules | created_at='2021-01-03T05:29:25Z', direction='egress', ethertype='IPv6', id='26d6494c-4bf5-44c1-91d4-0525c32046f5', updated_at='2021-01-03T05:29:25Z' |
| | created_at='2021-01-03T05:29:25Z', direction='egress', ethertype='IPv4', id='7a48bb5c-319d-4963-a7bc-406729e27a78', updated_at='2021-01-03T05:29:25Z' |
| stateful | True |
| tags | [] |
| updated_at | 2021-01-03T05:29:25Z |
+-----------------+-------------------------------------------------------------------------------------------------------------------------------------------------------+iptables Chain neutron-openvswi-i33d3e592-c 对应的 ipset NIPv4178a2ad5-f758-4af0-a920- members 为空，安全组工作正常。
stack@xiexianbin-vm:~$ sudo iptables -nvL neutron-openvswi-i33d3e592-c
Chain neutron-openvswi-i33d3e592-c (1 references)
pkts bytes target prot opt in out source destination
5 420 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED /* Direct packets associated with a known session to the RETURN chain. */
2 696 RETURN udp -- * * 0.0.0.0/0 10.0.0.29 udp spt:67 dpt:68
0 0 RETURN udp -- * * 0.0.0.0/0 255.255.255.255 udp spt:67 dpt:68
0 0 RETURN icmp -- * * 10.1.0.0/24 0.0.0.0/0
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 match-set NIPv4178a2ad5-f758-4af0-a920- src
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 state INVALID /* Drop packets that appear related to an existing connection (e.g. TCP ACK/FIN) but do not have an entry in conntrack. */
0 0 neutron-openvswi-sg-fallback all -- * * 0.0.0.0/0 0.0.0.0/0 /* Send unmatched traffic to the fallback chain. */
stack@xiexianbin-vm:~$ sudo ipset list NIPv4178a2ad5-f758-4af0-a920-
Name: NIPv4178a2ad5-f758-4af0-a920-
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 448
References: 1
Number of entries: 0
Members:https://www.xiexianbin.cn/openstack/neutron/2015-03-21-neutron-havana-allow-address-pairs/index.html