Etcd 集群部署

Etcd 集群安装部署

介绍

etcd 集群的启动方式包括：

• 静态启动
• etcd 动态发现
• NDS 发现

etcd 集群架构

• Raft 层是实现 etcd 数据一致性的关键，etcd 节点之间通过 Raft 协议实现通信
• Storage 层实现持久化键值对
• WAL 日志模块
• 快照模块

图片参考

环境

• 使用 docker 启动 3 台主机，模拟 vm

for ((i=1; i<=3; i++)); do
  docker run \
    -it \
    -d \
    --privileged \
    --hostname etcd${i} \
    --name etcd${i} \
    xiexianbin/ubuntu:20.04 \
    /lib/systemd/systemd
done

• IP 信息
  • etcd1: 172.17.0.3
  • etcd2: 172.17.0.4
  • etcd3: 172.17.0.5
• etcd 下载地址：https://github.com/etcd-io/etcd/releases

mkdir -p /opt/etcd ; cd /opt/etcd
wget https://github.com/etcd-io/etcd/releases/download/v3.4.27/etcd-v3.4.27-linux-amd64.tar.gz

• 复制到所有节点

for ((i=1; i<=3; i++)); do
  docker exec -it etcd${i} mkdir -p /opt/etcd/{data,etc}
  docker cp etcd-v3.4.27-linux-amd64.tar.gz etcd${i}:/opt/etcd
  docker exec -it etcd${i} bash -c "cd /opt/etcd; tar -zxf etcd-v3.4.27-linux-amd64.tar.gz; ln -s etcd-v3.4.27-linux-amd64 etcd"
done

• 规划目录说明
  • /opt/etcd/data etcd 保存数据的目录
  • /opt/etcd/etc/etcd.yaml 配置文件目录

配置文件

• 创建配置文件 /opt/etcd/etc/etcd.yaml，三个节点配置文件内容分别如下：

• etcd1

name: etcd1
data-dir: /opt/etcd/data
initial-advertise-peer-urls: http://172.17.0.3:2380
listen-peer-urls: http://172.17.0.3:2380
listen-client-urls: http://172.17.0.3:2379,http://127.0.0.1:2379
advertise-client-urls: http://172.17.0.3:2379
initial-cluster-token: etcd-cluster
initial-cluster: etcd1=http://172.17.0.3:2380,etcd2=http://172.17.0.4:2380,etcd3=http://172.17.0.5:2380
initial-cluster-state: new

• etcd2

name: etcd2
data-dir: /opt/etcd/data
initial-advertise-peer-urls: http://172.17.0.4:2380
listen-peer-urls: http://172.17.0.4:2380
listen-client-urls: http://172.17.0.4:2379,http://127.0.0.1:2379
advertise-client-urls: http://172.17.0.4:2379
initial-cluster-token: etcd-cluster
initial-cluster: etcd1=http://172.17.0.3:2380,etcd2=http://172.17.0.4:2380,etcd3=http://172.17.0.5:2380
initial-cluster-state: new

• etcd3

name: etcd3
data-dir: /opt/etcd/data
initial-advertise-peer-urls: http://172.17.0.5:2380
listen-peer-urls: http://172.17.0.5:2380
listen-client-urls: http://172.17.0.5:2379,http://127.0.0.1:2379
advertise-client-urls: http://172.17.0.5:2379
initial-cluster-token: etcd-cluster
initial-cluster: etcd1=http://172.17.0.3:2380,etcd2=http://172.17.0.4:2380,etcd3=http://172.17.0.5:2380
initial-cluster-state: new

配置参数说明：

• name：etcd 节点名称
• data-dir：数据存储目录
• initial-advertise-peer-urls：集群的其他节点通过该地址与当前节点通信
• listen-peer-urls：当前节点通过该地址监听集群其他节点发送的信息
• listen-client-urls：当前节点通过该地址监听客户端发送的信息
• advertise-client-urls：客户端通过该地址与当前节点通信
• initial-cluster-token：用于区分不同的集群，同一集群的所有节点配置相同的值
• initial-cluster：当前集群的所有节点信息，当前节点根据此信息与其他节点取得联系
• initial-cluster-state：本次是否为新建集群，有两个取值：new 和 existing

配置 service 服务

cat >/usr/lib/systemd/system/etcd.service<<EOF
[Unit]
Description=etcd
After=network.target

[Service]
Type=notify
LimitNOFILE=40000
TimeoutStartSec=0
ExecStart=/opt/etcd/etcd/etcd --config-file=/opt/etcd/etc/etcd.yaml

[Install]
WantedBy=multi-user.target
EOF

启动服务

for ((i=1; i<=3; i++)); do
  docker exec -it etcd${i} systemctl daemon-reload
  docker exec -it etcd${i} systemctl start etcd.service &
done

验证

• 查看 etcd 版本

$ /opt/etcd/etcd/etcd --version
WARNING: Package "github.com/golang/protobuf/protoc-gen-go/generator" is deprecated.
	A future release of golang/protobuf will delete this package,
	which has long been excluded from the compatibility promise.

etcd Version: 3.4.27
Git SHA: c92fb80f3
Go Version: go1.19.10
Go OS/Arch: linux/amd64

• etcdctl 软连接

ln -s /opt/etcd/etcd/etcdctl /usr/local/bin/etcdctl

• 查看集群成员列表

$ etcdctl member list
1af57124460c9172, started, etcd3, http://172.17.0.5:2380, http://172.17.0.5:2379, false
660aa483274d103a, started, etcd1, http://172.17.0.3:2380, http://172.17.0.3:2379, false
ad0233873e2a0054, started, etcd2, http://172.17.0.4:2380, http://172.17.0.4:2379, false

• 查看集群成员健康情况

$ etcdctl endpoint health --endpoints=http://172.17.0.3:2380,http://172.17.0.4:2380,http://172.17.0.5:2380
http://172.17.0.5:2380 is healthy: successfully committed proposal: took = 5.749126ms
http://172.17.0.3:2380 is healthy: successfully committed proposal: took = 5.820195ms
http://172.17.0.4:2380 is healthy: successfully committed proposal: took = 6.513411ms

其他

也可将参数添加到 systemctl 中

/etc/systemd/system/etcd.service

[Unit]
Description=etcd
Documentation=https://github.com/coreos/etcd

[Service]
Type=notify
Restart=always
RestartSec=5s
LimitNOFILE=40000
TimeoutStartSec=0
EnvironmentFile=-/etc/sysconfig/etcd-ssl

ExecStart=/usr/local/bin/etcd --name etcd1 \
  --data-dir /data/bkee/public/etcd \
  --listen-client-urls http://<ip1>:2379,http://127.0.0.1:2379 \
  --advertise-client-urls http://<ip1>:2379 \
  --listen-peer-urls http://<ip1>:2380 \
  --initial-advertise-peer-urls http://<ip1>:2380 \
  --initial-cluster etcd=http://<ip0>:2380,etcd1=http://<ip1>:2380,etcd2=http://<ip2>:2380 \
  --initial-cluster-state new \
  --initial-cluster-token etcd-cluster-token \
  --auto-compaction-retention 1

[Install]
WantedBy=multi-user.target

操作命令：

• systemctl enable etcd
• systemctl start etcd