ceph-deploy 部署 Ceph 集群

发布时间： 更新时间： 总字数：1381 阅读时间：3m 作者： IP上海 分享 网址

使用ceph-deploy安装部署Ceph集群示例

节点说明

  • ceph-admin: 192.168.122.100
  • 网络
    • cluster-network 192.168.2.0/24
    • public-network 192.168.122.0/24
  • ceph 其他节点
IP Domain Hostname Services
192.168.122.101 stor01.kb.cx c1 mon,osd
192.168.122.102 stor02.kb.cx c2 mgr,osd,mds
192.168.122.103 stor03.kb.cx c3 rgw,osd
  • 操作系统 ubuntu20.04

节点准备

192.168.122.101 stor01.kb.cx mon01
192.168.122.102 stor02.kb.cx mgr01 mds01
192.168.122.103 stor03.kb.cx rgw01
  • 关闭 iptable 和 firewalld
systemctl stop iptables
systemctl stop firewalld
systemctl disable iptables
systemctl disable firewalld
  • 关闭 SElinux
  • 创建部署Ceph的Linux账号，需要有sudo权限(使用 sudo -l 确认权限)，所有节点添加：
useradd cephadm && echo password | passwd --stdin cephadmin
echo "cephadm ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/cephadm
chmod 0440 /etc/sudoers.d/cephadm
  • 生成SSH证书，并分发到其他节点
ssh-keygen -t rsa -P ''
ssh-copy-id -i .ssh/id_rsa.pub cephadm@<ip>

安装

安装ceph-deploy

apt install ceph-deploy

并安装 python3.5，否则会报 [ceph_deploy][ERROR ] RuntimeError: AttributeError: module 'platform' has no attribute 'linux_distribution'

# install python3.5
apt install software-properties-common
add-apt-repository ppa:deadsnakes/ppa
apt install python3.5

# create python3.5 env
virtualenv py35 -p python3.5

# use python3.5
修改 /usr/bin/ceph-deploy 第一行： #!/usr/bin/python3.5

初始化 rados 集群

  • 创建目录，存储集群配置信息
mkdir ceph-cluster && cd ceph-cluster
  • 初始化 mon 节点，c1 必须为 hostname，网络配置也可以在 ceph.conf 中配置
ceph-deploy new c1

# help
usage: ceph-deploy new [-h] [--no-ssh-copykey] [--fsid FSID] [--cluster-network CLUSTER_NETWORK]
                       [--public-network PUBLIC_NETWORK]
                       MON [MON ...]

生成如下文件：

  1. ceph.conf
  2. ceph.mon.keyring
  • 安装 Ceph 集群
ceph-deploy install --no-adjust-repos c1 c2 c3

# help
ceph-deploy install --help
usage: ceph-deploy install [-h] [--stable [CODENAME] | --release [CODENAME] | --testing | --dev [BRANCH_OR_TAG]]
                           [--dev-commit [COMMIT]] [--mon] [--mgr] [--mds] [--rgw] [--osd] [--tests] [--cli] [--all]
                           [--adjust-repos | --no-adjust-repos | --repo] [--local-mirror [LOCAL_MIRROR]]
                           [--repo-url [REPO_URL]] [--gpg-url [GPG_URL]] [--nogpgcheck]
                           HOST [HOST ...]

Install Ceph packages on remote hosts.

positional arguments:
  HOST                  hosts to install on

optional arguments:
  -h, --help            show this help message and exit
  --stable [CODENAME]   [DEPRECATED] install a release known as CODENAME (done by default) (default: None)
  --release [CODENAME]  install a release known as CODENAME (done by default) (default: None)
  --testing             install the latest development release
  --dev [BRANCH_OR_TAG]
                        install a bleeding edge build from Git branch or tag (default: master)
  --dev-commit [COMMIT]
                        install a bleeding edge build from Git commit (defaults to master branch)
  --mon                 install the mon component only
  --mgr                 install the mgr component only
  --mds                 install the mds component only
  --rgw                 install the rgw component only
  --osd                 install the osd component only
  --tests               install the testing components
  --cli, --common       install the common component only
  --all                 install all Ceph components (mon, osd, mds, rgw) except tests. This is the default
  --adjust-repos        install packages modifying source repos
  --no-adjust-repos     install packages without modifying source repos
  --repo                install repo files only (skips package installation)
  --local-mirror [LOCAL_MIRROR]
                        Fetch packages and push them to hosts for a local repo mirror
  --repo-url [REPO_URL]
                        specify a repo URL that mirrors/contains Ceph packages
  --gpg-url [GPG_URL]   specify a GPG key URL to be used with custom repos (defaults to ceph.com)
  --nogpgcheck          install packages without gpgcheck
  • 配置初始 Mon 节点
ceph-deploy mon create-initial

执行后生成很多的key:

  1. ceph.bootstrap-[mds|osd|mgr|rgw].keyring 文件
  2. ceph.client.admin.keyring 管理员key，命令行认证文件
  • 将配置文件和admin密钥复制到Ceph集群的其他节点
ceph-deploy admin c1 c2 c3

# help
$ ceph-deploy admin --help
usage: ceph-deploy admin [-h] HOST [HOST ...]

Push configuration and client.admin key to a remote host.

positional arguments:
  HOST        host to configure for Ceph administration

optional arguments:
  -h, --help  show this help message and exit
  • 文件权限赋权
setfacl -m u:cephadm:rw /etc/ceph/ceph.client.admin.keyring
  • 配置 mgr 节点
ceph-deploy mgr create c2

# help
$ ceph-deploy mgr --help
usage: ceph-deploy mgr [-h] {create} ...

Ceph MGR daemon management

positional arguments:
  {create}
    create    Deploy Ceph MGR on remote host(s)

optional arguments:
  -h, --help  show this help message and exit
  • 查看集群的状态
# 工具包
apt install ceph-common

# 查看状态
ceph health
ceph -s

向 RADOS 集群添加 OSD

  • 列出OSD节点的可用磁盘
ceph-deploy disk list c1 c2 c3
  • 擦除磁盘数据
ceph-deploy disk zap c1 /dev/vdb
  • 添加 OSD，默认存储引擎为 bluestore
ceph-deploy osd create c1 --data /dev/vdb
ceph-deploy osd create c1 --data /dev/vdb

# 列出指定节点的 OSD
ceph-deploy osd list c1 c2 c3

# help
$ ceph-deploy osd --help
usage: ceph-deploy osd [-h] {list,create} ...

Create OSDs from a data disk on a remote host:

    ceph-deploy osd create {node} --data /path/to/device

For bluestore, optional devices can be used::

    ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device
    ceph-deploy osd create {node} --data /path/to/data --block-wal /path/to/wal-device
    ceph-deploy osd create {node} --data /path/to/data --block-db /path/to/db-device --block-wal /path/to/wal-device
...
$ ceph-deploy osd create --help
usage: ceph-deploy osd create [-h] [--data DATA] [--journal JOURNAL] [--zap-disk] [--fs-type FS_TYPE] [--dmcrypt]
                              [--dmcrypt-key-dir KEYDIR] [--filestore] [--bluestore] [--block-db BLOCK_DB]
                              [--block-wal BLOCK_WAL] [--debug]
                              [HOST]

positional arguments:
  HOST                  Remote host to connect

optional arguments:
  -h, --help            show this help message and exit
  --data DATA           The OSD data logical volume (vg/lv) or absolute path to device
  --journal JOURNAL     Logical Volume (vg/lv) or path to GPT partition
  --zap-disk            DEPRECATED - cannot zap when creating an OSD
  --fs-type FS_TYPE     filesystem to use to format DEVICE (xfs, btrfs)
  --dmcrypt             use dm-crypt on DEVICE
  --dmcrypt-key-dir KEYDIR
                        directory where dm-crypt keys are stored
  --filestore           filestore objectstore
  --bluestore           bluestore objectstore
  --block-db BLOCK_DB   bluestore block.db path
  --block-wal BLOCK_WAL
                        bluestore block.wal path
  --debug               Enable debug mode on remote ceph-volume calls

说明：

  1. data 存放 Ceph object 数据
  2. block-db 存放元数据
  3. block-wal 存放rocksdb数据库日志数据
  • 查看 OSD 的命令
ceph osd stat
ceph osd dump
ceph osd ls

从 RADOS 集群移除 OSD

  • L版本以及之后

设备故障时，移除设备的命令如下：

# 停用设备
ceph osd out <osd-num>

# 停止进程
sudo systemctl stop ceph-osd@<osd-num>

# 移除设备
ceph osd purge <osd-id> [--yes-i-really-mean-it]

若 ceph.conf 配置文件中有 [osd.id]\n host=<c1...> 需要手动删除

  • L版之前
# 从 CRUSH 运行图中移除设备
ceph osd crush remove <osd-num>

# 移除 OSD 认证key
ceph auth del <osd-num>

# 移除设备
ceph osd rm <osd-num>

扩展 mon 节点

Ceph 使用 Pasos 算法（可用节点数大于总监视器数量的一半）判断 mon 是否正常，一般配置奇数个监视器

ceph-deploy mon add c2
  • 查看mon和法定人的信息
ceph quorum_status --format json-pretty

扩展 mgr 节点

Ceph Manager 守护进程以Active/Standby模式运行

ceph-deploy mgr create c2

测试

测试 Ceph 集群

最近更新
相关文章
最新评论
Home Archives Categories Tags Statistics
本文总阅读量 次 本站总访问量 次 本站总访客数 人