1
2
3
4
5
6
7
作者：李晓辉

联系方式：

1. 微信：Lxh_Chat

2. 邮箱：939958092@qq.com

实验目标：将serverc上rbdtest池中的lxhrbdname镜像和serverf的rbdtest池中的同名镜像保持同步

创建RBD源池

创建一个名为rbdtest的池，稍后将会和第二个集群进行池同步

1
2
3
4
5
[root@serverc ~]# ceph osd pool create rbdtest
pool 'rbdtest' created
[root@serverc ~]# ceph osd pool application enable rbdtest rbd
enabled application 'rbd' on pool 'rbdtest'
[root@serverc ~]# rbd pool init -p rbdtest

创建备份集群目标RBD池

请参考ceph部署完成第二套集群部署

创建第二套集群中同名池为rbdtest的rbd池

1
2
3
4
5
[root@serverf ~]# ceph osd pool create rbdtest
pool 'rbdtest' created
[root@serverf ~]# ceph osd pool application enable rbdtest rbd
enabled application 'rbd' on pool 'rbdtest'
[root@server ~]# rbd pool init -p rbdtest

创建源镜像

在源集群的rbdtest池中创建一个名为lxhrbdname的镜像，大小为1000M，启用 exclusive-lock 和 journaling RBD 镜像功能

注意到mirroring state: disabled

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@serverc ~]# rbd create --size 1000M -p rbdtest lxhrbdname \
--image-feature=exclusive-lock,journaling
[root@serverc ~]# rbd info -p rbdtest lxhrbdname
rbd image 'lxhrbdname':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: dbc45a6069eb
        block_name_prefix: rbd_data.dbc45a6069eb
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Wed Aug  7 05:22:20 2024
        access_timestamp: Wed Aug  7 05:22:20 2024
        modify_timestamp: Wed Aug  7 05:22:20 2024
        journal: dbc45a6069eb
        mirroring state: disabled

启用池模式mirror

在源集群中的rbdtest池上启用mirror同步，注意info中的同步信息

注意到mirroring state: enabled

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
[root@serverc ~]# rbd mirror pool enable rbdtest pool
[root@serverc ~]# rbd info -p rbdtest lxhrbdname
rbd image 'lxhrbdname':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: dbc45a6069eb
        block_name_prefix: rbd_data.dbc45a6069eb
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Wed Aug  7 05:22:20 2024
        access_timestamp: Wed Aug  7 05:22:20 2024
        modify_timestamp: Wed Aug  7 05:22:20 2024
        journal: dbc45a6069eb
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: d91f0fe4-4f27-4df2-88b1-38e3ffe9b177
        mirroring primary: true

准备源同步token

在源集群中生成同步的key，将源集群的站点命名为site1，保存好，稍后用于复制到第二套集群

1
2
3
4
[root@serverc ~]# rbd mirror pool peer bootstrap create \
--site-name site1 rbdtest > site1.key
[root@serverc ~]# cat site1.key
eyJmc2lkIjoiMmFlNmQwNWEtMjI5YS0xMWVjLTky...
1
2
[root@serverc ~]# scp -p site1.key root@serverf:/root
site1.key                   100%  253   142.4KB/s   00:00

目标集群部署rbd-mirror进程

1
2
3
4
[root@serverf ~]# ceph orch apply rbd-mirror --placement=serverf.lab.example.com
Scheduled rbd-mirror update...
[root@serverf ~]# ceph orch ps | grep mirror
rbd-mirror.serverf.veenfh           serverf.lab.example.com  running (5s)  1s ago     5s   -              16.2.0-117.el8cp  2142b60d7974  c9b6b8ff89e8

导入源集群token密钥

导入源集群的key，并命名本地集群为backupceph，从info信息中看到mirroring primary: false

忽略包含以下文本的已知错误：auth: unable to find a keyring on …

import这一步执行成功后，在备份集群就已经可以看到源镜像了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
[root@serverf ~]# rbd mirror pool peer bootstrap import \
--site-name backupceph --direction rx-only rbdtest /root/site1.key
[root@serverf ~]#
[root@cephbackup ~]# rbd -p rbdtest ls
lxhrbdname
[root@serverf ~]# rbd -p rbdtest info lxhrbdname
rbd image 'lxhrbdname':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 37a88b1c5510
        block_name_prefix: rbd_data.37a88b1c5510
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Sun Sep 18 15:18:32 2022
        access_timestamp: Sun Sep 18 15:18:32 2022
        modify_timestamp: Sun Sep 18 15:18:32 2022
        journal: 37a88b1c5510
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 165a0e83-8e8b-43cb-8d93-6b1812e3c58c
        mirroring primary: false

查询同步状态

在两台机器上，分别执行这个命令，可以看到两个站点的信息

1
2
3
4
5
6
7
8
9
10
[root@serverf ~]# rbd mirror pool info rbdtest
Mode: pool
Site Name: backupceph

Peer Sites:

UUID: 6b9fb207-4555-4637-8954-f637453081c9
Name: site1
Direction: rx-only
Client: client.rbd-mirror-peer

在备份的集群中，查询守护进程和镜像的健康状态

1
2
3
4
5
6
[root@serverf ~]# rbd mirror pool status -p rbdtest
health: OK
daemon health: OK
image health: OK
images: 1 total
    1 replaying

再次到源集群去看看镜像状态

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
[root@serverc ~]# rbd -p rbdtest info lxhrbdname
rbd image 'lxhrbdname':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: d3d46acc6e3f
        block_name_prefix: rbd_data.d3d46acc6e3f
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:12:16 2023
        access_timestamp: Fri Sep  8 12:12:16 2023
        modify_timestamp: Fri Sep  8 12:12:16 2023
        journal: d3d46acc6e3f
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 57379079-48cc-4091-b08c-01e087562d9b
        mirroring primary: true

再次到源集群去看看池的同步状态

1
2
3
4
5
6
7
8
9
10
[root@serverc ~]# rbd mirror pool info rbdtest
Mode: pool
Site Name: site1

Peer Sites:

UUID: e060d31c-94c7-495d-8612-4ae890e0a4e7
Name: backupceph
Mirror UUID: eac9e8a9-b13c-46f3-9997-6c22555b77b7
Direction: tx-only

创建新镜像测试是否同步

重新在源集群的池中创建一个镜像，稍后再去第二套集群看看是否同步成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
[root@serverc ~]# rbd create --size 1000M -p rbdtest image2 \
> --image-feature=exclusive-lock,journaling
[root@serverc ~]# rbd info image2 -p rbdtest
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: d420d153d504
        block_name_prefix: rbd_data.d420d153d504
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:23:23 2023
        access_timestamp: Fri Sep  8 12:23:23 2023
        modify_timestamp: Fri Sep  8 12:23:23 2023
        journal: d420d153d504
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: true

在第二套集群中查看，发现已经同步成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
[root@serverf ~]# rbd -p rbdtest ls
image2
lxhrbdname
[root@serverf ~]# rbd -p rbdtest info image2
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 8575f69a1f
        block_name_prefix: rbd_data.8575f69a1f
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:23:25 2023
        access_timestamp: Fri Sep  8 12:23:25 2023
        modify_timestamp: Fri Sep  8 12:23:25 2023
        journal: 8575f69a1f
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: false

测试删除镜像的同步

这个删除，只能从源集群删除

1
2
3
4
5
6
7
[root@serverc ~]# rbd ls -p rbdtest
image2
lxhrbdname
[root@serverc ~]# rbd rm lxhrbdname -p rbdtest
Removing image: 100% complete...done.
[root@serverc ~]# rbd ls -p rbdtest
image2

到备份集群看看删除成功没

1
2
[root@serverf ~]# rbd ls -p rbdtest
image2

一些额外的mirror命令：

如果两个peer集群之间的状态不一致，rbd-mirror守护进程不会尝试mirror不一致的映像，在备份集群上使用rbd mirror image resync重新同步映像

1
2
[root@serverf ~]# rbd mirror image resync rbdtest/image2
Flagged image for resync from primary

使用rbd mirror image enable或rbd mirror image disable在两个对端存储集群上的整个池image模式中启用或禁用mirroring模式

1
2
rbd mirror image enable mypool/myimage
rbd mirror image disable mypool/myimage

使用基于快照的镜像，通过禁用镜像和启用快照，将基于日志的镜像转换为基于快照的镜像

1
2
rbd mirror image disable mypool/myimage
rbd mirror image enable mypool/myimage snapshot

故障转移过程

如果主要 RBD 镜像变得不可用，您可按照步骤启动对次要 RBD 镜像的访问：

  • 停止访问主要 RBD 镜像。也就是说，停止正在使用该镜像的所有应用和虚拟机。

  • 使用 rbd mirror image demote rbdtest/image2 命令来降级主要 RBD 镜像。

  • 使用 rbd mirror image promote rbdtest/image2 命令来升级次要 RBD 镜像。

  • 恢复访问 RBD 镜像。重新启动应用和虚拟机。

在非有序关机后发生故障转移时，您必须从备份存储集群中的 Ceph 监控器节点升级非主要镜像。使用 --force 选项，因为降级无法传播到主要存储集群

降级集群镜像

降级操作必须是源集群执行，除非源集群不存在，可以尝试用–force

我们发现降级后，serverc就不再是primary了

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
[root@serverc ~]# rbd info rbdtest/image2
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: d420d153d504
        block_name_prefix: rbd_data.d420d153d504
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:23:23 2023
        access_timestamp: Fri Sep  8 12:23:23 2023
        modify_timestamp: Fri Sep  8 12:23:23 2023
        journal: d420d153d504
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: true

[root@serverc ~]# rbd mirror image demote rbdtest/image2
Image demoted to non-primary

[root@serverc ~]# rbd info rbdtest/image2
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: d420d153d504
        block_name_prefix: rbd_data.d420d153d504
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:23:23 2023
        access_timestamp: Fri Sep  8 12:23:23 2023
        modify_timestamp: Fri Sep  8 12:23:23 2023
        journal: d420d153d504
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: false

提升集群镜像

在备份集群提升了镜像后，serverf变成了primary

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
[root@serverf ~]# rbd info rbdtest/image2
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 85757ad71c90
        block_name_prefix: rbd_data.85757ad71c90
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:32:14 2023
        access_timestamp: Fri Sep  8 12:32:14 2023
        modify_timestamp: Fri Sep  8 12:32:14 2023
        journal: 85757ad71c90
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: false
[root@serverf ~]# rbd mirror image promote rbdtest/image2
Image promoted to primary
[root@serverf ~]# rbd info rbdtest/image2
rbd image 'image2':
        size 1000 MiB in 250 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 85757ad71c90
        block_name_prefix: rbd_data.85757ad71c90
        format: 2
        features: exclusive-lock, journaling
        op_features:
        flags:
        create_timestamp: Fri Sep  8 12:32:14 2023
        access_timestamp: Fri Sep  8 12:32:14 2023
        modify_timestamp: Fri Sep  8 12:32:14 2023
        journal: 85757ad71c90
        mirroring state: enabled
        mirroring mode: journal
        mirroring global id: 644301a8-22b5-46e8-9027-103ecfbc7f74
        mirroring primary: true