1
2
3
4
5
6
7
作者：李晓辉

联系方式：

1. 微信：Lxh_Chat

2. 邮箱：939958092@qq.com

部署红帽 Ceph 存储 iSCSI 网关前，需满足下列前提条件：

  • 安装运行有红帽企业 Linux 8.3 或更新版本的 iSCSI 网关节点。

  • 拥有运行红帽 Ceph 存储 5 或更高版本的运行集群。

  • iSCSI 网关节点上要有 90MiB 的 RAM 用于作为目标公开的每个 RBD 镜像。

  • 在每个 Ceph iSCSI 节点的防火墙上打开 TCP 端口 3260 和 5000。

  • 创建新的 RADOS 块设备或使用现有可用设备。

创建iSCSI池

创建一个iSCSI存储池，用于存放iSCSI的存储卷

1
2
3
4
5
[root@serverc ~]# ceph osd pool create iscsipool
pool 'iscsipool' created
[root@serverc ~]# ceph osd pool application enable iscsipool rbd
enabled application 'rbd' on pool 'iscsipool'
[root@serverc ~]# rbd pool init -p iscsipool

部署iSCSI进程

iSCSI部署格式为：

1
ceph orch apply iscsi <pool> <api_user> <api_password> [<trusted_ip_list>]

将iSCSI进程部署到相应机器上，api账号密码为admin

1
2
3
4
5
6
[root@serverc ~]# ceph orch apply iscsi iscsipool \
admin admin 172.25.250.12 --placement='serverc.lab.example.com'
Scheduled iscsi.iscsipool update...

[root@serverc ~]# ceph orch ps | grep iscsi
iscsi.iscsipool.serverc.ydnyxu      serverc.lab.example.com  running (31s)  26s ago    31s  -              3.5               2142b60d7974  d9d369e7bb14

列出iSCSI网关

1
2
[root@serverc ~]# ceph dashboard iscsi-gateway-list
{"gateways": {"serverc.lab.example.com": {"service_url": "http://admin:admin@172.25.250.12:5000"}}}

配置iSCSI后端

这个步骤，用WebUi其实更方便

从上面的步骤中，可以看到iSCSI容器的id，用podman exec进入配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[root@serverc ~]# podman exec -it d9d369e7bb14 gwcli
Warning: Could not load preferences file /root/.gwcli/prefs.bin.
/> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29796506K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- iscsipool ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ....................................................................................................... [0.00Y, Disks: 0]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 0]
/>

创建iSCSI target

进到iSCSI子系统的/iscsi-targets，并完成target创建

1
2
/iscsi-targets> create iqn.2024-08.com.example.lab:test
ok

看看创建完是什么样子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
/iscsi-targets> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29796506K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- iscsipool ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ....................................................................................................... [0.00Y, Disks: 0]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 0]
      o- disks .......................................................................................................... [Disks: 0]
      o- gateways ............................................................................................ [Up: 0/0, Portals: 0]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0]

创建iSCSI gateway

创建一个用于客户端的target，记得要先创建本地的主机，主机名和IP地址要核对好，这个是客户端连接的地址

这里可以配置多个网关来实现多路径连接，比如多个网卡

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
/iscsi-targets> cd /iscsi-targets/iqn.2024-08.com.example.lab:test/gateways

/iscsi-target...test/gateways> create serverc.lab.example.com 172.25.250.12,172.25.249.12
Adding gateway, sync'ing 0 disk(s) and 0 client(s)
ok
/iscsi-target...test/gateways> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph .......................................................................................................... [HEALTH_WARN]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29656514K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29656514K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29656514K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29656514K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29656514K (0%), Used: 0.00Y]
  |   | o- iscsipool .......................................................... [(x3), Commit: 100G/29656514K (353%), Used: 314304K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29656514K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ........................................................................................................ [100G, Disks: 1]
  | o- iscsipool ................................................................................................ [iscsipool (100G)]
  |   o- lxhtestiscsi ...................................................................... [iscsipool/lxhtestiscsi (Online, 100G)]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com .............................................................. [172.25.250.12,172.25.249.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0]

授权客户端

添加好客户端的initiatorname

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/iscsi-target...test/gateways> cd /iscsi-targets/iqn.2024-08.com.example.lab:test/hosts
/iscsi-target...ab:test/hosts> create iqn.1994-05.com.redhat:cae4921ac1f
ok


/iscsi-target...t:cae4921ac1f> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29796506K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 0.00Y]
  |   | o- iscsipool ............................................................... [(x3), Commit: 0.00Y/29796506K (0%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29796506K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ....................................................................................................... [0.00Y, Disks: 0]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com ............................................................................ [172.25.250.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:cae4921ac1f ........................................................ [Auth: None, Disks: 0(0.00Y)]

上面的iqn这部分是客户端的iqn号，要和客户端完全一致，如果cat时文件不存在，就需要安装iscsi客户端

1
2
3
[root@clienta ~]# yum install iscsi-initiator-utils -y
[root@clienta ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:cae4921ac1f

在Ceph中创建并存储lun

进到/disks，在ceph的iSCSI存储池中，创建一个100G且名为lxhtestiscsi的硬盘

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
/iscsi-target...ab:test/disks> cd /disks
/disks> create pool=iscsipool image=lxhdisk size=100G
ok

/disks> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29778650K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- iscsipool .............................................................. [(x3), Commit: 100G/29778650K (352%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ........................................................................................................ [100G, Disks: 2]
  | o- iscsipool ................................................................................................ [iscsipool (100G)]
  |   o- lxhdisk ............................................................................... [iscsipool/lxhdisk (Unknown, 100G)]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com .............................................................. [172.25.250.12,172.25.249.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:cae4921ac1f ........................................................ [Auth: None, Disks: 0(0.00Y)]

添加存储Lun到iSCSI

将磁盘添加到客户端可用的列表中

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
/disks> cd /iscsi-targets/iqn.2024-08.com.example.lab:test/hosts/iqn.1994-05.com.redhat:cae4921ac1f
/iscsi-target...t:cae4921ac1f> disk add iscsipool/lxhdisk
ok

/iscsi-target...t:cae4921ac1f> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29778650K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- iscsipool .............................................................. [(x3), Commit: 100G/29778650K (352%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ........................................................................................................ [100G, Disks: 2]
  | o- iscsipool ................................................................................................ [iscsipool (100G)]
  |   o- lxhdisk ............................................................................... [iscsipool/lxhdisk (Unknown, 100G)]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 1]
      | o- iscsipool/lxhdisk .............................................................. [Owner: serverc.lab.example.com, Lun: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com .............................................................. [172.25.250.12,172.25.249.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:cae4921ac1f ......................................................... [Auth: None, Disks: 1(100G)]
          o- lun 0 ....................................................... [iscsipool/lxhdisk(100G), Owner: serverc.lab.example.com]

创建客户端使用的账号密码

添加CHAP用户密码用于安全

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
/iscsi-target...t:cae4921ac1f> cd /iscsi-targets/iqn.2024-08.com.example.lab:test/hosts/iqn.1994-05.com.redhat:cae4921ac1f
/iscsi-target...t:cae4921ac1f> auth username=lixiaohui password=lixiaohuitest
ok

/iscsi-target...t:cae4921ac1f> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29778650K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- iscsipool .............................................................. [(x3), Commit: 100G/29778650K (352%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ........................................................................................................ [100G, Disks: 2]
  | o- iscsipool ................................................................................................ [iscsipool (100G)]
  |   o- lxhdisk ................................................................................ [iscsipool/lxhdisk (Online, 100G)]
  |   o- lxhtestiscsi .................................................................................................. [NOT FOUND]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 1]
      | o- iscsipool/lxhdisk .............................................................. [Owner: serverc.lab.example.com, Lun: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com .............................................................. [172.25.250.12,172.25.249.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:cae4921ac1f ......................................................... [Auth: CHAP, Disks: 1(100G)]
          o- lun 0 ....................................................... [iscsipool/lxhdisk(100G), Owner: serverc.lab.example.com]

预览iSCSI后端配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
/iscsi-target...t:cae4921ac1f> cd /
/> ls /
o- / ......................................................................................................................... [...]
  o- cluster ......................................................................................................... [Clusters: 1]
  | o- ceph ............................................................................................................ [HEALTH_OK]
  |   o- pools .......................................................................................................... [Pools: 7]
  |   | o- .rgw.root ............................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 48K]
  |   | o- default.rgw.control ................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- default.rgw.log ........................................................ [(x3), Commit: 0.00Y/29778650K (0%), Used: 408K]
  |   | o- default.rgw.meta ...................................................... [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- device_health_metrics ................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 0.00Y]
  |   | o- iscsipool .............................................................. [(x3), Commit: 100G/29778650K (352%), Used: 24K]
  |   | o- rbdtest ................................................................. [(x3), Commit: 0.00Y/29778650K (0%), Used: 72K]
  |   o- topology ................................................................................................ [OSDs: 9,MONs: 4]
  o- disks ........................................................................................................ [100G, Disks: 2]
  | o- iscsipool ................................................................................................ [iscsipool (100G)]
  |   o- lxhdisk ................................................................................ [iscsipool/lxhdisk (Online, 100G)]
  o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
    o- iqn.2024-08.com.example.lab:test .................................................................. [Auth: None, Gateways: 1]
      o- disks .......................................................................................................... [Disks: 1]
      | o- iscsipool/lxhdisk .............................................................. [Owner: serverc.lab.example.com, Lun: 0]
      o- gateways ............................................................................................ [Up: 1/1, Portals: 1]
      | o- serverc.lab.example.com .............................................................. [172.25.250.12,172.25.249.12 (UP)]
      o- host-groups .................................................................................................. [Groups : 0]
      o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 1]
        o- iqn.1994-05.com.redhat:cae4921ac1f ......................................................... [Auth: CHAP, Disks: 1(100G)]
          o- lun 0 ....................................................... [iscsipool/lxhdisk(100G), Owner: serverc.lab.example.com]

确认无误后exit退出

1
2
/> exit
[root@serverc ~]#

开通防火墙

iscsi默认用的3260端口，添加到防火墙例外

1
2
3
4
[root@serverc ~]# firewall-cmd --add-port=3260/tcp --add-port=5000/tcp --permanent
success
[root@serverc ~]# firewall-cmd --reload
success

配置 iSCSI 客户端

安装必备软件包

如果不需要多路径，就不需要额外安装配置device-mapper-multipath

1
[root@clienta ~]# yum install iscsi-initiator-utils device-mapper-multipath -y

配置多路径I/O

启用并创建默认的多路径配置

1
[root@clienta ~]# mpathconf --enable --with_multipathd y

将以下内容追加到 /etc/multipath.conf 文件

1
2
3
4
5
6
7
8
9
10
11
12
13
14
devices {
    device {
        vendor                 "LIO-ORG"
        hardware_handler       "1 alua"
        path_grouping_policy   "failover"
        path_selector          "queue-length 0"
        failback               60
        path_checker           tur
        prio                   alua
        prio_args              exclusive_pref_bit
        fast_io_fail_tmo       25
        no_path_retry          queue
    }
}

以下是解释

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
devices {
    # 开始定义一个设备配置块
    device {
        # 定义设备特定的配置
        vendor                 "LIO-ORG"
        # 指定设备供应商，这里是LIO-ORG，通常是一个正则表达式匹配

        hardware_handler       "1 alua"
        # 指定硬件处理程序，"1 alua"表示使用ALUA（Asymmetric Logical Unit Access）策略，并且优先级为1

        path_grouping_policy   "failover"
        # 定义路径分组策略，"failover"表示当一条路径失败时，流量将切换到其他路径

        path_selector          "queue-length 0"
        # 定义选择活动路径的算法，"queue-length 0"表示选择队列长度为0的路径，即没有等待I/O的路径

        failback               60
        # 定义故障恢复时间，单位为秒。这里设置为60秒，表示在故障路径恢复后，将等待60秒再切换回该路径

        path_checker           tur
        # 定义路径检查器，"tur"表示使用Turbo模式，这是一种快速路径验证方法

        prio                   alua
        # 定义路径选择优先级，"alua"表示根据ALUA优先级选择路径

        prio_args              exclusive_pref_bit
        # 定义优先级参数，"exclusive_pref_bit"表示优先选择设置了独占偏好位的路径

        fast_io_fail_tmo       25
        # 定义快速I/O失败的超时时间，单位为秒。这里设置为25秒

        no_path_retry          queue
        # 当没有可用路径时的重试策略，"queue"表示将I/O请求排队等待
    }
} # 结束设备配置块

重新启动 multipathd 服务

1
[root@clienta ~]# systemctl restart multipathd

配置iscsi认证

在文件中找到这几个参数改一下，启用CHAP认证，并设置账号密码，或者直接在文件后加入也行

1
2
3
4
[root@cephbackup ~]# vim /etc/iscsi/iscsid.conf
node.session.auth.authmethod = CHAP
node.session.auth.username = lixiaohui
node.session.auth.password = lixiaohui0608

重启iSCSI相关服务

1
systemctl restart iscsi iscsid

执行iscsi发现

这里成功发现了多路径

1
2
3
[root@clienta ~]# iscsiadm -m discovery -t st -p 172.25.250.12
172.25.250.12:3260,1 iqn.2024-08.com.example.lab:test
172.25.249.12:3260,1 iqn.2024-08.com.example.lab:test

登陆iscsi

1
2
3
4
5
[root@clienta ~]# iscsiadm -m node -T iqn.2024-08.com.example.lab:test -l
Logging in to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.250.12,3260]
iscsiadm: Could not login to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.250.12,3260].
iscsiadm: initiator reported error (24 - iSCSI login failed due to authorization failure)
iscsiadm: Could not log into all portals

如果这一步登录失败，大概率是没有正确识别到用户名和密码，我们可以用下面的方法更新

1
2
3
4
5
6
iscsiadm -m node -T iqn.2024-08.com.example.lab:test \
--op update -n node.session.auth.authmethod -v CHAP
iscsiadm -m node -T iqn.2024-08.com.example.lab:test \
--op update -n node.session.auth.username -v lixiaohui
iscsiadm -m node -T iqn.2024-08.com.example.lab:test \
--op update -n node.session.auth.password -v lixiaohuitest

确认密码成功添加

1
2
3
4
5
6
7
8
9
[root@clienta ~]# iscsiadm -m node -T iqn.2024-08.com.example.lab:test --op
delete  new     show    update
[root@clienta ~]# iscsiadm -m node -T iqn.2024-08.com.example.lab:test --op show | grep 'node.session.auth'
node.session.auth.authmethod = CHAP
node.session.auth.username = lixiaohui
node.session.auth.password = ********
node.session.auth.username_in = <empty>
node.session.auth.password_in = <empty>
node.session.auth.chap_algs = MD5

再次登录

成功通过多个路径登录，在一条路断了的时候，可以用另一个路连接

1
2
3
4
5
[root@clienta ~]# iscsiadm -m node -T iqn.2024-08.com.example.lab:test -l
Logging in to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.250.12,3260]
Logging in to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.249.12,3260]
Login to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.250.12,3260] successful.
Login to [iface: default, target: iqn.2024-08.com.example.lab:test, portal: 172.25.249.12,3260] successful.

查询设备列表

发现在系统中添加了两个设备，这是因为我们用多路径管理的

1
2
3
4
5
6
[root@clienta ~]# lsblk
NAME     MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdb        8:16   0  100G  0 disk
└─mpatha 253:0    0  100G  0 mpath
sdc        8:32   0  100G  0 disk
└─mpatha 253:0    0  100G  0 mpath

对多路径设备进行分区

这里不要对单个硬盘进行分区，这个没用，需要要对多路径设备分区

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
[root@clienta ~]# fdisk /dev/mapper/mpatha

Welcome to fdisk (util-linux 2.32.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (2048-209715199, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199):

Created a new partition 1 of type 'Linux' and of size 100 GiB.

Command (m for help): w
The partition table has been altered.
Failed to add partition 1 to system: Invalid argument

The kernel still uses the old partitions. The new table will be used at the next reboot.
Syncing disks.

检查分区是否就绪

1
2
3
4
5
6
7
8
[root@clienta ~]# lsblk
NAME        MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdb           8:16   0  100G  0 disk
└─mpatha    253:0    0  100G  0 mpath
  └─mpatha1 253:1    0  100G  0 part
sdc           8:32   0  100G  0 disk
└─mpatha    253:0    0  100G  0 mpath
  └─mpatha1 253:1    0  100G  0 part

格式化挂载使用

1
2
3
4
5
6
7
8
9
10
11
12
[root@clienta ~]# mkfs.xfs /dev/mapper/mpatha1
meta-data=/dev/mapper/mpatha1    isize=512    agcount=4, agsize=6553536 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=26214144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=12799, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
Discarding blocks...Done.

挂载使用

1
2
3
4
5
6
7
8
9
10
11
[root@clienta ~]# mount /dev/mapper/mpatha1 /mnt
[root@clienta ~]# dd if=/dev/zero of=/mnt/out.file bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.0902139 s, 1.2 GB/s

[root@clienta ~]# ls /mnt
out.file

[root@clienta ~]# df -h | grep mnt
/dev/mapper/mpatha1  100G  846M  100G   1% /mnt

如果要永久写到/etc/fstab中实现自动挂载，要在option选项处加上_netdev，不然系统就起不来了

1
2
[root@clienta ~]# tail -n 1 /etc/fstab
/dev/mapper/mpatha1 /mnt xfs defaults,_netdev 0 0