1
2
3
4
5
6
7
作者：李晓辉

联系方式：

1. 微信：Lxh_Chat

2. 邮箱：939958092@qq.com

介绍容器化服务

最新版本的tripleO部署项目中，OpenStack 将所有主要服务作为容器运行。 每个服务都有一个与主节点隔离的命名空间。 在部署期间，OpenStack会从 Red Hat Container Catalog 中提取并部署容器镜像。

容器镜像和源注册表

直接从公网拉镜像太慢了，tripleo部署可以在 undercloud 节点上的端⼝ 8787 创建本地注册表并⾸先将每个所需的服务镜像⼀次提取到本地注册表。

列出所需的镜像列表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
(undercloud) [stack@director ~]$ openstack tripleo container image prepare \
> default --local-push-destination \
> --output-env-file local_image.yaml
# Generated with the following on 2023-12-31T12:33:04.865220
#
#   openstack tripleo container image prepare default --local-push-destination --output-env-file local_image.yaml
#

parameter_defaults:
  ContainerImagePrepare:
  - push_destination: true
    set:
      ceph_alertmanager_image: ose-prometheus-alertmanager
      ceph_alertmanager_namespace: registry.redhat.io/openshift4
      ceph_alertmanager_tag: 4.1
      ceph_grafana_image: rhceph-4-dashboard-rhel8
      ceph_grafana_namespace: registry.redhat.io/rhceph
      ceph_grafana_tag: 4
      ceph_image: rhceph-4-rhel8
      ceph_namespace: registry.redhat.io/rhceph
      ceph_node_exporter_image: ose-prometheus-node-exporter
      ceph_node_exporter_namespace: registry.redhat.io/openshift4
      ceph_node_exporter_tag: v4.1
      ceph_prometheus_image: ose-prometheus
      ceph_prometheus_namespace: registry.redhat.io/openshift4
      ceph_prometheus_tag: 4.1
      ceph_tag: latest
      name_prefix: openstack-
      name_suffix: ''
      namespace: registry.redhat.io/rhosp-rhel8
      neutron_driver: ovn
      rhel_containers: false
      tag: '16.1'
    tag_from_label: '{version}-{release}'

查看本地现有的镜像列表

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[root@director ~]# podman images
REPOSITORY                                                                                                            TAG       IMAGE ID       CREATED       SIZE
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-object                16.1-55   1d8c985afd81   3 years ago   695 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-server              16.1-53   74dfee7bc620   3 years ago   1.06 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-l3-agent            16.1-51   dbbad7c440bb   3 years ago   1.07 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-mistral-api                 16.1-51   7c6833dec0c9   3 years ago   1.1 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-nova-api                    16.1-55   e228c67575a8   3 years ago   1.13 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-openvswitch-agent   16.1-53   f0edb026c05e   3 years ago   920 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-zaqar-wsgi                  16.1-51   e03a9f37d88a   3 years ago   687 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-ironic-conductor            16.1-51   935759d55b09   3 years ago   903 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-nova-scheduler              16.1-53   29078ec6058c   3 years ago   1.26 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-keystone                    16.1-50   7deabb5b6e47   3 years ago   736 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-proxy-server          16.1-50   e9f9f8ecd1b3   3 years ago   740 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-dhcp-agent          16.1-51   5d6a113bf44e   3 years ago   1.07 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-glance-api                  16.1-47   5c5b1930cd96   3 years ago   970 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-mistral-engine              16.1-52   5661f7b7d3c6   3 years ago   1.08 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-heat-engine                 16.1-51   af83b6596fff   3 years ago   897 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-account               16.1-50   6759a357497b   3 years ago   695 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-ironic-pxe                  16.1-51   fe0c73d47517   3 years ago   764 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-container             16.1-51   84dae609c44d   3 years ago   695 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-mistral-executor            16.1-62   dde9afb5cf2e   3 years ago   1.5 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-heat-api                    16.1-51   4d4612a8f640   3 years ago   897 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-ironic-neutron-agent        16.1-54   c55aff21979f   3 years ago   920 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-ironic-api                  16.1-51   b7d635c35a4d   3 years ago   758 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-nova-compute-ironic         16.1-54   6802409ed58c   3 years ago   1.9 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-nova-conductor              16.1-53   7a120462042b   3 years ago   1.05 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-mistral-event-engine        16.1-53   4f03510be342   3 years ago   1.08 GB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-ironic-inspector            16.1-55   8d5c0a04bdb2   3 years ago   661 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-placement-api               16.1-55   1ddaea6fdc7f   3 years ago   612 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-tempest                     16.1-56   c61fe83c250a   3 years ago   943 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-rabbitmq                    16.1-57   fc63a558787f   3 years ago   567 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-haproxy                     16.1-57   007cc42591fc   3 years ago   523 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-cron                        16.1-57   84b32ff4015f   3 years ago   390 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-keepalived                  16.1-56   e765f910ac00   3 years ago   404 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-memcached                   16.1-57   f215559b8344   3 years ago   411 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-iscsid                      16.1-56   79890bbdaee3   3 years ago   395 MB
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-mariadb                     16.1-59   8e3530d51950   3 years ago   738 MB

列出特定的字段

1
2
3
4
[root@director ~]# podman images --format="{{.Repository}}:{{.Tag}}"
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-object:16.1-55
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-server:16.1-53
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-neutron-l3-agent:16.1-51

可用的完整字段获取方式如下：

1
podman images --format="{{json .}}"

容器基本命令

若要列出某个特定节点上运⾏的容器，请使⽤ podman ps：

1
2
3
4
[root@director ~]# podman ps
CONTAINER ID  IMAGE                                        COMMAND               CREATED         STATUS             PORTS  NAMES

7a3eb90360f6  openstack-nova-compute-ironic:16.1-54        kolla_start           23 months ago   Up 39 minutes ago         nova_compute

内容太多了的话，可以加入各种的filter和format定制返回内容，可用的filter和format参数参加一下链接:

1
2
https://docs.podman.io/en/latest/markdown/podman-ps.1.html#filter-f
https://docs.podman.io/en/latest/markdown/podman-ps.1.html#format-format

Exited 服务状态并不是故障。执⾏⼀次性配置的服务将会在它们完成时退出

1
2
3
4
5
6
[root@director ~]# podman ps -a --format="table {{.Names}} {{.Status}}" | grep heat
heat_api_cron                                                Up 46 minutes ago
heat_api                                                     Up 46 minutes ago
heat_engine_db_sync                                          Exited (0) 23 months ago
heat_engine                                                  Up 46 minutes ago
heat_init_log                                                Exited (0) 3 years ago

podman stats 命令显⽰实时容器资源使⽤情况统计数据。

1
2
3
4
5
6
7
8
9
10
11
[root@director ~]# podman stats nova_compute
ID             NAME           CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO            PIDS
7a3eb90360f6   nova_compute   --      149.5MB / 7.608GB   1.96%   -- / --   25.98MB / 45.06kB   2
ID             NAME           CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO            PIDS
7a3eb90360f6   nova_compute   --      149.5MB / 7.608GB   1.96%   -- / --   25.98MB / 45.06kB   2
ID             NAME           CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO            PIDS
7a3eb90360f6   nova_compute   9.31%   149.5MB / 7.608GB   1.96%   -- / --   25.98MB / 45.06kB   2
ID             NAME           CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO            PIDS
7a3eb90360f6   nova_compute   1.08%   149.5MB / 7.608GB   1.96%   -- / --   25.98MB / 45.06kB   2
ID             NAME           CPU %   MEM USAGE / LIMIT   MEM %   NET IO    BLOCK IO            PIDS
7a3eb90360f6   nova_compute   7.78%   149.5MB / 7.608GB   1.96%   -- / --   25.98MB / 45.06kB   2

使⽤ podman images 命令来显⽰所有可⽤镜像的存储库名称、⼤⼩、标记、镜像 ID 和创建⽇期。

1
2
3
[root@director ~]# podman images
REPOSITORY                                                                                                            TAG       IMAGE ID       CREATED       SIZE
director.ctlplane.overcloud.example.com:8787/gls-dle-dev-osp16-osp16_containers-openstack-swift-object                16.1-55   1d8c985afd81   3 years ago   695 MB

podman inspect 命令以 JSON 格式显⽰容器配置。使⽤ jq 命令从 JSON 输出中过滤或提取数据。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
[root@director ~]# podman inspect nova_compute | more
[
    {
        "Id": "7a3eb90360f6369aef9b4d59e11f1fed9aac6ef808be9eeadcbd16c9a7128e92",
        "Created": "2022-01-21T09:13:05.398208492-05:00",
        "Path": "dumb-init",
        "Args": [
            "--single-child",
            "--",
            "kolla_start"
        ],
        "State": {
            "OciVersion": "1.0.1-dev",
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2557,
            "ConmonPid": 2503,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2023-12-31T12:15:26.262182764-05:00",
            "FinishedAt": "2022-02-25T03:28:28.773966842-05:00",
            "Healthcheck": {
                "Status": "",
                "FailingStreak": 0,
                "Log": null
            }

jq技巧

获取第一级所有字段

1
2
3
4
5
6
7
8
9
10
11
12
13
[root@director ~]# podman inspect nova_compute | jq '.[0] | keys'
[
  "AppArmorProfile",
  "Args",
  "BoundingCaps",
  "Config",
  "ConmonPidFile",
  "Created",
  "Dependencies",
  "Driver",
  "EffectiveCaps",
  "ExecIDs",
  "ExitCommand",

查看其第二级参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[root@director ~]# podman inspect nova_compute | jq '.[0].State | keys'
[
  "ConmonPid",
  "Dead",
  "Error",
  "ExitCode",
  "FinishedAt",
  "Healthcheck",
  "OOMKilled",
  "OciVersion",
  "Paused",
  "Pid",
  "Restarting",
  "Running",
  "StartedAt",
  "Status"
]

使⽤ podman logs 命令来显⽰容器的控制台⽇志。

1
2
3
4
5
6
7
8
9
10
11
[root@director ~]# podman logs keystone | more
+ sudo -E kolla_set_configs
INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json
INFO:__main__:Validating config file
INFO:__main__:Kolla config strategy set to: COPY_ALWAYS
INFO:__main__:Copying service configuration files
INFO:__main__:Deleting /etc/keystone/fernet-keys
INFO:__main__:Creating directory /etc/keystone/fernet-keys
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/keystone/fernet-keys/0 to /etc/keystone/fernet-keys/0
INFO:__main__:Copying /var/lib/kolla/config_files/src/etc/keystone/fernet-keys/1 to /etc/keystone/fernet-keys/1
INFO:__main__:Deleting /etc/httpd/conf.d

请使⽤ podman exec 命令在容器中运⾏命令

1
2
3
4
5
6
7
[root@director ~]# podman exec -it keystone /bin/bash
()[root@director /]# ls
bin  boot  dev  etc  home  lib  lib64  lost+found  media  mnt  openstack  opt  proc  root  run  run_command  sbin  srv  sys  tmp  usr  var
()[root@director /]# exit
exit
[root@director ~]# podman exec -it keystone cat /etc/hostname
director.lab.example.com

使⽤ systemd 服务管理容器

最新版本的 RHOSP 使⽤ systemd 单元来管理服务容器⽣命周期。systemd 执⾏启动、停⽌和其他常⻅操作，像管理其他 systemd 单元和服务⼀样管理容器，使⽤ podman 命令与容器交互。

1
2
3
4
5
6
7
8
[root@director ~]# systemctl status tripleo
Display all 122 possibilities? (y or n)
tripleo_glance_api_healthcheck.service                tripleo_mysql_healthcheck.service
tripleo_glance_api_healthcheck.timer                  tripleo_mysql_healthcheck.timer
tripleo_glance_api.service                            tripleo_mysql.service
tripleo_haproxy.service                               tripleo_neutron_api_healthcheck.service
tripleo_heat_api_cron_healthcheck.service             tripleo_neutron_api_healthcheck.timer
...
1
systemctl restart tripleo_nova_compute.service

systemd 使⽤ systemd 定时器来监控容器健康检查，若要列出容器计时器，请使⽤ systemctl list-timers 命令。

需要注意的是，timer一般会和同名的service关联，除非你指定了Portof参数，下面我们查了一个timer的内容

1
2
3
4
5
6
7
8
9
10
11
[root@director ~]# systemctl cat tripleo_heat_api_healthcheck.timer
# /etc/systemd/system/tripleo_heat_api_healthcheck.timer
[Unit]
Description=heat_api container healthcheck
PartOf=tripleo_heat_api.service
[Timer]
OnActiveSec=120
OnUnitActiveSec=60
RandomizedDelaySec=45.0
[Install]
WantedBy=timers.target

文件分解

  1. [Unit] 部分：

    • Description: 描述定时器的用途。

    • PartOf: 指定这个定时器属于 tripleo_heat_api.service。

  2. [Timer] 部分：

    • OnActiveSec=120: 定时器将在服务激活后 120 秒启动。

    • OnUnitActiveSec=60: 定时器将在服务的上一个活动周期结束 60 秒后再次启动。

    • RandomizedDelaySec=45.0: 每次定时器启动前增加最多 45 秒的随机延迟。

  3. [Install] 部分：

    • WantedBy=timers.target: 指定这个定时器将在 timers.target 启动时启动。

你要验证的话，可以不断的去systemctl status tripleo_heat_api_healthcheck，你会看到60多秒后，服务会启动一次

⽇志⽂件位置

标准输出（stdout）和标准错误（stderr）整合到每个容器的⼀个⽂件中，该⽂件位于 /var/log/containers/stdouts ⽬录中

1
2
3
4
5
6
7
8
[root@director ~]# ls -1 /var/log/containers/stdouts/ | more
container-puppet-crond.log
container-puppet-glance_api.log
container-puppet-haproxy.log
container-puppet-heat_api.log
container-puppet-heat.log
container-puppet-ironic_api.log
container-puppet-ironic_inspector.log

配置⽂件位置

容器配置⽂件位于 /var/lib/config-data/puppet-generated/container_name ⽬录中

1
2
3
4
5
6
7
8
9
10
11
[root@director ~]# ls -1 /var/lib/config-data/puppet-generated/ | more
crond
crond.md5sum
glance_api
glance_api.md5sum
haproxy
haproxy.md5sum
heat
heat_api
heat_api.md5sum
heat.md5sum

容器化服务的 systemd 单元⽂件使⽤前缀 tripleo_ 进⾏命名，因为它们是由 TripleO 安装的。

1
2
3
4
5
6
7
[root@director ~]# ls -1 /etc/systemd/system/tripleo* | more
/etc/systemd/system/tripleo_glance_api_healthcheck.service
/etc/systemd/system/tripleo_glance_api_healthcheck.timer
/etc/systemd/system/tripleo_glance_api.service
/etc/systemd/system/tripleo_haproxy.service
/etc/systemd/system/tripleo_heat_api_cron_healthcheck.service
/etc/systemd/system/tripleo_heat_api_cron_healthcheck.timer