Ironic Inspector安装、使用及流程分析

发表于 更新于 分类于

Hardware introspection for OpenStack Bare Metal

This is an auxiliary service for discovering hardware properties for a node managed by Ironic. Hardware introspection or hardware properties discovery is a process of getting hardware parameters required for scheduling from a bare metal node, given it’s power management credentials (e.g. IPMI address, user name and password).

安装与配置ironic-inspector

ironic-inspector的流程说明：

ironic-inspector使用ironic-agent镜像，以及自身提供的一个dnsmasq服务预先布置BM，主要的工作流程为：

  1. 通过ironic api进入inspect阶段（也可以使用ironic-inspector的api）
  2. ironic启动BM（如BM已启动，则关闭再启动），BM进入pxe启动阶段；通过ironic-inspector-dnsmasq分配ip地址，成功后使用tftp将ironic-agent镜像传输至BM；BM从ironic-agent镜像启动
  3. ironic-python-agent开始工作，其与ironic-inspector取得通信，根据inspector中的rule，对BM进行inspect动作，主要目的为获取BM的硬件信息；收集完毕后ipa将数据传输给ironic-inspector；关闭BM
  4. ironic-inspector根据配置中store_data选择的driver来存储收集到的数据；inspect阶段完成

安装软件包

添加rdo的ocata源：

1
yum install https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-2.noarch.rpm -y

下载ironic-inspector及其client的rpm包：

1
yum install openstack-ironic-inspector python-ironic-inspector-client -y

（可用安装源码代替，但是需要手动配置，源码从github上openstack对应模块的stable/ocata branch下载）

创建ironic-inspector的认证信息及endpoint

1
2
3
4
5
6
openstack user create --domain default --project services --project-domain default --password ironic --enable ironic-inspector
openstack service create --name ironic-inspector --description 'Bare Metal Introspection Service' --enable baremetal-introspection
openstack role add --user ironic-inspector --project services --project-domain default --user-domain default admin
openstack endpoint create  --region RegionOne --enable ironic-inspector admin http://{ironic-inspector-server-address}:5050
openstack endpoint create  --region RegionOne --enable ironic-inspector internal http://{ironic-inspector-server-address}:5050
openstack endpoint create  --region RegionOne --enable ironic-inspector public http://{ironic-inspector-server-address}:5050

创建ironic-inspector的数据库

1
2
3
4
mysql -e "create database ironic_inspector;"
mysql -e "grant all on ironic_inspector.* to ironic_inspector@'localhost' identified by 'ironic_inspector';"
mysql -e "grant all on ironic_inspector.* to ironic_inspector@'%' identified by 'ironic_inspector';"
mysql -e "flush privileges;"
  • ironic-inspector包含有两个服务，一个是ironic-inspector服务，用于和ipa协作完成inspect流程任务；另一个为ironic-inspector-dnsmasq服务，用于在inspect阶段承担dhcp、tftp功能。

配置ironic-inspector服务

ps:以下仅显示必要配置项

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
vim /etc/ironic-inspector/inspector.conf

[DEFAULT]
listen_address = 0.0.0.0
listen_port = 5050
auth_strategy = keystone
debug = false
verbose = true

[database]
connection = mysql://ironic_inspector:ironic_inspector@{ironic-inspector-db-address}/ironic_inspector?charset=utf8

[firewall]
dnsmasq_interface = br-inspect //inspector-dnsmasq使用的网桥，用于发送dhcp、tftp的报文

[ironic]
auth_url = http://{keystone-address}:5000/v3
auth_strategy = keystone
auth_type = password
default_domain_name = default //使用keystonev3的默认domain，根据实际情况替换
ironic_url = http://{ironic-server-address}:6385/v1
os_region = RegionOne
password = {ironic-password}
project_domain_name = default //使用keystonev3的默认domain，根据实际情况替换
username = {ironic-username}

[keystone_authtoken]
project_name = services
password = ironic
username = ironic-inspector
auth_url = http://{keystone-address}:35357/v2.0
auth_type = password
region_name = RegionOne

[processing]
add_ports = all //用于发现bm的网卡，all表示添加所有bm网卡，pxe表示只添加pxe启动的那块网卡
keep_ports = all //用于决定保留哪些网卡，all表示保留所有添加的网卡，present表示只保留当前使用的网卡
ramdisk_logs_dir = /var/log/ironic-inspector/ramdisk
store_data = none //用于存储inspect过程收集出的数据的driver，可以使用swift，none表示不存储

安装及配置tftp、ironic-inspector-dnsmasq服务

安装tftp服务

1
yum install tftp-server -y

配置tftp与dhcp服务

ps:以下仅显示必要配置项
tftp通过xinetd来守护进程，其配置如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim /etc/xinetd.d/tftp

service tftp
{
    socket_type        = dgram
    protocol        = udp
        port            = 69
    wait            = yes
    user            = root
    server            = /usr/sbin/in.tftpd
        server_args        = -v -v -v -v -v --map-file /tftpboot/map-file /tftpboot
    #server_args        = -s /var/lib/tftpboot
    disable            = no
    per_source        = 11
    cps            = 100 2
    flags            = IPv4
}

tftp目录结构：

1
2
3
4
5
6
7
8
9
[root@ironic ~(keystone_admin_46)]# tree /tftpboot/
/tftpboot/
├── chain.c32 // 拷贝/usr/share/syslinux/chain.c32到此处
├── ironic-agent.initramfs // 拷贝diskimage-builder生成的ironic-agent.initramfs到此处
├── ironic-agent.kernel // 拷贝diskimage-builder生成的ironic-agent.kernel到此处
├── map-file
├── pxelinux.0 // 拷贝/usr/share/syslinux/pxelinux.0到此处
└── pxelinux.cfg
    └── default

map-file内容：

1
2
3
4
5
[root@ironic tftpboot(keystone_admin)]# cat map-file
re ^(/tftpboot/) /tftpboot/\2
re ^/tftpboot/ /tftpboot/
re ^(^/) /tftpboot/\1
re ^([^/]) /tftpboot/\1

pxelinux.cfg/default内容如下，如果需要收集lldp信息，需要加上ipa-collect-lldp=True：

1
2
3
4
5
6
7
8
[root@ironic tftpboot(keystone_admin)]# cat pxelinux.cfg/default
default introspect

label introspect
kernel ironic-agent.kernel
append initrd=ironic-agent.initramfs ipa-inspection-callback-url=http://{ironic-inspector-server-address}:5050/v1/continue systemd.journald.forward_to_console=yes ipa-collect-lldp=True

ipappend 3

ironic-inspector-dnsmasq的配置文件如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/etc/ironic-inspector/dnsmasq.conf

# This is the recommend minimum for using introspection
port=0
bind-interfaces
enable-tftp
dhcp-sequential-ip

# These values do not have reasonable defaults
tftp-root=/tftpboot //tftp目录路径
interface=br-inspect //用于pxe启动BM的网桥名字，可查看上面的拓扑图
dhcp-range=192.168.0.50,192.168.0.150 //用于inspect阶段的dhcp地址
dhcp-boot=pxelinux.0

log-facility=/var/log/dnsmasq.log

配置完成后启动两个服务：

1
2
systemctl start xinetd
systemctl start openstack-ironic-inspector-dnsmasq

配置ironic中的inspector相关参数

ps:以下仅显示必要配置项

1
2
3
4
5
6
7
8
9
10
11
12
vim /etc/ironic/ironic.conf

[inspector]
enabled=true
service_url=http://{ironic-inspector-server-address}:5050
project_domain_id = default //使用keystonev3的默认domain，根据实际情况替换
user_domain_id = default //使用keystonev3的默认domain，根据实际情况替换
project_name = services
password = ironic
username = ironic-inspector
auth_type = password
auth_url=http://{keystone-address}:35357/v3

同步数据库

1
ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade

制作ipa镜像

使用diskimage-builder工具生成ironic-agent镜像，其中包含了ironic-python-agent，用于在inspect阶段与ironic-inspector协作

1
disk-image-create ironic-agent fedora -o ironic-agent

结果如下：

1
2
3
4
drwxr-xr-x  3 root   root          26 Mar 15 09:11 ironic-agent.d
-rw-r--r--  1 root   root   246951185 Mar 15 09:12 ironic-agent.initramfs
-rwxr-xr-x  2 root   root     6894536 Mar 15 09:12 ironic-agent.kernel
-rwxr-xr-x  2 root   root     6894536 Mar 15 09:12 ironic-agent.vmlinuz

上传镜像至glance

1
2
glance image-create --name deploy-vmlinuz --visibility public  --disk-format aki --container-format aki --file ./ironic-agent.vmlinuz
glance image-create --name deploy-initrd --visibility public --disk-format ari --container-format ari --file ./ironic-agent.initramfs

Inspect阶段

inspect阶段的前置步骤如下：

  1. 使用ironic创建node，选用agent系列的driver，如agent_ipmitool（用于部署服务器裸机）、agent_ssh（用于部署虚拟机裸机）
  2. 配置node的driver_info/deploy_ramdisk及driver_info/deploy_kernel属性，分别为上述deploy-initrd、deploy-vmlinuz的glance uuid
  3. 如上面的拓扑图所示，br-inspect作为inspect网络的网桥，连接了Baremetal Node的eth2网卡，所以配置ironic port，使用Baremetal Node的eth2网卡mac地址关联node uuid
  4. 将node设置为manage状态
    1
    ironic node-set-provision-state {node} manage

    开始inspect流程

    1
    ironic node-set-provision-state {node} inspect

以下[ironic-inspector-server]表示在inspector服务所在节点，inspector的流程；[pxe]表示BM的pxe阶段；[ironic-python-agent]表示BM系统部署完成并启动后，运行ironic-python-agent的流程

[ironic-inspector-server] ironic_inspector.main.py

1
2
3
4
5
6
7
# TODO(sambetts) Add API discovery for this endpoint
@app.route('/v1/introspection/<node_id>', methods=['GET', 'POST'])
@convert_exceptions
def api_introspection(node_id):
        introspect.introspect(node_id,
                              new_ipmi_credentials=new_ipmi_credentials,
                              token=flask.request.headers.get('X-Auth-Token'))

[ironic-inspector-server] ironic_inspector.introspect.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
def introspect(node_id, new_ipmi_credentials=None, token=None):

    # 使用ironicclient获取BM的mac地址
    bmc_address = ir_utils.get_ipmi_address(node)
    # 在inspector的数据库中cache目标node的信息
    node_info = node_cache.start_introspection(node.uuid,
                                               bmc_address=bmc_address,
                                               ironic=ironic)
    # 多线程执行后台任务，跳转到_background_introspect函数
    future = utils.executor().submit(_background_introspect, ironic, node_info)

def _background_introspect(ironic, node_info):
    # lock数据库中的node信息，转到_background_introspect_locked函数
    node_info.acquire_lock()
    try:
        _background_introspect_locked(node_info, ironic)
    finally:
        node_info.release_lock()

@node_cache.fsm_transition(istate.Events.wait)
def _background_introspect_locked(node_info, ironic):
    # TODO(dtantsur): pagination
    macs = list(node_info.ports())
    if macs:
        node_info.add_attribute(node_cache.MACS_ATTRIBUTE, macs)
        LOG.info(_LI('Whitelisting MAC\'s %s on the firewall'), macs,
                 node_info=node_info)
        # 更新ironic-inspector所在节点的firewall，将目标节点之外的mac地址加入黑名单
        firewall.update_filters(ironic)

        try:
            # 重启BM，准备进入pxe启动
            ironic.node.set_power_state(node_info.uuid, 'reboot')
        except Exception as exc:
            raise utils.Error(_('Failed to power on the node, check it\'s '
                                'power management configuration: %s'),
                              exc, node_info=node_info)

[pxe] BM进入pxe启动阶段， BM的eth2发出dhcp请求报文，ironic-inspector-dnsmasq通过br-inspect网桥回应dhcp请求并分发地址
[pxe] BM得到地址后，开始tftp请求，ironic-inspector-dnsmasq继续响应，将ironic-agent镜像传输至BM，BM开始启动ironic-agent镜像
[pxe] ironic-agent镜像启动后，会启动镜像中的ironic-python-agent服务，需要在其配置项中配置ironic-inspector的callback地址，ironic-python-agent才会执行inspect流程并将数据返回，该配置需要写在tftp服务的pxelinux配置文件中

[ironic-python-agent] ironic_python_agent.agent.py

1
2
3
4
5
6
7
8
class IronicPythonAgent(base.ExecuteCommandMixin):
    """Class for base agent functionality."""

    def run(self):

            if cfg.CONF.inspection_callback_url:
                # 因配置了callback_url，跳转到inspect
                uuid = inspector.inspect()

[ironic-python-agent] ironic_python_agent.inspector.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
def inspect():

    try:
        # 对应该文件中的collect_*函数，默认为default，对应collect_default
        ext_mgr = extension_manager(collector_names)
        collectors = [(ext.name, ext.plugin) for ext in ext_mgr]
    except Exception as exc:
        with excutils.save_and_reraise_exception():
            failures.add(exc)
            call_inspector(data, failures)

    for name, collector in collectors:
        try:
            # 例：此处为collect_default
            collector(data, failures)

    # 跳转到call_inspector函数，该函数用于发送data给inspector
    resp = call_inspector(data, failures)

def collect_default(data, failures):
    # 调用ironic_python_agent.hardware.py中的list_hardware_info函数
    inventory = hardware.dispatch_to_managers('list_hardware_info')
    # 添加到data中
    data['inventory'] = inventory

[ironic-python-agent] 可以看到除了collect_default，还提供了collect_logs、collect_extra_hardware、collect_pci_devices_info三个函数，分别用于收集系统日志、收集benchmark、收集pci设备信息
[ironic-python-agent] ironic_python_agent.hardware.py 可以看看collect_default收集了哪些信息

1
2
3
4
5
6
7
8
9
10
11
12
13
@six.add_metaclass(abc.ABCMeta)
class HardwareManager(object):

    def list_hardware_info(self):
        hardware_info = {}
        hardware_info['interfaces'] = self.list_network_interfaces()
        hardware_info['cpu'] = self.get_cpus()
        hardware_info['disks'] = self.list_block_devices()
        hardware_info['memory'] = self.get_memory()
        hardware_info['bmc_address'] = self.get_bmc_address()
        hardware_info['system_vendor'] = self.get_system_vendor_info()
        hardware_info['boot'] = self.get_boot_info()
        return hardware_info

[ironic-inspector-server] ipa收集BM信息并将其发送给ipa-inspection-callback-url
[ironic-inspector-server] ironic_inspector.main.py

1
2
3
4
5
6
@app.route('/v1/continue', methods=['POST'])
@convert_exceptions
def api_continue():
    data = flask.request.get_json(force=True)
    跳转到ironic_inspector.process.py的process函数
    return flask.jsonify(process.process(data))

[ironic-inspector-server] ironic_inspector.process.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def process(introspection_data):

    # 获取node信息并lock
    node_info = _find_node_info(introspection_data, failures)
    if node_info:
        node_info.acquire_lock()

    # 多线程处理_store_unprocessed_data函数，存储数据
    utils.executor().submit(_store_unprocessed_data, node_info,
                            unprocessed_data)

    try:
        # 从node_info中提取node
        node = node_info.node()

    try:
        # 跳转到_process_node函数
        result = _process_node(node_info, node, introspection_data)

@node_cache.fsm_transition(istate.Events.process, reentrant=False)
def _process_node(node_info, node, introspection_data):

    # 这些interfaces是从ipa收集到的BM信息中提取出来的，包括但不限于之前创建的用于pxe的ironic port
    interfaces = introspection_data.get('interfaces')
    # 调用ironicclient创建这些ports
    node_info.create_ports(list(interfaces.values()))
    # 存储数据，如ironic-inspector配置中store_data为none，则不存储
    _store_data(node_info, introspection_data)
    # 因为此时可能新创建了一些ports，要更新firewall的黑名单
    firewall.update_filters(ironic)

    # 根据ipa收集的数据，再根据ironic-inspector配置的rule，更新node的信息
    node_info.invalidate_cache()
    rules.apply(node_info, introspection_data)

    resp = {'uuid': node.uuid}

    else:
        # 结束inspect流程
        utils.executor().submit(_finish, node_info, ironic, introspection_data,
                                power_off=CONF.processing.power_off)

inspect阶段完成之后，ironic向用户提供该BM，此时node状态变为available

1
ironic node-set-provision-state {node} provide

参考