Ironic Inspector安装、使用及流程分析

Hardware introspection for OpenStack Bare Metal

This is an auxiliary service for discovering hardware properties for a node managed by Ironic. Hardware introspection or hardware properties discovery is a process of getting hardware parameters required for scheduling from a bare metal node, given it’s power management credentials (e.g. IPMI address, user name and password).

安装与配置ironic-inspector

ironic-inspector的流程说明:

ironic-inspector使用ironic-agent镜像,以及自身提供的一个dnsmasq服务预先布置BM,主要的工作流程为:

  1. 通过ironic api进入inspect阶段(也可以使用ironic-inspector的api)
  2. ironic启动BM(如BM已启动,则关闭再启动),BM进入pxe启动阶段;通过ironic-inspector-dnsmasq分配ip地址,成功后使用tftp将ironic-agent镜像传输至BM;BM从ironic-agent镜像启动
  3. ironic-python-agent开始工作,其与ironic-inspector取得通信,根据inspector中的rule,对BM进行inspect动作,主要目的为获取BM的硬件信息;收集完毕后ipa将数据传输给ironic-inspector;关闭BM
  4. ironic-inspector根据配置中store_data选择的driver来存储收集到的数据;inspect阶段完成

安装软件包

添加rdo的ocata源:

1
yum install https://repos.fedorapeople.org/repos/openstack/openstack-ocata/rdo-release-ocata-2.noarch.rpm -y

下载ironic-inspector及其client的rpm包:

1
yum install openstack-ironic-inspector python-ironic-inspector-client -y

(可用安装源码代替,但是需要手动配置,源码从github上openstack对应模块的stable/ocata branch下载)

创建ironic-inspector的认证信息及endpoint

1
2
3
4
5
6
openstack user create --domain default --project services --project-domain default --password ironic --enable ironic-inspector
openstack service create --name ironic-inspector --description 'Bare Metal Introspection Service' --enable baremetal-introspection
openstack role add --user ironic-inspector --project services --project-domain default --user-domain default admin
openstack endpoint create --region RegionOne --enable ironic-inspector admin http://{ironic-inspector-server-address}:5050
openstack endpoint create --region RegionOne --enable ironic-inspector internal http://{ironic-inspector-server-address}:5050
openstack endpoint create --region RegionOne --enable ironic-inspector public http://{ironic-inspector-server-address}:5050

创建ironic-inspector的数据库

1
2
3
4
mysql -e "create database ironic_inspector;"
mysql -e "grant all on ironic_inspector.* to ironic_inspector@'localhost' identified by 'ironic_inspector';"
mysql -e "grant all on ironic_inspector.* to ironic_inspector@'%' identified by 'ironic_inspector';"
mysql -e "flush privileges;"
  • ironic-inspector包含有两个服务,一个是ironic-inspector服务,用于和ipa协作完成inspect流程任务;另一个为ironic-inspector-dnsmasq服务,用于在inspect阶段承担dhcp、tftp功能。

配置ironic-inspector服务

ps:以下仅显示必要配置项

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
vim /etc/ironic-inspector/inspector.conf

[DEFAULT]
listen_address = 0.0.0.0
listen_port = 5050
auth_strategy = keystone
debug = false
verbose = true

[database]
connection = mysql://ironic_inspector:ironic_inspector@{ironic-inspector-db-address}/ironic_inspector?charset=utf8

[firewall]
dnsmasq_interface = br-inspect //inspector-dnsmasq使用的网桥,用于发送dhcp、tftp的报文

[ironic]
auth_url = http://{keystone-address}:5000/v3
auth_strategy = keystone
auth_type = password
default_domain_name = default //使用keystonev3的默认domain,根据实际情况替换
ironic_url = http://{ironic-server-address}:6385/v1
os_region = RegionOne
password = {ironic-password}
project_domain_name = default //使用keystonev3的默认domain,根据实际情况替换
username = {ironic-username}

[keystone_authtoken]
project_name = services
password = ironic
username = ironic-inspector
auth_url = http://{keystone-address}:35357/v2.0
auth_type = password
region_name = RegionOne

[processing]
add_ports = all //用于发现bm的网卡,all表示添加所有bm网卡,pxe表示只添加pxe启动的那块网卡
keep_ports = all //用于决定保留哪些网卡,all表示保留所有添加的网卡,present表示只保留当前使用的网卡
ramdisk_logs_dir = /var/log/ironic-inspector/ramdisk
store_data = none //用于存储inspect过程收集出的数据的driver,可以使用swift,none表示不存储

安装及配置tftp、ironic-inspector-dnsmasq服务

安装tftp服务

1
yum install tftp-server -y

配置tftp与dhcp服务

ps:以下仅显示必要配置项
tftp通过xinetd来守护进程,其配置如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
vim /etc/xinetd.d/tftp

service tftp
{
socket_type = dgram
protocol = udp
port = 69
wait = yes
user = root
server = /usr/sbin/in.tftpd
server_args = -v -v -v -v -v --map-file /tftpboot/map-file /tftpboot
#server_args = -s /var/lib/tftpboot
disable = no
per_source = 11
cps = 100 2
flags = IPv4
}

tftp目录结构:

1
2
3
4
5
6
7
8
9
[root@ironic ~(keystone_admin_46)]# tree /tftpboot/
/tftpboot/
├── chain.c32 // 拷贝/usr/share/syslinux/chain.c32到此处
├── ironic-agent.initramfs // 拷贝diskimage-builder生成的ironic-agent.initramfs到此处
├── ironic-agent.kernel // 拷贝diskimage-builder生成的ironic-agent.kernel到此处
├── map-file
├── pxelinux.0 // 拷贝/usr/share/syslinux/pxelinux.0到此处
└── pxelinux.cfg
└── default

map-file内容:

1
2
3
4
5
[root@ironic tftpboot(keystone_admin)]# cat map-file
re ^(/tftpboot/) /tftpboot/\2
re ^/tftpboot/ /tftpboot/
re ^(^/) /tftpboot/\1
re ^([^/]) /tftpboot/\1

pxelinux.cfg/default内容如下,如果需要收集lldp信息,需要加上ipa-collect-lldp=True:

1
2
3
4
5
6
7
8
[root@ironic tftpboot(keystone_admin)]# cat pxelinux.cfg/default
default introspect

label introspect
kernel ironic-agent.kernel
append initrd=ironic-agent.initramfs ipa-inspection-callback-url=http://{ironic-inspector-server-address}:5050/v1/continue systemd.journald.forward_to_console=yes ipa-collect-lldp=True

ipappend 3

ironic-inspector-dnsmasq的配置文件如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
/etc/ironic-inspector/dnsmasq.conf

# This is the recommend minimum for using introspection
port=0
bind-interfaces
enable-tftp
dhcp-sequential-ip

# These values do not have reasonable defaults
tftp-root=/tftpboot //tftp目录路径
interface=br-inspect //用于pxe启动BM的网桥名字,可查看上面的拓扑图
dhcp-range=192.168.0.50,192.168.0.150 //用于inspect阶段的dhcp地址
dhcp-boot=pxelinux.0

log-facility=/var/log/dnsmasq.log

配置完成后启动两个服务:

1
2
systemctl start xinetd
systemctl start openstack-ironic-inspector-dnsmasq

配置ironic中的inspector相关参数

ps:以下仅显示必要配置项

1
2
3
4
5
6
7
8
9
10
11
12
vim /etc/ironic/ironic.conf

[inspector]
enabled=true
service_url=http://{ironic-inspector-server-address}:5050
project_domain_id = default //使用keystonev3的默认domain,根据实际情况替换
user_domain_id = default //使用keystonev3的默认domain,根据实际情况替换
project_name = services
password = ironic
username = ironic-inspector
auth_type = password
auth_url=http://{keystone-address}:35357/v3

同步数据库

1
ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade

制作ipa镜像

使用diskimage-builder工具生成ironic-agent镜像,其中包含了ironic-python-agent,用于在inspect阶段与ironic-inspector协作

1
disk-image-create ironic-agent fedora -o ironic-agent

结果如下:

1
2
3
4
drwxr-xr-x  3 root   root          26 Mar 15 09:11 ironic-agent.d
-rw-r--r-- 1 root root 246951185 Mar 15 09:12 ironic-agent.initramfs
-rwxr-xr-x 2 root root 6894536 Mar 15 09:12 ironic-agent.kernel
-rwxr-xr-x 2 root root 6894536 Mar 15 09:12 ironic-agent.vmlinuz

上传镜像至glance

1
2
glance image-create --name deploy-vmlinuz --visibility public  --disk-format aki --container-format aki --file ./ironic-agent.vmlinuz
glance image-create --name deploy-initrd --visibility public --disk-format ari --container-format ari --file ./ironic-agent.initramfs

Inspect阶段

inspect阶段的前置步骤如下:

  1. 使用ironic创建node,选用agent系列的driver,如agent_ipmitool(用于部署服务器裸机)、agent_ssh(用于部署虚拟机裸机)
  2. 配置node的driver_info/deploy_ramdisk及driver_info/deploy_kernel属性,分别为上述deploy-initrd、deploy-vmlinuz的glance uuid
  3. 如上面的拓扑图所示,br-inspect作为inspect网络的网桥,连接了Baremetal Node的eth2网卡,所以配置ironic port,使用Baremetal Node的eth2网卡mac地址关联node uuid
  4. 将node设置为manage状态
    1
    ironic node-set-provision-state {node} manage

    开始inspect流程

    1
    ironic node-set-provision-state {node} inspect

以下[ironic-inspector-server]表示在inspector服务所在节点,inspector的流程;[pxe]表示BM的pxe阶段;[ironic-python-agent]表示BM系统部署完成并启动后,运行ironic-python-agent的流程

[ironic-inspector-server] ironic_inspector.main.py

1
2
3
4
5
6
7
# TODO(sambetts) Add API discovery for this endpoint
@app.route('/v1/introspection/<node_id>', methods=['GET', 'POST'])
@convert_exceptions
def api_introspection(node_id):
introspect.introspect(node_id,
new_ipmi_credentials=new_ipmi_credentials,
token=flask.request.headers.get('X-Auth-Token'))

[ironic-inspector-server] ironic_inspector.introspect.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
def introspect(node_id, new_ipmi_credentials=None, token=None):

# 使用ironicclient获取BM的mac地址
bmc_address = ir_utils.get_ipmi_address(node)
# 在inspector的数据库中cache目标node的信息
node_info = node_cache.start_introspection(node.uuid,
bmc_address=bmc_address,
ironic=ironic)
# 多线程执行后台任务,跳转到_background_introspect函数
future = utils.executor().submit(_background_introspect, ironic, node_info)

def _background_introspect(ironic, node_info):
# lock数据库中的node信息,转到_background_introspect_locked函数
node_info.acquire_lock()
try:
_background_introspect_locked(node_info, ironic)
finally:
node_info.release_lock()

@node_cache.fsm_transition(istate.Events.wait)
def _background_introspect_locked(node_info, ironic):
# TODO(dtantsur): pagination
macs = list(node_info.ports())
if macs:
node_info.add_attribute(node_cache.MACS_ATTRIBUTE, macs)
LOG.info(_LI('Whitelisting MAC\'s %s on the firewall'), macs,
node_info=node_info)
# 更新ironic-inspector所在节点的firewall,将目标节点之外的mac地址加入黑名单
firewall.update_filters(ironic)

try:
# 重启BM,准备进入pxe启动
ironic.node.set_power_state(node_info.uuid, 'reboot')
except Exception as exc:
raise utils.Error(_('Failed to power on the node, check it\'s '
'power management configuration: %s'),
exc, node_info=node_info)

[pxe] BM进入pxe启动阶段, BM的eth2发出dhcp请求报文,ironic-inspector-dnsmasq通过br-inspect网桥回应dhcp请求并分发地址
[pxe] BM得到地址后,开始tftp请求,ironic-inspector-dnsmasq继续响应,将ironic-agent镜像传输至BM,BM开始启动ironic-agent镜像
[pxe] ironic-agent镜像启动后,会启动镜像中的ironic-python-agent服务,需要在其配置项中配置ironic-inspector的callback地址,ironic-python-agent才会执行inspect流程并将数据返回,该配置需要写在tftp服务的pxelinux配置文件中

[ironic-python-agent] ironic_python_agent.agent.py

1
2
3
4
5
6
7
8
class IronicPythonAgent(base.ExecuteCommandMixin):
"""Class for base agent functionality."""

def run(self):

if cfg.CONF.inspection_callback_url:
# 因配置了callback_url,跳转到inspect
uuid = inspector.inspect()

[ironic-python-agent] ironic_python_agent.inspector.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
def inspect():

try:
# 对应该文件中的collect_*函数,默认为default,对应collect_default
ext_mgr = extension_manager(collector_names)
collectors = [(ext.name, ext.plugin) for ext in ext_mgr]
except Exception as exc:
with excutils.save_and_reraise_exception():
failures.add(exc)
call_inspector(data, failures)

for name, collector in collectors:
try:
# 例:此处为collect_default
collector(data, failures)

# 跳转到call_inspector函数,该函数用于发送data给inspector
resp = call_inspector(data, failures)

def collect_default(data, failures):
# 调用ironic_python_agent.hardware.py中的list_hardware_info函数
inventory = hardware.dispatch_to_managers('list_hardware_info')
# 添加到data中
data['inventory'] = inventory

[ironic-python-agent] 可以看到除了collect_default,还提供了collect_logs、collect_extra_hardware、collect_pci_devices_info三个函数,分别用于收集系统日志、收集benchmark、收集pci设备信息
[ironic-python-agent] ironic_python_agent.hardware.py 可以看看collect_default收集了哪些信息

1
2
3
4
5
6
7
8
9
10
11
12
13
@six.add_metaclass(abc.ABCMeta)
class HardwareManager(object):

def list_hardware_info(self):
hardware_info = {}
hardware_info['interfaces'] = self.list_network_interfaces()
hardware_info['cpu'] = self.get_cpus()
hardware_info['disks'] = self.list_block_devices()
hardware_info['memory'] = self.get_memory()
hardware_info['bmc_address'] = self.get_bmc_address()
hardware_info['system_vendor'] = self.get_system_vendor_info()
hardware_info['boot'] = self.get_boot_info()
return hardware_info

[ironic-inspector-server] ipa收集BM信息并将其发送给ipa-inspection-callback-url
[ironic-inspector-server] ironic_inspector.main.py

1
2
3
4
5
6
@app.route('/v1/continue', methods=['POST'])
@convert_exceptions
def api_continue():
data = flask.request.get_json(force=True)
跳转到ironic_inspector.process.py的process函数
return flask.jsonify(process.process(data))

[ironic-inspector-server] ironic_inspector.process.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
def process(introspection_data):

# 获取node信息并lock
node_info = _find_node_info(introspection_data, failures)
if node_info:
node_info.acquire_lock()

# 多线程处理_store_unprocessed_data函数,存储数据
utils.executor().submit(_store_unprocessed_data, node_info,
unprocessed_data)

try:
# 从node_info中提取node
node = node_info.node()

try:
# 跳转到_process_node函数
result = _process_node(node_info, node, introspection_data)

@node_cache.fsm_transition(istate.Events.process, reentrant=False)
def _process_node(node_info, node, introspection_data):

# 这些interfaces是从ipa收集到的BM信息中提取出来的,包括但不限于之前创建的用于pxe的ironic port
interfaces = introspection_data.get('interfaces')
# 调用ironicclient创建这些ports
node_info.create_ports(list(interfaces.values()))
# 存储数据,如ironic-inspector配置中store_data为none,则不存储
_store_data(node_info, introspection_data)
# 因为此时可能新创建了一些ports,要更新firewall的黑名单
firewall.update_filters(ironic)

# 根据ipa收集的数据,再根据ironic-inspector配置的rule,更新node的信息
node_info.invalidate_cache()
rules.apply(node_info, introspection_data)

resp = {'uuid': node.uuid}

else:
# 结束inspect流程
utils.executor().submit(_finish, node_info, ironic, introspection_data,
power_off=CONF.processing.power_off)

inspect阶段完成之后,ironic向用户提供该BM,此时node状态变为available

1
ironic node-set-provision-state {node} provide

参考