Skip to content

Commit

Permalink
add custom ipxe firmware building support
Browse files Browse the repository at this point in the history
This commit adds the following:

  - Support for detecting iPXE certificates
  - When iPXE certs are detected, the iPXE services
    and conigs will be advertised with https addresses
  - Enforce chainloading of the iPXE firmware provided by
    the user even if the client machine comes with an iPXE
    firmware by default
  - Adds modular iPXE apache2 config file to enable separate
    iPXE specific virtual host (port) where TLS is enforced
  - Adds ipxe builder script that can be run as an entry point
    to standalone container or as a K8s pod init container
  - iPXE build script provides options to enable IPV6 and TLS support
  - option to embed a script to the iPXE firmware is added but for
    now it is only used to break chainloading
  - custom iPXE config file template has been added thus Ironic can
    generate node specific iPXE config files that use the custom iPXE
    firmware
  - a set of environment variable that control the new iPXE related
    features
  - option to build ipxe from a cache specified as a file path or
    pulling remote iPXE repo and building that
  • Loading branch information
Rozzii committed Aug 8, 2023
1 parent eff31e4 commit db492a8
Show file tree
Hide file tree
Showing 14 changed files with 296 additions and 5 deletions.
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ COPY --from=ironic-builder /tmp/ipxe/src/bin/undionly.kpxe /tmp/ipxe/src/bin-x86
COPY --from=ironic-builder /tmp/esp.img /tmp/uefi_esp.img

COPY ironic-config/ironic.conf.j2 /etc/ironic/
COPY ironic-config/inspector.ipxe.j2 ironic-config/httpd-ironic-api.conf.j2 /tmp/
COPY ironic-config/inspector.ipxe.j2 ironic-config/httpd-ironic-api.conf.j2 ironic-config/ipxe_config.template /tmp/

# DNSMASQ
COPY ironic-config/dnsmasq.conf.j2 /etc/
Expand All @@ -55,6 +55,7 @@ COPY ironic-config/dnsmasq.conf.j2 /etc/
COPY ironic-config/httpd.conf.j2 /etc/httpd/conf/
COPY ironic-config/httpd-modules.conf /etc/httpd/conf.modules.d/
COPY ironic-config/apache2-vmedia.conf.j2 /etc/httpd-vmedia.conf.j2
COPY ironic-config/apache2-ipxe.conf.j2 /etc/httpd-ipxe.conf.j2

# IRONIC-INSPECTOR #
RUN mkdir -p /var/lib/ironic /var/lib/ironic-inspector && \
Expand Down
16 changes: 16 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,22 @@ functionality:
- `DNS_IP` - DNS IP address to use for ironic dnsmasq(dhcpd)
- `IRONIC_IPA_COLLECTORS` - Use a custom set of collectors to be run on
inspection. (default `default,logs`)
- `IPXE_BUILD_FROM_CACHE` - specifies whether the ipxebuilder should build
the firmware from a local directory or not. (default `true`)
- `IPXE_BUILD_FROM_REPO` specifies whether the ipxebuilder should build
from a remote git repo. (default `false`)
- `IPXE_SHARED_FIRMWARE_SOURCE` path to the iPXE source directory
(default `/shared/ipxe-source`)
- `IPXE_EMBED_SCRIPT` path to the iPXE script that will be embedded in the
custom iPXE firmware built by `the ipxebuilder`. (default `/bin/embed.ipxe`)
- `IPXE_EMBED_SCRIPT_TEMPLATE` path to the jinja template of the iPXE script
that will be embedded in the custom firmware built by the ipxebuilder. This
template is rendered in runtime by the builder script
(default `/bin/embed.ipxe.j2`)
- `IPXE_RELEASE_BRANCH` the iPXE source code should be pulled from this branch
, only relevant when `IPXE_BUILD_FROM_REPO` is `true` `(default `v1.21.1`)
- `IPXE_ENABLE_IPV6` build the iPXE firmware with IPV6 support enabled `false`
- `IPXE_ENABLE_HTTPS` build the iPXE firmware with TLS support enabled `false`

The ironic configuration can be overridden by various environment variables.
The following can serve as an example:
Expand Down
35 changes: 35 additions & 0 deletions ironic-config/apache2-ipxe.conf.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
Listen {{ env.IPXE_TLS_PORT }}

<VirtualHost *:{{ env.IPXE_TLS_PORT }}>
ErrorLog /dev/stderr
LogLevel debug
CustomLog /dev/stdout combined

SSLEngine on
SSLProtocol {{ env.IPXE_SSL_PROTOCOL }}
SSLCertificateFile {{ env.IPXE_CERT_FILE }}
SSLCertificateKeyFile {{ env.IPXE_KEY_FILE }}

<Directory "/shared/html">
Order Allow,Deny
Allow from all
</Directory>
<Directory "/shared/html/(redfish|ilo|images)/">
Order Deny,Allow
Deny from all
</Directory>
</VirtualHost>

<Location ~ "^/grub.*/">
SSLRequireSSL
</Location>
<Location ~ "^/pxelinux.cfg/">
SSLRequireSSL
</Location>
<Location ~ "^/.*\.conf/">
SSLRequireSSL
</Location>
<Location ~ "^/(([0-9]|[a-z]).*-){4}([0-9]|[a-z]).*/">
SSLRequireSSL
</Location>

12 changes: 11 additions & 1 deletion ironic-config/dnsmasq.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -29,13 +29,23 @@ dhcp-option=option{% if ":" in env["DNS_IP"] %}6{% endif %}:dns-server,{{ env["D
# IPv4 Configuration:
dhcp-match=ipxe,175
# Client is already running iPXE; move to next stage of chainloading
{%- if env.IPXE_TLS_SETUP == "true" %}
# iPXE with (U)EFI
dhcp-boot=tag:efi,tag:ipxe,http://{{ env.IRONIC_URL_HOST }}:{{ env.HTTP_PORT }}/custom-ipxe/snponly.efi
# iPXE with BIOS
dhcp-boot=tag:ipxe,http://{{ env.IRONIC_URL_HOST }}:{{ env.HTTP_PORT }}/custom-ipxe/undionly.kpxe
{% else %}
dhcp-boot=tag:ipxe,http://{{ env.IRONIC_URL_HOST }}:{{ env.HTTP_PORT }}/boot.ipxe
{% endif %}

# Note: Need to test EFI booting
dhcp-match=set:efi,option:client-arch,7
dhcp-match=set:efi,option:client-arch,9
dhcp-match=set:efi,option:client-arch,11
# Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader
# Client is PXE booting over EFI without iPXE ROM; send EFI version of iPXE chainloader do the same also if iPXE ROM boots but TLS is enabled
{%- if env.IPXE_TLS_SETUP == "true" %}
dhcp-boot=tag:efi,tag:ipxe,snponly.efi
{% endif %}
dhcp-boot=tag:efi,tag:!ipxe,snponly.efi

# Client is running PXE over BIOS; send BIOS version of iPXE chainloader
Expand Down
2 changes: 1 addition & 1 deletion ironic-config/httpd.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ DocumentRoot "/shared/html"
Require all granted
</Directory>

{%- if env.HTTPD_SERVE_NODE_IMAGES %}
{%- if env.HTTPD_SERVE_NODE_IMAGES == "true" %}
<Directory "/shared/html/images">
Options Indexes FollowSymLinks
AllowOverride None
Expand Down
81 changes: 81 additions & 0 deletions ironic-config/ipxe_config.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
#!ipxe

set attempts:int32 10
set i:int32 0

goto deploy

:deploy
imgfree
{%- if pxe_options.deployment_aki_path %}
{%- set aki_path_https_elements = pxe_options.deployment_aki_path.split(':') %}
{%- set aki_port_and_path = aki_path_https_elements[2].split('/') %}
{%- set aki_afterport = aki_port_and_path[1:]|join('/') %}
{%- set aki_path_https = ['https:', aki_path_https_elements[1], ':8084/', aki_afterport]|join %}
{%- endif %}
{%- if pxe_options.deployment_ari_path %}
{%- set ari_path_https_elements = pxe_options.deployment_ari_path.split(':') %}
{%- set ari_port_and_path = ari_path_https_elements[2].split('/') %}
{%- set ari_afterport = ari_port_and_path[1:]|join('/') %}
{%- set ari_path_https = ['https:', ari_path_https_elements[1], ':8084/', ari_afterport]|join %}
{%- endif %}
kernel {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ aki_path_https }} selinux=0 troubleshoot=0 text {{ pxe_options.pxe_append_params|default("", true) }} BOOTIF=${mac} initrd={{ pxe_options.initrd_filename|default("deploy_ramdisk", true) }} || goto retry

initrd {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ ari_path_https }} || goto retry
boot

:retry
iseq ${i} ${attempts} && goto fail ||
inc i
echo No response, retrying in ${i} seconds.
sleep ${i}
goto deploy

:fail
echo Failed to get a response after ${attempts} attempts
echo Powering off in 30 seconds.
sleep 30
poweroff

:boot_anaconda
imgfree
kernel {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ aki_path_https }} text {{ pxe_options.pxe_append_params|default("", true) }} inst.ks={{ pxe_options.ks_cfg_url }} {% if pxe_options.repo_url %}inst.repo={{ pxe_options.repo_url }}{% else %}inst.stage2={{ pxe_options.stage2_url }}{% endif %} initrd=ramdisk || goto boot_anaconda
initrd {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ ari_path_https }} || goto boot_anaconda
boot

:boot_ramdisk
imgfree
{%- if pxe_options.boot_iso_url %}
sanboot {{ pxe_options.boot_iso_url }}
{%- else %}
kernel {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ aki_path_https }} root=/dev/ram0 text {{ pxe_options.pxe_append_params|default("", true) }} {{ pxe_options.ramdisk_opts|default('', true) }} initrd=ramdisk || goto boot_ramdisk
initrd {% if pxe_options.ipxe_timeout > 0 %}--timeout {{ pxe_options.ipxe_timeout }} {% endif %}{{ ari_path_https }} || goto boot_ramdisk
boot
{%- endif %}

{%- if pxe_options.boot_from_volume %}

:boot_iscsi
imgfree
{% if pxe_options.username %}set username {{ pxe_options.username }}{% endif %}
{% if pxe_options.password %}set password {{ pxe_options.password }}{% endif %}
{% if pxe_options.iscsi_initiator_iqn %}set initiator-iqn {{ pxe_options.iscsi_initiator_iqn }}{% endif %}
sanhook --drive 0x80 {{ pxe_options.iscsi_boot_url }} || goto fail_iscsi_retry
{%- if pxe_options.iscsi_volumes %}{% for i, volume in enumerate(pxe_options.iscsi_volumes) %}
set username {{ volume.username }}
set password {{ volume.password }}
{%- set drive_id = 129 + i %}
sanhook --drive {{ '0x%x' % drive_id }} {{ volume.url }} || goto fail_iscsi_retry
{%- endfor %}{% endif %}
{% if pxe_options.iscsi_volumes %}set username {{ pxe_options.username }}{% endif %}
{% if pxe_options.iscsi_volumes %}set password {{ pxe_options.password }}{% endif %}
sanboot --no-describe || goto fail_iscsi_retry

:fail_iscsi_retry
echo Failed to attach iSCSI volume(s), retrying in 10 seconds.
sleep 10
goto boot_iscsi
{%- endif %}

:boot_whole_disk
sanboot --no-describe || exit 0
3 changes: 3 additions & 0 deletions ironic-config/ironic.conf.j2
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ ramdisk_image_download_source = {{ env.IRONIC_BOOT_ISO_SOURCE }}
{% endif %}
{% if env.IRONIC_EXTERNAL_HTTP_URL %}
external_http_url = {{ env.IRONIC_EXTERNAL_HTTP_URL }}
{% elif env.IPXE_TLS_SETUP == "true" %}
external_http_url = https://{{ env.IRONIC_URL_HOST }}:{{ env.IPXE_TLS_PORT }}
{% elif env.IRONIC_VMEDIA_TLS_SETUP == "true" %}
external_http_url = https://{{ env.IRONIC_URL_HOST }}:{{ env.VMEDIA_TLS_PORT }}
{% endif %}
Expand Down Expand Up @@ -204,6 +206,7 @@ kernel_append_params = nofb nomodeset vga=normal ipa-insecure=1 {% if env.IRONIC
enable_netboot_fallback = true
# Enable the fallback path to ironic-inspector
ipxe_fallback_script = inspector.ipxe
ipxe_config_template = /tmp/ipxe_config.template

[redfish]
use_swift = false
Expand Down
4 changes: 2 additions & 2 deletions prepare-image.sh
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ echo "install_weak_deps=False" >> /etc/dnf/dnf.conf
# Tell RPM to skip installing documentation
echo "tsflags=nodocs" >> /etc/dnf/dnf.conf

dnf install -y python3 python3-requests 'dnf-command(config-manager)'
dnf install -y python3 python3-requests 'dnf-command(config-manager)' gcc make git-core perl xz-devel

# NOTE(elfosardo): building the container using ironic RPMs is
# now deprecated and it will be removed in the future.
Expand All @@ -28,7 +28,7 @@ if [[ "$INSTALL_TYPE" == "source" ]]; then
IRONIC_GID=994
INSPECTOR_UID=996
INSPECTOR_GID=993
BUILD_DEPS="python3-devel gcc git-core python3-setuptools python3-jinja2"
BUILD_DEPS="python3-devel python3-setuptools python3-jinja2"
dnf upgrade -y
# NOTE(dtantsur): pip is a requirement of python3 in CentOS
# shellcheck disable=SC2086
Expand Down
85 changes: 85 additions & 0 deletions scripts/buildipxe
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
#!/usr/bin/bash

set -euxo pipefail
# ENV VARIABLE CONFIG #
# shellcheck disable=SC1091
. /bin/tls-common.sh
. /bin/ironic-common.sh

# This script should be used as an init container's entry point thus
# it is expected that the user would like to build the ipxe when this script
# is started. By default the script will try to build from cache and fails
# if the cache is unavailable.
export IPXE_BUILD_FROM_CACHE="${IPXE_BUILD_FROM_CACHE:-true}"
export IPXE_BUILD_FROM_REPO="${IPXE_BUILD_FROM_REPO:-false}"
export IPXE_SHARED_FIRMWARE_SOURCE="${IPXE_SHARED_FIRMWARE_SOURCE:-/shared/ipxe-source}"
export IPXE_EMBED_SCRIPT="${IPXE_EMBED_SCRIPT:-/bin/embed.ipxe}"
export IPXE_EMBED_SCRIPT_TEMPLATE="${IPXE_EMBED_SCRIPT_TEMPLATE:-/bin/embed.ipxe.j2}"
export IPXE_RELEASE_BRANCH="${IPXE_RELEASE_BRANCH:-v1.21.1}"
export IPXE_ENABLE_IPV6="${IPXE_ENABLE_IPV6:-false}"
export IPXE_ENABLE_HTTPS="${IPXE_ENABLE_TLS:-false}"
# BUILD_OPTIONS are not configurable directly
export IPXE_BUILD_OPTIONS="NO_WERROR=1 EMBED=${IPXE_EMBED_SCRIPT}"
# PREPARE SOURCE #

# Create debug folder
mkdir -p "/shared/ipxe-debug"

# In case building ipxe firmware from shared volume
if [[ "${IPXE_BUILD_FROM_CACHE}" == "true" ]]; then
if [[ -r "${IPXE_SHARED_FIRMWARE_SOURCE}" ]]; then
cp -r "${IPXE_SHARED_FIRMWARE_SOURCE}" "/tmp"
ls -all "/tmp/ipxe-source"
else
>&2 echo "ERROR: can't build ipxe from cache, there is no path!"
exit 1
fi
fi

# In case building ipxe firmware from upstream git repo directly
# Requires Internet access!
if [[ "${IPXE_BUILD_FROM_CACHE}" == "false" ]] \
&& [[ "${IPXE_BUILD_FROM_REPO}" == "true" ]]; then
git clone --depth 1 --branch "${IPXE_RELEASE_BRANCH}" \
"https://github.com/ipxe/ipxe.git" \
"/tmp/ipxe-source"
fi

if [[ ! -r "/tmp/ipxe-source" ]]; then
>&2 echo "ERROR: The ipxe firmware source is missing, check the env vars!"
exit 1
fi

# BUILD #
ARCH=$(uname -m | sed 's/aarch/arm/')
# NOTE(elfosardo): warning should not be treated as errors by default
cd "/tmp/ipxe-source/src"
if [[ "${IPXE_ENABLE_IPV6}" == "true" ]]; then
sed -i 's/^\/\/#define[ \t]NET_PROTO_IPV6/#define\tNET_PROTO_IPV6/g' \
"config/general.h"
fi
if [[ "${IPXE_ENABLE_TLS}" == "true" ]]; then
if [[ ! -r "${IPXE_CERT_FILE}" ]]; then
>&2 echo "ERROR: iPXE TLS support is enabled but cert is missing!"
exit 1
fi
sed -i 's/^#define[ \t]DOWNLOAD_PROTO_HTTP/#undef\tDOWNLOAD_PROTO_HTTP/g' \
"config/general.h"
sed -i 's/^#undef[ \t]DOWNLOAD_PROTO_HTTPS/#define\tDOWNLOAD_PROTO_HTTPS/g' \
"config/general.h"
echo "IPXE BUILD OPTIONS ARE EXTENDED WITH CERTS!!!"
render_j2_config "${IPXE_EMBED_SCRIPT_TEMPLATE}" "${IPXE_EMBED_SCRIPT}"
export IPXE_BUILD_OPTIONS="${IPXE_BUILD_OPTIONS} CERT=${IPXE_CERT_FILE} TRUST=${IPXE_CERT_FILE}"
fi

sed -i 's/^\/\/#define[ \t]CONSOLE_SERIAL/#define\tCONSOLE_SERIAL/g' \
"config/console.h"

/usr/bin/make "bin/undionly.kpxe" "bin-${ARCH}-efi/snponly.efi" ${IPXE_BUILD_OPTIONS[@]}

mkdir -p "${IPXE_CUSTOM_FIRMWARE_DIR}"
# These files will be copied by the rundnsmasq script to the shared volume.
cp "/tmp/ipxe-source/src/bin/undionly.kpxe" \
"/tmp/ipxe-source/src/bin-${ARCH}-efi/snponly.efi" \
"${IPXE_CUSTOM_FIRMWARE_DIR}"

14 changes: 14 additions & 0 deletions scripts/embed.ipxe.j2
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
#!ipxe

:chainload
echo Initiating DHCP based interface configuration
dhcp
echo Start chainloading https://{{ env.IRONIC_IP }}:{{ env.IPXE_TLS_PORT }}/boot.ipxe
chain https://{{ env.IRONIC_IP }}:{{ env.IPXE_TLS_PORT }}/boot.ipxe || goto boot_failed
echo Chainloading succeeded!

:boot_failed
echo Chainloading failed!
echo Press any key to reboot...
prompt --timeout 60
reboot
1 change: 1 addition & 0 deletions scripts/ironic-common.sh
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ IRONIC_IP="${IRONIC_IP:-}"
PROVISIONING_INTERFACE="${PROVISIONING_INTERFACE:-}"
PROVISIONING_IP="${PROVISIONING_IP:-}"
PROVISIONING_MACS="${PROVISIONING_MACS:-}"
IPXE_CUSTOM_FIRMWARE_DIR="${IPXE_CUSTOM_FIRMWARE_DIR:-/shared/custom_ipxe_firmware}"

get_provisioning_interface()
{
Expand Down
8 changes: 8 additions & 0 deletions scripts/rundnsmasq
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@ set -eux

# shellcheck disable=SC1091
. /bin/ironic-common.sh
# shellcheck disable=SC1091
. /bin/tls-common.sh

export HTTP_PORT=${HTTP_PORT:-80}
DNSMASQ_EXCEPT_INTERFACE=${DNSMASQ_EXCEPT_INTERFACE:-lo}
Expand All @@ -19,7 +21,13 @@ mkdir -p /shared/html/images
mkdir -p /shared/html/pxelinux.cfg

# Copy files to shared mount
if [[ -r "${IPXE_CUSTOM_FIRMWARE_DIR}" ]]; then
cp "${IPXE_CUSTOM_FIRMWARE_DIR}/undionly.kpxe" \
"${IPXE_CUSTOM_FIRMWARE_DIR}/snponly.efi" \
"/shared/tftpboot"
else
cp /tftpboot/undionly.kpxe /tftpboot/snponly.efi /shared/tftpboot
fi

# Template and write dnsmasq.conf
# we template via /tmp as sed otherwise creates temp files in /etc directory
Expand Down
12 changes: 12 additions & 0 deletions scripts/runhttpd
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,9 @@ wait_for_interface_or_ip
mkdir -p /shared/html
chmod 0777 /shared/html

mkdir -p /shared/html/custom-ipxe
chmod 0777 /shared/html/custom-ipxe

IRONIC_BASE_URL="${IRONIC_SCHEME}://${IRONIC_URL_HOST}"

INSPECTOR_EXTRA_ARGS=" ipa-inspection-callback-url=${IRONIC_BASE_URL}:${IRONIC_INSPECTOR_ACCESS_PORT}/v1/continue"
Expand Down Expand Up @@ -76,6 +79,15 @@ if [[ "$IRONIC_VMEDIA_TLS_SETUP" == "true" ]]; then
render_j2_config /etc/httpd-vmedia.conf.j2 /etc/httpd/conf.d/vmedia.conf
fi

# Render httpd TLS configuration for /shared/html
if [[ "$IPXE_TLS_SETUP" == "true" ]]; then
render_j2_config "/etc/httpd-ipxe.conf.j2" "/etc/httpd/conf.d/ipxe.conf"
render_j2_config "/bin/embed.ipxe.j2" "/bin/embed.ipxe"
cp "${IPXE_CUSTOM_FIRMWARE_DIR}/undionly.kpxe" \
"${IPXE_CUSTOM_FIRMWARE_DIR}/snponly.efi" \
"/shared/html/custom-ipxe"
fi

# Set up inotify to kill the container (restart) whenever cert files for ironic inspector change
if [[ "$IRONIC_INSPECTOR_TLS_SETUP" == "true" ]] && [[ "${RESTART_CONTAINER_CERTIFICATE_UPDATED}" == "true" ]]; then
# shellcheck disable=SC2034
Expand Down
Loading

0 comments on commit db492a8

Please sign in to comment.