Daniel Hilst
ff20388fed
A compute node fetches the legacy Genesis kernel and initramfs, the kernel starts, and then nothing else happens: doxcat never runs, the node acquires no address, sshd refuses every connection and the node stays at status=powering-on. The five genesis test cases in xCAT-test/autotest/testcase/genesis have never passed. VersatusHPC/xcat-internal#78. Three holes in the image, each fatal on its own. dracut_105/el/xcat-cmdline.sh ends in `while :; do tmux attach-session -t doxcat || tmux new-session -s doxcat doxcat; done`, and the image carries no locale data, so tmux exits with "need UTF-8 locale" and the loop spins without ever reaching doxcat. module-setup.sh does not install /usr/libexec/openssh/sshd-session, which OpenSSH 9.8 and later exec for every connection and which EL9 now ships. xCAT-genesis-base.spec does not BuildRequire dhcp-client, so dhclient is absent from the build chroot; dracut_install reports the missing binary and returns, and the module install function keeps going, so the image ships without it. xcat-cmdline.sh now resolves xcat_console_mode() once and runs doxcat directly when the terminal multiplexer cannot start a session; the same shape replaces the screen loop on Ubuntu. module-setup.sh installs the OpenSSH session helpers and the C.utf8 locale where they exist. The spec BuildRequires dhcp-client on the releases that package it, and runs the new xCAT-genesis-builder/verify-genesis-payload over the extracted payload, which fails the build when sshd needs a helper the image lacks, when tmux has no UTF-8 locale, or when a binary the caller named is missing. The same runs exposed four defects in the test cases themselves. test.sh defined its synthetic node as ppc64le whatever the management node was, so nodeset could not find a genesis kernel on x86_64. genesistest.pl get_os() matched neither AlmaLinux nor Rocky and reported the OS as unsupported. The -g check read $? instead of check_genesis_file()'s return value, so it could never fail. And testxdsh() met "REMOTE HOST IDENTIFICATION HAS CHANGED" from the second boot on, because Genesis makes new host keys every boot and nothing dropped the stale known_hosts entry. test.sh now derives the node arch from uname and takes the tftp root from TFTPDIR, get_os() recognises the redhat family, report_genesis_files() carries the result to an exit status, and forget_host_keys() runs makeknownhosts -r before each probe. Tests: genesis_console_mode.t drives xcat_console_mode() with the multiplexer shadowed; genesis_payload_verification.t drives the verifier over payload trees carrying each hole; genesis_testcase_helpers.t drives get_os(), check_genesis_file(), report_genesis_files() and testxdsh(); genesis_incorrectmasterip_check.t runs test.sh against a scratch tftp root. Each fails on the parent commit. The verifier also reports all three holes against the released xCAT-genesis-base-x86_64-2.19.0-snap202609021858 payload. Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com> (cherry picked from commit cb6021eb3cdb3abc75e4dd6704cb42b28074e140) Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xCAT Genesis builder
OpenEmbedded builder
OpenEmbedded is the current Genesis build path. It builds a small, pinned system instead of copying files from the build host.
Install the host packages required by OpenEmbedded, then install kas in a
Python virtual environment:
python3 -m venv .venv
. .venv/bin/activate
pip install -r xCAT-genesis-builder/oe/requirements.txt
xCAT-genesis-builder/oe/build x86_64
The build command accepts x86, x86_64, ppc64, ppc64le, armv7hf,
aarch64, and riscv64. Multiple architectures are built in the order given.
Artifacts are written below xCAT-genesis-builder/oe/.work/build/tmp/deploy/images.
The x86 artifact uses an i686 CPU baseline. The build carries the reviewed
Yocto release key in oe/keys and verifies its fingerprint locally.
The serial console normally opens a read-only Newt status screen. F1 shows
help, F2 shows diagnostics, and F3 shows live logs. F12 opens a confirmed
root maintenance shell; exiting returns to the status screen. Add
xcat.console=plain to the kernel command line for line output. In plain mode,
type shell and press Enter to open the same confirmed maintenance shell.
Export an image for xCAT after the build:
xCAT-genesis-builder/oe/export x86_64 \
xCAT-genesis-builder/oe/.work/build/tmp/deploy \
/tmp/xcat-genesis-x86_64
Copy the exported directory intact to the management node at
/opt/xcat/share/xcat/netboot/genesis/x86_64, then publish it:
mknb x86_64
mknb verifies the kernel and initramfs checksums before replacing the files
under the configured TFTP root.
Signed extensions
Build an extension recipe with the same machine configuration as the Genesis image. Then export it with the site's Ed25519 release key:
xCAT-genesis-builder/oe/export-extension x86_64 my-extension \
xCAT-genesis-builder/oe/.work/build/tmp/deploy \
/secure/genesis-extension.key /secure/genesis-extension.pub \
/tmp/my-extension-bundle
The private key stays outside the source tree. The exported bundle contains the extension image, manifest, signature, public key, and checksums.
A site layer can include that directory in its Genesis image with a small append file:
FILESEXTRAPATHS:prepend := "${THISDIR}/files:"
SRC_URI += "file://my-extension-bundle"
XCAT_GENESIS_EXTENSION_BUNDLE = "my-extension-bundle"
Place the exported directory below the append file's files directory.
Genesis verifies every bundled extension before registration and stops the
boot workflow if the image, manifest, signature, key, release, or architecture
does not match.
Genesis records registration time and memory use in /run/xcat/metrics.env.
After copying that file from a test VM, create a report with image sizes and
runtime measurements:
xCAT-genesis-builder/oe/report --runtime /tmp/metrics.env \
x86_64 /tmp/xcat-genesis-x86_64 > /tmp/report.env
Compare a later build with the saved report:
xCAT-genesis-builder/oe/report --runtime /tmp/metrics.env \
--baseline /tmp/report.env \
x86_64 /tmp/xcat-genesis-x86_64
Positive deltas mean a larger image, more memory use, or a slower registration.
See the architecture plan for the network, hardware, license, and signed-extension contracts.
Legacy RPM builder
The distribution-derived builder below is retained for the existing Genesis packages. It is not used by the OpenEmbedded images.
For every architecture in your cluster, be it x86_64, or ppc64, you need to have a default
initrd image for performing the initial boot and deploying the diskless operating system.
If for some reason, the versions included in the xCAT xcat-dep repository are insufficient, you can simply
run this utility on the target architecture which will build a xCAT-genesis-base RPM designed for the target
architecture. You can then transfer that RPM to your xCAT Management Node, install it, and then rebuild
the various netboot images.
Pre-requisites
The xCAT-genesis-builder package is designed to be run from a Red Hat compatible operating system
that support the Red Hat Package Manager (rpm) development tools. The script buildrpm will
attempt to install some of these core packages if they are not already present.
Instructions for buiding xCAT-genesis-builder RPM
Latest version 2.16.3 of xCAT-genesis-builder RPM available at https://www.xcat.org/files/xcat/xcat-dep/2.x_Linux/beta and was verified to install and run on Fedora28 and Red Hat 8. Earlier version 2.14.5 of xCAT-genesis-builder RPM available at https://www.xcat.org/files/xcat/xcat-dep/2.x_Linux/beta and was verified to install and run on Fedora26 and Red Hat 7.
If a new version of xCAT-genesis-builder RPM needs to be built:
- Clone
xcat-coregit repository:
git clone -b master https://github.com/xcat2/xcat-core.git
- Build new
xCAT-genesis-builderRPM:
cd xcat-core
./makerpm-genesisbuilder
Instructions for buiding xCAT-genesis-base RPM
First, you need to clone the xcat-core repo to the target architecture using the following command:
git clone -b master https://github.com/xcat2/xcat-core.git
Once there, you can choose to either build the entire package, or more simply, do the following (assuming 2.16.10 is your xCAT version):
cd xcat-core/xCAT-genesis-builder
If needed, update version number inside xCAT-genesis-base.spec
./buildrpm
If this command is successful, runs error free, it will generate a xCAT-genesis-base RPM that you can transfer
to your xCAT server and install and then rebuild your netboot images prior to deploying
the netboot images to your nodes.
The RPM will be placed into the following directory /root/rpmbuild/RPMS/noarch if the build is successful.
When running the buildrpm the output should look similar to the following:
[root@vmhost6 xcat-core]# cd xCAT-genesis-builder
[root@vmhost6 xCAT-genesis-builder]# ./buildrpm
Last metadata expiration check: 3:03:21 ago on Fri 27 Aug 2021 06:07:01 AM EDT.
Package rpmdevtools-8.10-8.el8.noarch is already installed.
Package rpm-build-4.14.3-14.el8_4.x86_64 is already installed.
Package screen-4.6.2-12.el8.x86_64 is already installed.
Package lldpad-1.0.1-13.git036e314.el8.x86_64 is already installed.
Package mstflint-4.15.0-1.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Last metadata expiration check: 3:03:23 ago on Fri 27 Aug 2021 06:07:01 AM EDT.
Package efibootmgr-16-1.el8.x86_64 is already installed.
Package bc-1.07.1-5.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
/root/xcat-core/xCAT-genesis-builder
cp: -r not specified; omitting directory '/root/xcat-core/xCAT-genesis-builder/debian'
Creating the initramfs in /tmp/xcatgenesis.351827.rfs using dracut ...
Expanding the initramfs into /tmp/xcatgenesis.351827/opt/xcat/share/xcat/netboot/genesis/x86_64/fs ...
623218 blocks
Adding perl libary /usr/share/perl5
Adding perl libary /usr/lib64/perl5
Adding kernel /boot/vmlinuz-x86_64 ...
/root/xcat-core/xCAT-genesis-builder
Tarring /tmp/xcatgenesis.351827/opt into /root/rpmbuild/SOURCES/xCAT-genesis-base-x86_64.tar.bz2 ...
Building xCAT-genesis-base rpm from /root/rpmbuild/SOURCES/xCAT-genesis-base-x86_64.tar.bz2 and /root/xcat-core/xCAT-genesis-builder/xCAT-genesis-base.spec ...
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.z5jzb9
+ umask 022
+ cd /root/rpmbuild/BUILD
+ exit 0
Executing(%build): /bin/sh -e /var/tmp/rpm-tmp.Q5Rg87
+ umask 022
+ cd /root/rpmbuild/BUILD
+ exit 0
Executing(%install): /bin/sh -e /var/tmp/rpm-tmp.F7t435
+ umask 022
+ cd /root/rpmbuild/BUILD
+ '[' /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64 '!=' / ']'
+ rm -rf /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
++ dirname /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ mkdir -p /root/rpmbuild/BUILDROOT
+ mkdir /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ rm -rf /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ mkdir -p /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ cd /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ tar jxf /root/rpmbuild/SOURCES/xCAT-genesis-base-x86_64.tar.bz2
+ cd -
/root/rpmbuild/BUILD
+ :
Processing files: xCAT-genesis-base-x86_64-2.16.10-snap202108270912.noarch
Provides: xCAT-genesis-base-x86_64 = 2:2.16.10-snap202108270912
Requires(interp): /bin/sh
Requires(rpmlib): rpmlib(BuiltinLuaScripts) <= 4.2.2-1 rpmlib(CompressedFileNames) <= 3.0.4-1 rpmlib(FileDigests) <= 4.6.0-1 rpmlib(PartialHardlinkSets) <= 4.0.4-1 rpmlib(PayloadFilesHavePrefix) <= 4.0-1
Requires(post): /bin/sh
Conflicts: xCAT-genesis-scripts-x86_64 < 1:2.13.10
warning: Arch dependent binaries in noarch package
Checking for unpackaged file(s): /usr/lib/rpm/check-files /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
Wrote: /root/rpmbuild/SRPMS/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.src.rpm
Wrote: /root/rpmbuild/RPMS/noarch/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.yShjx5
+ umask 022
+ cd /root/rpmbuild/BUILD
+ /usr/bin/rm -rf /root/rpmbuild/BUILDROOT/xCAT-genesis-base-x86_64-2.16.10-snap202108270912.x86_64
+ exit 0
Once you transfer the RPM to your xCAT server, you should install it. However, if there is
already an xCAT-genesis-base-* RPM for your architecture installed, you will have to uninstall
it first using the command below, assuming x86_64 architecture. We use the
--nodeps option as the uninstall will fail due to other dependencies.
rpm -e --nodeps xCAT-genesis-base-x86_64*
To install the new RPM, issue the following command, using the 2.16.10 example from above:
rpm -ivh xCAT-genesis-base*.rpm
Now, assuming that the architecture that you have built the xCAT-genesis-base for was ppc64,
you would then, on the xCAT server run the following command:
mknb ppc64
At this point in time, you can discover nodes by following https://xcat-docs.readthedocs.io/en/stable/guides/admin-guides/manage_clusters/ppc64le/discovery/mtms/discovery.html
Open Source License
xCAT is made available under the EPL license: https://opensource.org/licenses/eclipse-1.0.php
Developers
Want to help? Check out the developers guide!