A riscv64 run lost perl-Crypt-SSLeay to a transient mirror problem: the
bootstrap chroot's dnf got HTTP 404 for BaseOS primary.xml.gz on every Rocky
mirror (metadata mid-sync) and mock exited 30 (YumError) from --buildsrpm,
which the builder treated as a hard failure. Give the mock-driven builders --
mockbuild-perl-packages.pl, ipmitool, grub2-xcat and conserver -- a small
run_mock wrapper that reruns the same mock command once when it exits 30
(the package-manager failure code; build failures exit 10 and are not
retried), and use it for their --buildsrpm/--rebuild invocations.
mockbuild-all.pl only knew the mock-core-configs targets <os>+epel-<rel>-<arch>
and only ever built the host arch. Teach it the forcearch targets shipped in
mock-configs/ (today rocky-10-riscv64-xcat), selected with --target, so an
x86_64 host cross-builds and deploys rh10/riscv64 exactly like the native
per-EL repos (rh<rel>/<arch> layout, xcat-dep.repo with the xcat.org baseurl,
mklocalrepo.sh, buildinfo.txt, optional signing).
A target now has a profile (target_profile): EL release, arch of the rpms,
the mock config for its noarch deps, the dep builders to run and the required
set asserted after deploy. The native targets keep today's profile (every
builder, host arch, the full required set). The riscv64 profile:
- builds ipmitool-xcat, conserver-xcat and goconserver for riscv64
(--target-arch riscv64 to the builders: emulated mock rebuilds for the C
ones, GOARCH cross-compile + rpmbuild --target for goconserver);
- builds grub2-xcat, a noarch packaging, in the native EPEL-free
rocky-10-<host arch> chroot rather than the emulated one;
- does not build the x86 bootloaders (elilo-xcat, syslinux-xcat, xnba-undi)
and does not require them;
- runs mockbuild-perl-packages.pl with --target-arch riscv64,
--noarch-mock-cfg rocky-10-<host arch> and --epel-gap, since riscv64 has
no EPEL to take xCAT's other perl deps from;
- installs mock-configs/<target>.cfg into /etc/mock/ when it is missing there
(mock and the builders' include('/etc/mock/<target>.cfg') overlays need
it); a host copy that differs from the shipped one is an error, never
silently used.
$arch is now the arch of the target being built (set per target), $host_arch
the uname -m one used to pick the default rh8/rh9/rh10 targets; the deploy and
the repo metadata take the arch from the target profile. --scrub-all-chroots
also scrubs the noarch chroot. BUILD.md documents the riscv64 build, what it
produces and the known exclusions (perl-DB_File, perl-SOAP-Lite).
EL10 x86_64/ppc64le take these perl deps of xCAT from EPEL; riscv64 has no EPEL
and Rocky Linux 10 riscv64 BaseOS/AppStream/CRB do not carry them, so xcat-dep
builds them for that arch. Add them to mockbuild-perl-packages.pl's package
table, and a --epel-gap switch that appends them to the build list (the
default list6 build is unchanged):
perl-Crypt-Blowfish EPEL 10 src.rpm (2.14-25.el10_0, vendored now; the spec
in the dir is a SUSE one); its optional tests need
Crypt::CBC, so it 'needs' our perl-Crypt-CBC
perl-Crypt-CBC perl-Crypt-CBC.spec + Crypt-CBC-2.33.tar.gz: the spec
is now the Fedora one of the vendored fc29 src.rpm
(release 21) with the BuildRequires an EL10 buildroot
lacks (make, perl-interpreter); it replaces the SUSE
spec (perl-macros) that could not build on EL (noarch)
perl-Crypt-Rijndael perl-Crypt-Rijndael-1.13-10.fc29.src.rpm
perl-Digest-SHA1 perl-Digest-SHA1-2.13-23.fc28.src.rpm
perl-Expect perl-Expect-1.35-6.fc29.src.rpm (noarch)
perl-Mail-Sender perl-Mail-Sender-0.903-7.fc29.src.rpm (noarch)
perl-Net-DNS Net-DNS.spec + Net-DNS-0.80.tar.gz, now built with
--noxs as noarch (the cpan2rpm spec hard-coded
buildarch x86_64 for the XS dn_expand) and with the
BuildRequires cpan2rpm specs never carry; release 2
perl-Net-IP perl-Net-IP-1.26-30.el10_0.src.rpm (noarch)
perl-Path-Class new dir, EPEL 10 src.rpm (0.37-24.el10_0, noarch): only
a build dep -- Crypt-SSLeay's Makefile.PL needs it --
so perl-Crypt-SSLeay 'needs' it; on x86_64 EPEL still
provides it and nothing changes there
Two EPEL-only deps of xCAT are deliberately not built: perl-SOAP-Lite
(1.27-3.fc29 src.rpm is in the table for completeness, but its BuildRequires
IO::SessionData, MIME::Lite, XML::Parser::Lite and Test::XML are EPEL-only as
well, so it can neither be built nor installed without EPEL; xCAT uses it for
HP blade/VirtualBox support only) and perl-DB_File (needs libdb, which EL10
dropped and Rocky Linux 10 riscv64 does not have at all; only xCAT-server's
Confluent client uses it and xCAT-server merely recommends the package).
The per-package builders take the arch of what they build from 'uname -m', which
is wrong for a forcearch mock config such as rocky-10-riscv64-xcat built on an
x86_64 host: the chroot produces riscv64 rpms and the scripts then reject them
("Unexpected RPM arch"). Give every arch-producing builder a --target-arch
option (default: uname -m, so nothing changes for native builds):
- ipmitool/mockbuild.pl, conserver/mockbuild.pl: look for and verify
<target-arch> rpms. ipmitool's install smoke test cannot install a foreign
rpm on the host, so for a cross build it installs the rpm into the build
chroot with mock --install and runs ipmitool-xcat -V there (conserver already
smoke-tests in the chroot).
- goconserver/mockbuild.pl: the binaries are built on the host, so cross-compile
with GOARCH (x86_64 amd64, aarch64 arm64, riscv64 riscv64, ...) and package
with rpmbuild --target <arch>; rpm refuses 'BuildArch: <foreign arch>' on
this host ("No compatible architectures found for build"), so that line is
only emitted for native builds. A cross build cannot install its rpm on the
host either, so the smoke test unpacks it and runs goconserver and congo
through the binfmt handler (qemu-user-static) the forcearch mock builds of
the other deps need anyway.
- mockbuild-perl-packages.pl: --target-arch for the 'native' rpm check and the
default result/log dirs; the 'native' (XS) packages of a cross build are
smoke-tested inside the build chroot (mock --install, then perl -M<module>
there) instead of on the host; --noarch-mock-cfg to build the noarch
packages in a native chroot of the same release instead of the emulated one
(the rpms are identical for every arch, and an emulated perl build is an
order of magnitude slower); and a 'needs' key in the package table: a
package that needs others is built after them, in waves, with their rpms
installed into its chroot via mock --additional-package. Needs outside the
selected set are ignored, so the default list6 build is unchanged (the
chroot provides the module, e.g. from EPEL). This is what lets an EPEL-free
chroot build perl-Crypt-Blowfish on top of our own perl-Crypt-CBC, or
perl-Crypt-SSLeay with perl-Path-Class.
xCAT::NetworkUtils uses Net::IP, and perl-Net-IP is EPEL-only on EL10, so an
architecture without EPEL (riscv64) has to get it from xcat-dep. Vendor the
EPEL 10 source rpm, as the other per-package dirs do with their Fedora
src.rpms:
perl-Net-IP/perl-Net-IP-1.26-30.el10_0.src.rpm (noarch)
and register it in mockbuild-perl-packages.pl's package table in 'srpm' mode
(not in the default build list: EL10 x86_64/ppc64le keep taking it from EPEL).
xcat-dep has no riscv64 build host, and mock-core-configs' rocky-10-riscv64.cfg
cannot be used from the x86_64 hosts that build the repos: it only admits a
riscv64 host (legal_host_arches) and names its chroot 'rocky-10-x86_64'.
Ship mock-configs/rocky-10-riscv64-xcat.cfg: the stock templates/rocky-10.tpl
(BaseOS/AppStream/CRB/extras for $basearch=riscv64) with root
'rocky-10-riscv64-xcat', target_arch riscv64, legal_host_arches x86_64+riscv64
and forcearch riscv64, so mock runs the Rocky Linux 10 riscv64 chroot through
user-mode QEMU and the packages are built by the chroot's own riscv64
toolchain. There is no EPEL for riscv64, so nothing EPEL-only is reachable from
this chroot (the build scripts deal with that in the following commits).
Document in BUILD.md what the host needs before this config works: a static
qemu-riscv64 registered in binfmt_misc with the F flag (EL10 has no
qemu-user-static-riscv package; take the binary from a Fedora container), the
/usr/bin/qemu-riscv64-static file mock insists on, podman for the bootstrap
image, golang for the goconserver cross-compile, the install of the config
into /etc/mock/ and the commands that validate the setup. Verified on an
x86_64 EL10 host with mock 6.7: chroot init in about 3.5 minutes, 'uname -m'
inside the chroot reports riscv64.
riscv64 nodes boot through UEFI firmware and grub2 only, and xCAT hands
DHCP client architecture 27 (0x001b) the boot file
boot/grub2/grub2.riscv64. Nothing shipped that file.
Add the unmodified grub2 UEFI image from the Rocky Linux 10 riscv64
BaseOS tree (EFI/BOOT/grubriscv64.efi, the copy used for installation
media and network boot, built from grub2-2.12-46.el10_2.rocky.0.1.src.rpm,
GPL-3.0-or-later, provenance and checksum recorded in the readme) as
Source2, install it under /tftpboot/boot/grub2/riscv64-efi/ and let the
rpm and deb scripts copy it to /tftpboot/boot/grub2/grub2.riscv64 the
same way grub2.ppc is made from the POWER core image. mockbuild.pl
verifies the image type (it reads the PE header itself: file(1) only
learned the RISC-V machine types in 5.38, so an EL8 build host would
reject a good image), the rpm payload and the installed file (byte for
byte the source image); the deb installs the same tree.
Pin that a locked APT directory is refused with a message naming it, that
--force-unlock takes over and the lock is released at the end, and that a run
which skips building keeps the build results it exists to collect.
Everything from the pool wipe to the signature is one transaction over a shared
tree, and the Genesis packages are verified inside it, so a second writer between
that verification and apt-ftparchive would be indexed and signed unchecked --
or, arriving later, would leave clients with checksum failures against metadata
this run signed.
Take the tree for the duration with an atomic mkdir lock, the same NFS-safe
scheme mockbuild-all.pl uses for its output, released on exit and overridable
with --force-unlock when a killed run left one behind.
The staging repositories are cleared per invocation, but the builder results are
not: a reused --run-id leaves the previous run's packages under build-results,
where a step that fails this time is collected from the last time it succeeded.
Individual failures are tolerated by design, so this can mix two invocations in
one signed repository.
Start the result tree empty whenever the run builds. --skip-build collects the
repository-level build-output tree instead and keeps what is there.
Pin the rule that decides whether an invocation produced anything: every
attempted step failing is a total failure, one survivor is not, and a run with
no steps to attempt is unaffected.
Tolerating individual dep-builder failures is deliberate: some packages are el-
or arch-pinned and are expected to fail on some targets. Tolerating all of them
is not -- it means the builder is unusable, the invocation produced nothing, and
whatever the run publishes came from somewhere other than this build.
Count the failures on both the serial and the parallel path and stop when they
account for every attempted step, before collection can take an earlier run's
artifacts for this one's. The rule itself lives in BuildUtils, where it can be
exercised without a builder.
A package left in the staging repository by an earlier run is invisible to
collection but visible to createrepo and deploy. Pin that the run clears it, so
the empty-collection guard cannot be satisfied by an earlier invocation's output.
The collection guard counts what this run copied, but the staging repositories
were never cleared: packages left by an earlier run with the same --run-id sat
there unseen by collection, were indexed by createrepo and published by
deploy_target, where the name-only dependency assertion accepted them. A run
whose builders all failed could therefore ship a previous run's packages.
Empty the staging repositories before collection, so everything they hold
afterwards comes from this invocation.
Pin that every pooled Genesis package is checked against the release manifest
again before apt-ftparchive reads the pool, so the check cannot be dropped
without a failing test.
apt-ftparchive indexes and the signature covers whatever is in the pool at that
moment, while the packages were checked when they were copied. Verify them again
against the release manifest immediately before the indexes are generated, so a
package that changed in between cannot be published as a verified one.
Vinícius Ferrão
Daniel Hilst