genesis_ubuntu_build_root.t cut REQUIRED_PACKAGES out of
builddeb-genesis-base with a regex and evaluated it through bash -c. It
then matched the apt-cache fallback calls as text. Reformatting the
script broke it, and a wrong choice between renamed packages passed it.
genesis_payload_verification.t ran the verifier script and parsed its
stderr.
genesis_ubuntu_build_root.t now calls required_packages() with a chosen
set of carried packages and asserts the exact result: the amd64 extras,
the name picked for each renamed package, the order apt is asked,
tzdata-legacy, and the error for a release that carries neither name.
It reads the mandatory commands from XCAT::GenesisPayload, the code the
build uses. genesis_payload_verification.t calls the XCAT::GenesisPayload
functions with chosen payload trees and asserts the exact missing paths
and results.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
builddeb-genesis-base built its package list inline, between apt-get
update and apt-get install. A test could reach the list only by cutting
the REQUIRED_PACKAGES assignment out of the script and evaluating it.
It could not reach the choice between renamed packages (bind9-dnsutils
or dnsutils, util-linux-extra or util-linux) at all. The payload check
in verify-genesis-payload was bash, so a test could only run the script
and read its stderr.
XCAT::GenesisBuildRoot::required_packages() now returns the list for a
dpkg architecture. A code ref says which packages the release carries;
the default asks apt-cache. builddeb-genesis-base calls it at the same
point in the build and installs the same packages.
XCAT::GenesisPayload holds the mandatory-command list and the payload
check. verify-genesis-payload runs its main() and keeps the same
arguments, exit codes and messages. buildrpms.pl stages the module
beside the script. Both modules use core Perl only, because the build
root has perl-base and nothing more.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The error-command test ended with
[ ! -e /run/testnode-logs.tar ]
meant to show the archive had not gone to the default path. It cannot show that.
The file may exist for reasons that have nothing to do with this test, in which case
the assertion fails while nothing is wrong; and its absence would be equally true if
the override had never worked at all. It answers a question about the host, not
about the run.
The positive assertion above it already carries the proof: the tar shadow writes a
marker, and the test greps for that marker in the path it passed. The output being
there is what shows the redirection went there.
Removing it changes nothing about what the test catches. With the override taken out
of the template, so the archive path is hard-coded again, the remaining assertion
still fails.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Brings in #7866, which removes --database from the createrepo call. Without it the
branch cannot index a repository on the shared build tree: createrepo_c tries to
write primary.sqlite there and the NFS re-export answers
Cannot open .repodata/primary.sqlite: Can not create db_info table: disk I/O error
which failed every EL build in xcat-ci #44 while the Ubuntu builds, which do not use
createrepo, passed.
Killing the command is not killing the build. sh() ran the command through /bin/sh,
and cancellation signalled that shell alone -- but dpkg-buildpackage starts workers
of its own, and those survive their shell. The lock was then released while they
were still writing debian/changelog and debian/control, which is the state the lock
exists to prevent: the next build takes the checkout and the two rewrite it
together.
The command now runs in its own process group, so cancellation can take all of it.
Both sides call setpgid, so neither depends on which runs first, and INT and TERM
are blocked across the fork so cancellation cannot land in the window before the
group exists.
Cancellation escalates from the caught signal to KILL, and then CHECKS: a shell that
has exited is not a build that has stopped, so it waits for the whole group to
disappear rather than for the leader to be reaped. If the group is still there after
that, the locks are RETAINED and the process exits non-zero. Releasing a lock while
a worker may still be writing is worse than leaving a lock behind for a person to
clear -- the first corrupts a build, the second stops one.
cancel_build ignores INT and TERM while it runs, so a second Ctrl-C cannot interrupt
the cleanup half way and release the lock early.
sh() also reports a signalled command as 128+signal instead of 0. $? >> 8 is zero
for a child killed by a signal, so a build stopped mid-way looked to its caller like
one that had succeeded.
Two cases added to builddebs_lock_cancellation.t: a build whose worker is a
grandchild, and a command killed by a signal. Verified by signalling the pid instead
of the group, which leaves the worker running and turns the first red.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor holds the later connections for a node whose handler
is still running, and forks the next one when it reaps that handler.
SIGCHLD is what brings the parent back to look: it interrupts the accept.
A handler can exit after the parent checks its queue and before the
accept begins. The signal is then handled where there is no accept to
interrupt, and the parent blocks in accept with a connection already
queued and ready to run. That connection waits until some other node
calls in. A single node retrying on its own waits until it times out.
do_installm_service now waits through wait_for_installm_connection, which
selects on the listening socket. The wait is bounded by
$installm_wakeup_seconds while connections are queued, so the parent
looks at its queue again instead of waiting for another client. An idle
monitor with an empty queue still waits without a bound, because a
handler that exits then leaves nothing to do.
The new case asserts the wait ends on its own bound with nothing to
accept, and ends at once when a connection is already there. It fails
when the bound is ignored.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The install monitor answers a node's requests in the order they arrived.
Two ways to lose that order had no test.
The first is a queued client that gives up. The parent holds the later
connections for a busy node unread, so a client that closes its socket
must not let the request behind it overtake the request that is running.
The new case runs one request, queues two more, drops the middle client,
and asserts the last request starts after the running one ends. It fails
when the per-node queue is removed.
The second is a fork that fails. The monitor then answers the node
itself, in line. The new case makes one fork fail and asserts the request
is answered and that the node's next request starts only after it. It
fails when the fallback drops the connection instead.
The file also asserts the lifted service holds no literal /var/run path.
Every access to the pid file goes through $installm_pidfile, so pointing
that variable at the scratch tree redirects all of them, and a path
written out again inside the routine would reach the host file.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
xcatd_install_monitor_concurrency.t failed about one run in five. The
assertion "the node is released only after the destiny advance finished"
compares the time the test read "done" from the socket against the time
the plugin stand-in recorded when it finished. The stand-in wrote that
time with %.3f, which rounds up, so a recorded time can be later than the
moment it was taken. The test then failed on the rounding and not on the
order:
'1790220468.11892' >= '1790220468.119'
The events file now holds whole microseconds from gettimeofday, and the
comparisons read the same clock. gettimeofday rounds nothing.
Each case also writes an events file of its own. A handler forked by one
case outlives the monitor that forked it, so it could append to the case
that runs next.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The build lock is released in DESTROY, and perl does not run DESTROY when a signal
ends the process. A build stopped with SIGTERM or SIGINT therefore left its lock
directory behind, and the next build of that checkout died on
FATAL: another build of <path> already holds <dir> (held by [pid=NNNN])
naming a pid that had already exited. Nothing clears it but a person. One such
directory blocked an openSUSE target across three consecutive CI runs before anyone
looked at what the lock actually said.
buildrpms.pl has released its lock on cancellation for some time, through an END
block and an abort handler. This is the Debian builder catching up.
The order matters, and is the reason this is not simply an END block. The command in
flight is stopped BEFORE the lock is released: handing the checkout to a second build
while dpkg-buildpackage is still rewriting debian/changelog and debian/control in it
is worse than holding the lock a moment longer. The wait for that command is bounded,
so a subprocess that ignores the signal cannot hold the lock for ever either.
sh() now forks and execs rather than calling system(), because system() gives no pid
and a handler cannot stop what it cannot name. The child _exits rather than exits, so
it never runs the parent's END block and releases a lock the parent still holds.
The handler is installed by XCAT::BuildUtils::install_build_cancellation rather than
written inline in the builder, so a test can use the same wiring the builder uses. A
test that installs an equivalent handler of its own proves the helper works while
saying nothing about whether anything calls it -- the first version of this test did
exactly that, and passed with the wiring removed.
Release is idempotent: a signal handler and then DESTROY both reach it, and the
second must not remove a directory a LATER build has since taken.
builddebs_lock_cancellation.t terminates the holder, then takes the lock again, and
checks no build subprocess was orphaned. Verified by removing the wiring: assertions
9 through 12 fail, naming the leaked lock, the refused build and the stray process.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The project had no written release process, and the 2.18 and 2.19
releases missed steps: the release table in the documentation, the
docs version, signed tags, and the wiki and website index pages.
A new Releases section describes the branch and version model, the
label and milestone that each pull request needs, and a checklist for
release candidates, publishing, the signed tag, the GitHub release, the
release notes, the website and the announcement. The build hosts, the
signing key and the publish procedure stay in the private repository of
the maintainers.
The OpenEmbedded metadata job ran on every pull request and took about
20 minutes. It reads only xCAT-genesis-builder/oe and
xCAT-genesis-scripts, and it pins its upstream sources to fixed commits.
A pull request that changes neither directory gets the same result each
time.
The job moves unchanged to its own workflow, which runs only when those
paths or the workflow file change. xcat_pr_test stays in xcat_test.yml
and runs on every pull request, because a required check that does not
run blocks the merge.
2.19.0 is released, and master still builds packages that call themselves
2.19.0. Every snapshot built from master since then carries the released
version, so a candidate cannot be told from the release it follows.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
A build tree can live on an NFS re-export. The kernel refuses locks on one --
"Clients are not allowed to get file locks or delegations from a reexport
server" -- so every flock() there answers errno 524, and a build that takes one
dies before it starts.
buildrpms.pl's per-target lock and BuildUtils.pm's take_build_lock, which
builddebs.pl calls for the Ubuntu core build, are both atomic mkdir claims now.
Each records its owner and names it when it refuses.
A directory is not released by a filehandle closing, which is how both locks
were freed before. buildrpms.pl releases from END, and again in abort_builds
because that handler re-raises the signal with DEFAULT and END blocks do not run
then -- a killed build would otherwise strand the lock for every later one.
BuildUtils returns a small object whose DESTROY releases it, preserving the
caller's "hold the returned value" contract.
Both releases are guarded by owning pid: both scripts fork, and the flock they
replace could not be released by a child.
builddebs_lock.t closed the returned value to prove the lock is released, which
is "Not a GLOB reference" against the new contract. It now lets the value go out
of scope. What it asserts is unchanged: a second build of the same checkout is
refused, and the next one succeeds once the first releases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
createrepo_dir passed --database, which writes *.sqlite.bz2. Building those
needs SQLite, and SQLite needs POSIX locks. A build tree can live on an NFS
re-export, where the kernel refuses locks outright: every attempt answers
errno 524.
So every target died with "Cannot open .repodata/primary.sqlite: Can not create
db_info table: disk I/O error", and the build staged nothing.
Measured on such a share, with a local control: a bare sqlite3 connect fails
there and succeeds on local disk; createrepo_c fails with --database and
succeeds without it, emitting primary/filelists/other as *.xml.zst.
Nothing this project ships reads the sqlite metadata. dnf on el8+ and zypper
both read the XML.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
site.cleanupdiskfullxcatpost names the node type it applies to, and site.5.rst
says "on the diskfull nodes". xcatdsklspost serves every type: updatenode calls
it for a diskless or statelite node as well, and the site value is one row that
reaches all of them. With cleanupdiskfullxcatpost=yes and cleanupxcatpost=no,
append_xcatpost_cleanup still appended the delete for those nodes.
The diskful branch now reads NODESETSTATE from the generated mypostscript and
appends nothing for netboot or statelite. That is the signal remoteshell,
hardeths, configeth, otherpkgs and servicenode already use for this question,
and mypostscript.tmpl exports it.
cleanupxcatpost is unchanged. It names no node type and keeps applying to every
one, which xcatdsklspost_xcatpost_cleanup.bats now asserts so a guard added to
the wrong branch shows up.
Four cases cover it: a netboot node and a statelite node keep their postscripts,
a diskful node is still cleaned, and cleanupxcatpost still empties a netboot
node. The first two fail against the parent commit.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
master changed the same three files this branch changes, so the pull request could not
merge. Conflicts and how each was resolved:
xCAT-genesis-builder/builddeb-genesis-base. Both sides teach the build root to cope with
a package whose name moved between releases. master added optional_packages(), which
keeps a package only where apt has a candidate, and called it for util-linux-extra. This
branch added add_first_available(), which takes the first name apt carries and fails when
it carries none, and calls it for bind9-dnsutils/dnsutils and util-linux-extra/util-linux,
plus add_if_available() for tzdata-legacy. The branch covers master's case and two more,
so its helpers are kept and optional_packages() goes with its only caller. Every other
master change to this file, including the DHCP client fix, is preserved.
xCAT-test/unit/genesis_payload_verification.t. master has four assertions this branch does
not: two payloads missing an absolute path. Its version is kept. The branch replaced
plan skip_all with a fail(), because skipping covers nothing when the file under test is
the gate itself, and that change is applied to master's version.
xCAT-test/unit/genesis_ubuntu_build_root.t. master's added assertions drive
optional_packages() directly, which the resolved builder no longer has. This branch's
version matches the implementation that survives, and it already dies rather than skipping
when the builder is missing, so it is kept whole.
prove -j4 -r xCAT-test/unit passes: 212 files, 5892 tests.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The ordering between two requests for one node was a chain of pipes: each child
held the write end and the next child for that node read the previous one to end
of file. End of file there means the previous process is gone, not that its work
finished, so a child that died released the one behind it -- and released the
wrong one, because each child waits on its immediate predecessor rather than on
the request actually in flight. The fork-failure path closed the predecessor and
served the request in line without waiting at all.
The parent now owns the order. %installm_busy names the handler serving a node,
%installm_queue holds the connections accepted for that node meanwhile, and the
next one is forked when the handler ahead of it is reaped. A handler that dies
cannot release the one behind it, and the fork-failure path has nothing to fall
back over, because it is reached only when the node has no handler.
Three other changes the same design makes possible or necessary:
- The answer to a destiny advance now follows the advance. Every other request
is still answered before it runs, because its result does not change what
the node does next. Holding one node costs no other node anything now, and
it lets the node retry an advance whose handler died -- which the old order
could not, because "done" was already on the wire.
- The monitor drains its handlers before it exits. Without this a restart
orphans them into the systemd service cgroup, where anything still running
at TimeoutStopSec is killed after its answer was already sent.
- SIGCHLD is caught, so a handler exiting interrupts accept and the parent
comes back to look for a connection queued for that node.
The pid file path is a variable, so the test can point the lifted routine at a
scratch file instead of the one a restarting xcatd reads to tell the running
monitor to let go of the port.
xcatd_install_monitor_concurrency.t grew the cases for all of it. Four
mutations, each caught by one assertion: forking every connection at once turns
the ordering case red; answering a destiny advance before the plugin turns the
release case red; removing the drain turns the stand-down case red; and leaving
the emptied queue entry behind turns the leak case red. The full unit suite is
194 files, 5662 tests, green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The step that fills lib/firmware in the Ubuntu netboot initrd asked modinfo
about $rootimg_dir/$module and looked a firmware name up under lib/firmware.
The copy step beside it takes a module from $customdir or $pathtofiles first,
and the kernel looks a firmware name up under updates/<kernel>, updates/,
<kernel>/ and lib/firmware. So a custom driver reached the initrd with no
firmware, even when the root image carried it, and a firmware override was left
out of the initrd altogether.
initrd_firmware_files now takes the module directories the copy step searches
and the kernel release. It resolves each module in that order before asking
modinfo, and keeps every firmware file that exists in the four directories the
kernel searches, so the override still wins on the node.
ubuntu_genimage_initrd_firmware.t covers both: its two new cases are red on the
commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
ubuntu_genimage_initrd_firmware.t drove the firmware step over a root image that
holds every module and every firmware file. It covered neither of the two places
genimage reads from beside the root image.
The test now puts a driver in the custom directory, with its firmware in the
root image, and a firmware override under lib/firmware/updates/<kernel>. Both
are red: the step asks modinfo about the module under the root image, where a
custom driver is not, and it looks for a firmware name under lib/firmware only,
where an override is not.
The five assertions that were there stay green.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
reg_linux_diskless_installation_flat corrupts the KVM machine type, proves the
node fails to boot, restores the machine type and then removes it again. On
ppc64 the two ladders name different machine types: the restore writes
machine:pseries-rhel7.6.0 and the cleanup removes machine:pseries-7.6.0. The sed
matches nothing, so the node keeps machine:pseries-rhel7.6.0 after the case ends
and the next case runs against a node the previous one changed.
The cleanup ladder now names the machine type the restore ladder writes.
diskless_flat_vmothersetting_machine.bats covers it: the ppc64le cleanup cases
are red on the commit before this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
diskless_flat_vmothersetting_machine.bats checked the restore with a substring
match and checked the cleanup only for the absence of "unary operator
expected". A cleanup that writes the machine type back, or leaves it in place,
passed both.
The test now reads the value chdef receives. The restore must write exactly
machine:<type>, and must keep a setting the node already carries. The cleanup
must write an empty value when the machine type is all there is, and must leave
the other setting behind when there is one. The chdef stub brackets its
arguments so an empty value is not the same as no call.
ppc64le is red on the cleanup: the restore ladder writes
machine:pseries-rhel7.6.0 and the cleanup ladder removes machine:pseries-7.6.0,
so the node keeps the machine type. x86_64 and riscv64 pass.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
run_zypper_local_repo extracted six lines of the zypper branch of otherpkgs and
evaluated each one in turn. The branch it measures is an if/else: the repository
is deleted only when the refresh fails. Evaluating the lines separately ran the
delete every time, so the test showed that zypper sd is reachable and never that
a repository which refreshes is kept.
The helper now evaluates the whole branch, from the #use zypper comment to the
apt branch that follows, with pmatch lifted from the same file. The success case
asserts rc=0 and no zypper sd; the failure case keeps its delete assertion.
Flipping the refresh test in otherpkgs to "if [ $? -ne 0 ]" turns both cases
red. The same mutation left the previous helper green, because it never
evaluated that line.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
deb_belongs_to_dist read any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease and --release
takes whatever the caller gives it, so `--release 1~rc1` left every package
excluded from every suite, with --genesis nowhere in the command and the run
still reporting what it published. Only xcat-genesis-base carries a codename,
because only it carries a kernel, so only it is asked.
genesis_dists() names the releases a Genesis image can be built on. A plain
--genesis run took the release list the rest of the build uses, and focal is on
it: focal ships debhelper 12.10 and the package declares debhelper-compat
(= 13), so sbuild stopped on the build dependencies before dracut ran and the
run ended on its first release. The default plan now leaves focal out and says
so. A release named with --genesis-dist is still built as asked.
genesis_deb_per_codename.t covers both. Applying the codename rule to every
package again fails five assertions; a genesis_dists that filters nothing, or
an image that loses its own rule, fails two each.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
deb_belongs_to_dist reads any trailing ~word in a package version as the
codename the deb was built for. Debian uses ~ for a prerelease, and --release
takes whatever the caller gives it, so `--release 1~rc1` puts one in every
package name:
xcat-client_2.19.0-1~rc1_all.deb noble=0 focal=0
The deb is then published into no suite at all, with --genesis nowhere in the
command, and the run still reports how many packages it published.
Only the Genesis image is built per codename, so only it can be excluded by
one. The new cases assert a prerelease version reaches every suite, that a
Genesis SCRIPTS deb is not treated as the image, and that the image itself is
still confined to its own suite when its version carries both.
The second half asserts genesis_dists(), which does not exist yet: a plain
--genesis run takes the release list the rest of the build uses, and focal is
on it. focal ships debhelper 12.10 -- measured on the focal management node --
against the package's debhelper-compat (= 13), so sbuild stops on the build
dependencies and the run ends on its first release.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The first subiquity error command redirects into /run/#HOSTNAME#-logs.tar. A
test can shadow tar, but not the redirection: the shell opens that file
whether or not tar runs, so the test writes to the host it runs on. Unprivileged
the open fails and "exit 0" hides it; CI runs as root, where the same line
creates or truncates the file.
The path is read from XCAT_ERROR_ARCHIVE, falling back to the same default, the
way XCAT_ERROR_CONSOLE already does on the next line. An install sets neither
variable and writes where it always did.
The test points the variable inside BATS_TEST_TMPDIR, has the tar shadow emit a
marker so the output can be traced to the file it named, and asserts the default
path under /run was not touched. Reverting the template to the fixed path fails
that case.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The first subiquity error command redirects tar's output into
/run/#HOSTNAME#-logs.tar. The test shadows tar, but the redirection belongs to
the shell and happens whether or not tar runs:
$ bash -c 'tar() { :; }; ( tar ... >/run/testnode-logs.tar 2>/dev/null; exit 0 )'
bash: line 1: /run/testnode-logs.tar: Permission denied
exit=0
Unprivileged the open fails, "exit 0" swallows it and the test passes. CI runs
as root, where the same line creates or truncates that file on the host.
The archive path now comes from the environment, the way XCAT_ERROR_CONSOLE
already does in the same block, and points inside BATS_TEST_TMPDIR. The tar
shadow emits a marker so the test can tell the output reached the file it
named, and a second assertion says the default path under /run was not touched.
The new case fails until the template reads that variable.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
REQUIRED_PACKAGES named util-linux-extra for every release. focal and jammy
have no such package -- apt reports "Candidate: (none)" -- so apt-get install
exits non-zero and, under set -euo pipefail, the build stops before dracut
runs. hwclock is in util-linux there, which is essential and already present.
optional_packages() keeps a package only where apt has a candidate for it, and
util-linux-extra goes through it. The unconditional list keeps isc-dhcp-client
and ifenslave, which every release has and neither of which the build root
carries by itself.
The call to verify-genesis-payload goes with it. That script is added by the
genesis payload branch, not this one, so the line stopped the build at the
point it was meant to guard.
Also corrects the plan count and a dereference in the test committed before
this one.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Daniel Hilst
Vinícius Ferrão
Vinícius Ferrão
Gary Skouson