run_cases counts the Passed and Failed lines a run prints, so a case that
crashed before either one was counted nowhere and the phase reported
success over cases that never ran. Each phase now knows how many cases it
asked for and fails when the verdicts do not add up.
A failed query for the case list returned an empty list, which read as a
phase with nothing to do; it is a failure now. A backend whose setup failed
runs its teardown before the next one starts, rather than leaving the
machine configured for a backend that is no longer under test.
The branch's own comments had grown to explanations of the reasoning
behind each decision. Trim them to the fact and its consequence: why a
line exists, and what breaks without it. Long blocks come down to a
hundred words, most to twenty or fewer, and the file headers keep only
their usage tables and the invariant each suite rests on.
No code changes. Every unit test, scenario validation and syntax check
still passes.
The comment named only the DHCP cases; the provisioning ones are kept
out of that phase for the same kind of reason -- they add an
interface, nodes, a network and a rewritten zone.
The provisioning cases go where a node meets them: after the whole
ci_test set, so they are not reconfiguring the machine underneath it, and
before the DHCP phases.
run_provtest_unit_tests runs the offline half first -- the Python suite
and a parse of every scenario -- so a file that does not even load is
reported as that rather than as a case failure, and skips rather than
fails when python3 is absent. run_prov_wire_test then runs the prov_wire
cases once, not once per backend: nothing in them depends on which DHCP
server is installed, and what a node is told over DHCP is dhcptest's
subject.
The CI run is now three phases in order: the perl and bash unit tests,
the ci_test set, then the DHCP on-wire cases.
The wire cases carry dhcp_wire and deliberately not ci_test. They are the
only cases that do not leave the machine alone while they run -- a veth
pair appears, site.dhcpinterfaces and site.dhcpbackend change, the DHCP
daemon is stopped and started -- and although each one restores all of it
from a trap, a ci_test case running in between would be sharing a
management node that is mid-reconfiguration. Holding them out of the set
is also what stops them being run twice.
run_dhcp_wire_test runs the whole set once per backend installed: against
ISC, then against Kea, each pass bracketed by `dhcpfixture.sh
backend-setup` and `backend-teardown`. It reports separately from the
fast regression test and counts case runs rather than cases, since each
case is run once per backend and each failure carries the backend name.
The phase writes its own xcattest configuration without a dhcpbackend
key: xcattest applies [Table_site] before every case, so the regression
conf's `dhcpbackend=isc` would have reset the backend case by case and
left the run silently testing ISC twice.
Which DHCP backend a management node runs is an implementation default of
its distro -- kea on the newer releases, isc on the older ones -- so a
node booting on the same network has to be told the same things either
way. Until now CI exercised whichever backend the runner happened to
configure, which proves half of that and hides every drift between the
two.
The wire cases now carry a dhcp_wire label, and the CI driver holds them
back from the main pass: it asks dhcpfixture.sh which backends are
installed, then runs the whole set against one and again against the
other. Each pass is bracketed by backend-setup, which points
site.dhcpbackend at the backend and stops the other daemon -- two servers
on one wire both answer the same DISCOVER -- and backend-teardown, which
restores the site table and restarts what was running before.
Running every case under one backend and then every case under the other,
rather than switching inside each case, reconfigures the daemon once per
pass instead of once per case, and gives each failure in the summary the
backend name it belongs to. The summary counts case runs rather than
cases, since a wire case is run once per backend.
The check gate is relaxed from [ -x ] to [ -f ]: the fixture invokes
dhcptest as `python3 src/dhcptest`, so the execute bit only matters to
someone running it directly, and testing it made every wire case skip on
Debian.
dhcptest_backend_switch is dropped, along with the fixture's backend and
alt-backend actions: with every case running under both backends there is
nothing left for a case that switches one.
xCAT's DHCP behaviour is tested from two directions today, neither of which
touches the wire: xCAT-test/unit/dhcp_*.t assert on generated config text, and
xCAT-test/integration/dhcp_{isc,kea}_config_validation.t feed that config to a
real daemon and check it parses. Both stop at "the server accepted our
config". Nothing verifies that a client sending option 93 = 0x000b actually
gets an aarch64 loader back.
The one existing wire tool, xCAT-probe/subcmds/detect_dhcpd, hand-packs a
fixed DISCOVER carrying no option 60, 77 or 93 (so it never reaches any arch
branch), parses replies by regexing tcpdump output, and binds an
IO::Socket::INET to a local address, so it cannot run on a provisioning NIC
that has no address yet.
dhcptest fills that gap. It is a Python 3 tool using only the standard library
plus Scapy, speaking raw Layer 2, driving DHCP transactions declared in INI
files and asserting on what came back:
dhcptest run -i eth1 --set net=10.0.0.0/24 conf/full-lease.conf
dhcptest validate conf/*.conf # no root, no network, no scapy
dhcptest list conf/*.conf
dhcptest discover -i eth1 # ad-hoc, no conf file
Scenarios cover DISCOVER/OFFER, the full lease, renew and rebind, reserved
versus pooled addresses, deliberate silence, the PXE architecture matrix and
the iPXE user-class split. Assertions are a small DSL -- target, operator,
value -- over message type, BOOTP header fields, options by number or name,
subnet membership, and the boot file a client would actually use (option 67
or the header, since servers differ on which they fill).
Two boundaries are enforced by unit test rather than by convention:
- The tool is implementation-agnostic. It exchanges packets and checks
fields; it never reads the xCAT database, never runs an xCAT command, and
names no DHCP implementation. Anything that differs between servers is
expressed by whoever writes the .conf.
- Only runner.py imports scapy, so validate and list stay usable in CI on a
host with neither scapy nor root.
Host networking is never modified. The client MAC defaults to a synthetic
locally-administered address, everything goes over a raw L2 socket, no leased
address is ever configured, and the ARP responder answers only for addresses
the session was actually granted.
Variables in .conf files are configparser's own BasicInterpolation, %(name)s,
fed from [vars] and --set. $offer.address and friends are resolved separately
at step-execution time, because they name a reply that has not arrived when
the file is read; BasicInterpolation gives $ no meaning, so the two coexist
without escaping.
Packaging and CI:
- xCAT-test.spec installs dhcptest alongside unit/ and integration/, and
only recommends python3-scapy: it lives in EPEL on EL, and a hard Requires
would make xCAT-test uninstallable on a management node without EPEL.
The deb depends on it outright.
- github_action_xcat_test.pl runs the unit tests and `dhcptest validate`
last, after the fast regression.
- autotest/testcase/dhcptest/cases0 carries the two offline cases plus a
wire case that stands down with an explanation when no provisioning NIC
is available.
Verified against a real DHCP server in a network namespace: 34 lease, renew,
rebind, reservation and silence assertions and 22 PXE/iPXE assertions all
pass, and host networking is untouched before and after.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Shell unit tests were introduced under xCAT-test/autotest/bats, but the existing source-tree unit suite already lives directly under xCAT-test/unit. Keeping the BATS suite under xCAT-test/bats makes the unit-test layout consistent and keeps autotest reserved for xcattest-driven functional cases.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The go-xcat shell behavior tests were written as Perl harnesses, which made the shell assertions harder to read and kept shell-specific setup outside a native shell test framework.
Add BATS to the GitHub Actions dependency set, run BATS tests from the same preserved source tree as the Perl unit suite, and move the go-xcat repository checks into xCAT-test/autotest/bats.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
Four fixes from @viniciusferrao's review plus the CI break his review predates.
orig tarball version. dpkg looks for <source>_<upstream>.orig.tar.gz with no
Debian revision, and the call site passed the full Version-Release. The rule now
lives in BuildUtils::upstream_version and orig_tarball_name applies it, so the
call site cannot get it wrong whichever string it is handed. Currently dormant --
every package is Format: 1.0, so the quilt branch does not run, which is why the
differential build did not catch it.
--dest could write to the filesystem root. Cwd::abs_path returns undef when a
PARENT component is missing (a missing leaf is fine), and the caller interpolated
that, so `--dest /no/such/parent/out` became `/debs` and `/xcat-core` at /.
Replaced with BuildUtils::resolve_dest, which is rel2abs and purely lexical --
correct for an output directory that does not exist yet.
Generated debian/control left behind. xCAT-genesis-scripts has no debian/control
of its own; it is generated from control-<arch>. The cleanup restored only files
that already existed, so the generated one stayed. Worse than dirty: ppc64el ran
last, so the restore put back the amd64 BACKUP and the leftover was the wrong
architecture's control, which a later single-arch build would have started from.
with_prepared_tree now records created files and removes them. Verified by a real
build: the checkout is byte-clean afterwards, matching the oracle.
CI install step. build-ubunturepo wrote its repo to $curdir/../../xcat-core,
which under GitHub's work/<repo>/<repo> layout IS $RUNNER_WORKSPACE, so
install_xcat's `./mklocalrepo.sh` happened to be in the directory it chdir'd to.
builddebs.pl writes inside the checkout instead -- that outside-the-checkout path
is what used to rm -rf the tree -- so install_xcat now names the script by its
real location and fails with a clear message if the build produced no repository.
This is what reddened xcat_pr_test at 2m13s; the builder itself was fine (the
exact CI invocation, `./builddebs.pl --force` with no --dest, returns 0 with all
14 packages).
The executable bit was already fixed before the review landed.
Both new helpers are tested and mutation-verified: not stripping the revision
reddens 3 assertions, swapping rel2abs back to abs_path reddens 2. Equivalence
re-measured after these changes -- all 14 packages identical to build-ubunturepo
in control and in every non-changelog file by md5.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
build-ubunturepo was 710 lines of shell doing the Debian half of what
buildrpms.pl does for rpms, with no code in common and a different CLI. It also
carried paths that are dead: GSA uploads, the PROMOTE/PREGA release flows, and a
-d mode that built an xcat-dep repository from a different project's packages.
builddebs.pl replaces it and mirrors buildrpms.pl -- Getopt::Long options, one
package list, build then index then sign -- so the two builders read the same way
and share BuildUtils.pm.
The design rests on one fact: xcat-core debs are Perl. They are byte-identical
for every Ubuntu release, so they are built ONCE and the same files are published
into every codename. Only xCAT, xCATsn and xCAT-genesis-scripts carry an
architecture, and there the difference is packaging metadata, not compiled
output. That is why this needs no sbuild and no per-codename chroot -- unlike
xcat-dep, whose packages are compiled and genuinely differ per release.
BuildUtils.pm holds what both builders need and what was worth making testable:
the Version-Release derivation from the commit time, the xCAT-probe helper
staging, the deb arch and dist tables, the debian/control version pinning, the
changelog rewrite, the reprepro conf generation, and the build lock. Every
function is pure or takes its side effect as an argument, so build_utils.t (45
assertions) drives each one rather than grepping a builder for evidence that it
is called. Verified by mutation: shrinking the arch table reddens 1, dropping
the /g from the control pin reddens 2.
The env-var CLI maps to options: BUILDALL=1 -> --force, GPGSIGN=1 -> --gpg-sign,
GPG_HOME -> --gpg-home, DEST -> --dest, DISTS -> --dist (repeatable). UP=0 has no
equivalent because uploading is gone -- the CD pipeline's deploy step publishes.
Callers updated: github_action_xcat_test.pl and travis.pl. The comment in
github_action_xcat_test.pl explaining why CI copies the tree before building is
corrected -- build-ubunturepo rm -rf'd $curdir/../../xcat-core, which under
GitHub's work/<repo>/<repo> layout is the checkout's own parent; builddebs.pl
writes under dist/debs inside the checkout and restores every file it edits, so
the copy is now only isolating the tests from build residue.
Two tests moved with it. build_ubunturepo_lock.t extracted the lock out of the
shell with a regex and ran that; the lock is now a function, so builddebs_lock.t
calls it -- and asserts what actually matters, that two builds of one checkout
fail fast while two builds of different checkouts run concurrently.
ubuntu_2604_pkglist.t asserted that resolute appeared in a shell fragment of
build-ubunturepo's source; it now asks BuildUtils for the release list and checks
a resolute stanza reaches conf/distributions. That assertion would have passed on
any file containing the fragment and broken on a reflow that changed nothing.
Verified: prove -r xCAT-test/unit fails on 6 files here against 7 on
upstream/master, the difference being apache_config_sources.t, fixed by the
preceding commit. The remaining 6 are missing DB modules on the machine that ran
it and are identical on both.
NOT done here, and required before this can merge: the Ubuntu core CD pipelines
still invoke ./build-ubunturepo (ci/ubuntu/Jenkinsfile.core-ubuntu-{devel,stable}
in VersatusHPC/xcat-core-ci-cd, and the inline script in each live Jenkins job).
Those must be switched to builddebs.pl in the same change window.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The PR test job ran apt-get against the xcat.org repository with no
time limit of its own. When the repository stalled, the job hung on the
update step until the sixty minute workflow limit canceled the run, and
the log gave no reason.
Wrap the apt-get update and install steps in a timeout command and give
apt a transfer timeout with retries. A stalled repository now fails the
step in minutes, the driver prints its install error report, and a
rerun is possible at once.
run_fast_regression_test() prints a case's output only when it fails.
For 250 cases that is the right default, but it leaves no way to tell
whether a passing case did real work or skipped everything. That is not
academic for cases wrapping prove: prove exits 0 both when tests pass
and when every test skips, so integration_tests reports green either
way and the log cannot distinguish them.
Add @verbose_cases. A case named there has its output printed on a pass
as well, and the failure branch no longer prints a second copy. Seed it
with integration_tests to find out which of the three integration tests
actually run on a runner -- in particular whether
dhcp_kea_config_validation.t validates from /etc/kea now that the case
runs as root, or still skips. Emptying the list restores the previous
behaviour exactly.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The unit test stage failed on the first CI run with
Cannot detect source of 'xCAT-test/unit/*.t'
Files=0, Tests=0
Result: NOTESTS
the glob reached prove unexpanded because it matched nothing. The source
tree is gone by the time the tests run: build-ubunturepo sets
local_core_repo_path="$curdir/../../xcat-core"
and rm -rf's it before creating the apt repository there. GitHub checks
out into work/<repo>/<repo>, so for /home/runner/work/xcat-core/xcat-core
that path resolves to the checkout's own parent and the build wipes the
checkout, leaving an empty directory of the same name behind. The cd
still succeeds, which is why prove was handed a literal glob rather than
failing outright. This is also why every testcase that predates this
change proves /opt/xcat/share/xcat/tools/autotest/unit: after the build
the installed copy is the only one left.
Copy the checkout aside in preserve_source_tree() before the build and
prove that copy, so FindBin still resolves to a real source tree. Switch
to `prove -r xCAT-test/unit` as well, so a missing directory fails loudly
instead of silently degrading to a no-op the way an unmatched glob does.
Reproduced and verified by replaying the build under GitHub's directory
layout: the checkout drops to 0 test files, the preserved copy keeps all
47 and proves clean.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The 47 test files under xCAT-test/unit/ were shipped but almost never
executed on a pull request. Only three hand-written xcattest cases
reached them -- dhcp_unit, ipmi_unit and xcatprobe_unit -- and each
proved a single glob against the installed copy, so the majority of the
suite had never run at all. Real drift went unnoticed as a result:
ubuntu_subiquity_template.t still asserted the pre-86e77bcd7 shape of
compute.subiquity.tmpl and failed against the current template.
Run `prove xCAT-test/unit/*.t` directly from github_action_xcat_test.pl.
The tests resolve xCAT modules and fixture files relative to the repo
root through FindBin, so they must be proved from the checkout and not
from /opt/xcat/share/xcat/tools/autotest/unit; install_xcat() chdir's
away, hence the getcwd() captured up front. The step runs after the
install because the suite needs the perl dependencies xCAT pulls in
(Net::DNS, XML::Simple) and a usable xCAT database.
Drop the three prove testcases so their tests do not run twice, and
refresh the two stale ubuntu_subiquity_template.t assertions: the
identity section is now intentional (86e77bcd7) and the MAC
normalization gained cut filters ahead of the tr (c6e38483f), which the
loosened regex plus a new assertion for the suffix stripping now cover.
Signed-off-by: Daniel Hilst <392820+dhilst@users.noreply.github.com>
The xcat-dep APT repo now publishes only the noble (24.04) codename at
.../apt/latest/xcat-dep; the bionic/devel dist no longer exists, so
install_xcat failed on apt-get update/install. Run the workflow on
ubuntu-24.04 and point the apt source at latest/xcat-dep noble main.
Drop the apt-key step: apt-key is removed on Ubuntu 24.04 and the legacy
apt.key is gone from the xcat-dep dir; the repo already installs via
allow-insecure-repositories/allow-unauthenticated.
Daniel Hilst
Vinícius Ferrão
Mark Gurevich