For tests run in parallel, show file number as |nnn| (vs [nnn])
Teach logformatter to distinguish the two, adding 'p' to anchors
in parallel tests. Necessary because in this scheme we run bats
twice, thus see 'ok 1' twice, and we want to differentiate them.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Plus, I think my ampersand-quot change earlier this month
caused problems for firefox. We no longer need it (pull-option
does not need the funky double-quoted curly-brace string),
so, remove it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Run root e2e & system tests using composefs on rawhide.
Write magic settings to storage.conf. That part is easy.
e2e tests, however, ignore storage.conf. They require everything
to be specified on the command line. And "everything", in the
case of composefs, includes a long complicated --pull-options
string which in turn requires containers-storage PR 1966
which, as of this writing, is finally vendored into podman.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Two enormous misunderstandings:
1) $CIRRUS_BASE_SHA is worthless. I thought it was, you know,
the BASE SHA of the current commit, but (as best I can tell)
it seems to be the SHA of the most recent commit on the
destination branch. Cirrus docs are unhelpful. Anyhow,
it's clearly not anything useful. Stop using it.
2) $EPOCH_TEST_COMMIT is closer to what we want. It is
defined in Makefile as the git merge-base. But for unknown
reasons it was being clobbered in CI scripts, and it
doesn't seem to work in all contexts, so, eliminate it
from CI setup scripts. Leave it only in Makefile.
This leaves us with no option other than defining our own
merge-base variable, PR_BASE_SHA. Do so and pass it along
to rootless jobs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Lots of small special-case tweaks to logformatter because Macs
have to be different.
Also fix:
- Wrong slash in printf-newline, leading to gray [It] blocks
- echo gitCommit, so we can link to sources
- --image-path is deprecated
Signed-off-by: Ed Santiago <santiago@redhat.com>
When an e2e test fails in AfterEach, ginkgo "helpfully" adds
a heading in that test log block:
TOP-LEVEL [AfterEach]
/path/to/source.go
Podman Desc Blah
That TOP-LEVEL line screws up our in-page links. Ignore it
both in the heading and in the bottom failure-summary lines.
Add test, including tests for bottom-summary.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Ginkgo test names can have more than two levels: there can be
a nested series of Describes() before the final It(). (e.g.,
quadlet_test.go). Handle that.
Before: we just assumed that the third-or-maybe-fourth line
after a "-----" divider was the test name.
Now: examine every line after the "-----" divider, until the
first empty line. Lines with /path/to/source/file are ignored,
lines with text strings are assembled together to make anchors.
This is still imperfect but it's much better than before.
SPECIAL NOTE: in order to allow linking to timing results
in the AfterSuite, I've changed the test name from Leaf to Full.
This will now be a much longer string, and hence much less
readable, but I'm inclined to think it's more correct. Please
review carefully and lmk if I should revert.
Finally, as an unrelated add-on, add links (at top) to original
log, journal, and (if applicable) podman-remote server logs.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Fix an oops in my ginkgo-v2 handling: subtest-status was
getting re-reset back to "passed", resulting in incorrect
and misleading green titles on failed tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
In run_podman(), display a nanosecond-level timestamp next to
each command and its output.
Because this clutters the results, teach logformatter to grok
these new timestamps, strip them, and display a more human-readable
time delta in the left-hand timestamp column. logformatter started off
as a mess and is now, well, 🤮. I'm sorry. I just hope its results
make it worthwhile.
Signed-off-by: Ed Santiago <santiago@redhat.com>
With -p, ginkgo emits timing information on the first line
after dashes. Without -p, it's at the end.
Most tests use -p. Podman-machine does not. Deal with that.
Signed-off-by: Ed Santiago <santiago@redhat.com>
It looks like tell/seek don't work in CI-land: important test log
information is being lost. Revert this commit, maybe some day I'll
come up with a better solution.
This reverts commit 1bff0108f6.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Super-complicated and ugly code merely to make in-page anchors
link three lines above the subtest name, to the subtest status
and timing line.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Checkpoint as of Thursday April 13 afternoon.
Please do not merge with this; the .t file (tests) needs to
be updated, and there are probably many more fixes/improvements
I still need to make.
Signed-off-by: Ed Santiago <santiago@redhat.com>
* There's a new --db-backend option. Hide it, and actually
just assume that any new --foo-backend option will
have an arg that needs to be hidden.
* Friendly-ize "Running: quadlet ..." because those messages
include a "with QUADLET_FOO=bar" string that looks like
it's part of the command line but is not, and it's really
hard for a human eye to see what's command line and what's
extra comment.
Add tests.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This is to help me in my flake analysis.
Scenario: I'm looking at recent flakes, and see Flake X. I know
that Flake X was "fixed" two weeks ago. Did the fix not work?
Or is this just one of those PRs that was created three weeks
ago (or three months ago) and never rebased to pick up the fix?
This adds a one-click way for me to tell.
Signed-off-by: Ed Santiago <santiago@redhat.com>
add a new failblock style, with light red background all across
the entire page, and use it for bats "FAIL" blocks.
Also highlight logrus level=(debug|info|warning|error|fatal)
messages in increasingly prominent styles
Signed-off-by: Ed Santiago <santiago@redhat.com>
Reason: task IDs are unique and permanent; linking by
build ID and task name is non-unique, because Re-run.
Fixes: #14863
Signed-off-by: Ed Santiago <santiago@redhat.com>
One day we may use AWS for part of CI. Do you want to maintain
two separate code paths in this script for linking to artifacts
in multiple cloud providers? Can you say no? I knew you could.
Cirrus already knows the location of the artifacts and provides
a transparent mechanism for accessing them. Use it.
This PR exposed a nasty bug in our environment-variable handling:
envariables passed through to the containerized environment were
being double-space-escaped, so "FOO=a b" ended up as "FOO=a\ b"
(with a backslash), with one consequence being invalid URLs.
The solution is simple: run 'podman -e FOO', not '-e FOO=value'.
Finally, reinstate the environment-variable dump (in comments).
I had removed this in a moment of panic over leaking secrets,
but no, that doesn't happen. Exclude scary-sounding vars anyway.
Signed-off-by: Ed Santiago <santiago@redhat.com>
This reverts commit cc3790f332.
We can't change rootful to rootfull because `rootful` is written into the machine config. Changing this will break json unmarshalling, which will break existing machines.
[NO NEW TESTS NEEDED]
Signed-off-by: Ashley Cui <acui@redhat.com>
The default verbosity level does not show the classes or function names.
This makes it difficult to debug problems like hangs. Also, separate
the bats and python-based tests into two sections. This allows for
easier debugging, since isolation can be done in `runner.sh` rather than
mucking with the `Makefile`. Lastly, update the logformatter script
to `autoflush stdout` (thanks @edsantiago).
Signed-off-by: Chris Evich <cevich@redhat.com>
We are inconsistent on the name, we should stick with rootfull.
[NO NEW TESTS NEEDED] Existing tests should handle this and no tests for
machines exists yet.
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
ginkgo netavark logs (and, to a lesser extent, cni logs)
are unreadable because the hide-boring-opts code did not
know about --network-backend. Now it does.
Manually filtered an existing netavark log to confirm there
are no other new options we should know about.
Signed-off-by: Ed Santiago <santiago@redhat.com>
We already link to ginkgo sources, now add links to bats.
Ugly, because we need to hardcode containers/podman (git
repo) and test/system (test file path): those can't be
determined from the log results like they can in ginkgo.
Also, great suggestion from @Luap99: in addition to the
'Annotated results' link which we append to the basic log,
include a short summary of failures. This should help a
viewer see exactly which test(s) failed, which in turn
can be helpful for diagnosing known-flake or real-problem.
Signed-off-by: Ed Santiago <santiago@redhat.com>
We've got some python tests running in CI, and they're really hard
to troubleshoot. This PR:
1) colorizes python unittest lines (ok / skipped / fail), and
2) links to source files
The color is nice for skimming, but it's the linking that might
make it much easier to diagnose future failures.
(Context: failure today in test/python/docker/compat/test_images.py)
Signed-off-by: Ed Santiago <santiago@redhat.com>
Since this option will also be used for netavark we should rename it to
something more generic. It is important that --cni-config-dir still
works otherwise we could break existing container cleanup commands.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
A test name beginning with non-alpha, e.g., "--build should ...",
was not being recognized and linkified:
https://storage.googleapis.com/cirrus-ci-6707778565701632-fcae48/artifacts/containers/podman/6500723916537856/html/int-podman-fedora-34-rootless-host.log.html
Fix that. Also fix two other cases (single/double quotes) that were
resulting in weird unreliable links.
While I'm at it, add a few usability enhancements:
* Colorize [SKIPPING] and [SLOW TEST]
* Deemphasize '[It] testname' when it appears mid-test
* Replace 'Running:' with a (deemphasized) '#' or '$' prompt
Add regression tests
Signed-off-by: Ed Santiago <santiago@redhat.com>
docker-compose test continues to flake even after #9961.
Let's try to get some useful data from the failures, by:
* adding -S (--show-error) to curl. With just -s (--silent),
curl is completely quiet. With -S, it displays errors.
(Not in TAP form, but I'm OK with that)
* oops, adding safety checks to the fix from #9961 (it
was inadvertently clobbering the curl exit status)
And, as long as I'm in this code: logformatter was not
highlighting these results, because the '1..N' TAP line
needs to be spit out at the end. Have test-compose emit
a 'TAP' header <http://testanything.org/> and make
logformatter recognize it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Add a :hover style to rows in the 'integration timing results'
section. Without that, it's really hard for my eye to scan
across and match a time to a test name.
Signed-off-by: Ed Santiago <santiago@redhat.com>
(that got accidentally dropped in the new Cirrus makeover).
Note that 'dotest' does not actually 'do tests', it's
only used for a small subset of tests.
Also, make logformatter work better in the new Cirrus setup.
Remove duplicate test/subtest, remove no-longer-used SPECIALMODE,
and make the Cirrus build/task display a little cleaner.
Signed-off-by: Ed Santiago <santiago@redhat.com>
At the top of each generated page, add a Synopsis table with:
PR number/name, and link to github
Author name(s)
Test name (fedora/ubuntu, rootless, etc)
Cirrus build ID (usually uninteresting)
Cirrus task ID (usu. important), with link to Cirrus
The value of $SPECIALMODE
This is all we can get from the Cirrus environment in
which logformatter runs; we can't get things like
cgroup manager or username that the test runs under.
Note that the table is at the top, which is usually
unseen because we autoscroll to the bottom on
page load. I tentatively think that top is a more
natural place for this info than bottom, but am
willing to listen to arguments against.
Also, one minor tweak: highlight podman commands in
the BATS output. The idea is to make it easier for the eye
to spot those, then copy/paste them to find a reproducer.
And, sigh, disable the new 'podman network create'
system test. It is flaking much too much.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Problem: formatted logs no longer have live links to sources
in error-report lines.
Cause: script was searching for '/libpod'.
Solution: make it more flexible.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Oops! Logs of podman-remote tests are unreadable, they have
multiple (useless) --remote options plus '--url /something/long'
that makes it impossible to read the actual command being run.
This commit strips off '--remote' entirely, and hides '--url'
and its arg in the only-on-mouse-hover '[options]' text.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Fallout from libpod->podman repo name move: the HTML logs
created by logformatter are no longer accessible. They
render as:
https://storage.googleapis.com/SECRET-5385732420009984-fcae48/artifacts/containers/podman/6313596734930944/html/integration_test.log.html
(yes, "SECRET" instead of "cirrus-ci". Possibly because
the GCE_SSH_USERNAME key, "cirrus-ci", was overzealously
encrypted, making Cirrus censor any instances of the
string in output. Let's see if this fixes it. But anyway
this is a secondary unrelated bug).
Reason: it looks like Cirrus "generated a new magic blob"
when we renamed libpod -> podman. Chris was kind enough to
locate the new magic blob and to give me a link to where
we can discover it ourselves. I added that as a code comment.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Reversion of one part of #6679: my handling of 'realpath'
would not work when $PODMAN is 'podman-remote --url etc'.
Trying to handle that case got unmaintainable; so instead
let's just force 'make {local,remote}system' to invoke
with a full PODMAN path. This breaks down if someone
runs the tests with a manual 'bats' invocation, but I
think I'm the only one who ever does that.
Since podman path will now be very long in the logs,
add code to logformatter to abbreviate it like we do
for the ginkgo logs.
And, one thing that has bugged me for a long time:
in the error logs, show a different prompt ('#' vs '$')
to distinguish root vs rootless. This should make it
much easier to see at-a-glance whether a log file
is root or not. Add tests for it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
BATS emits a summary line (number of tests passed/failed)...
but only on a tty or when run with --pretty! In our CI
context, with TAP output, it gives no end summary.
Fix that. Keep track of 'ok', 'not ok', and 'skipped',
and display the counts at the end.
Also: add a regression test. You don't need to review
or even read it: it's stark, and I'm not even enabling
it for CI because it almost certainly won't run due to
missing Perl library modules. It's just something I
need on my end.
Signed-off-by: Ed Santiago <santiago@redhat.com>
apiv2 tests emit TAP-compliant output; recognize it and
highlight it the same way we do BATS tests.
Add anchor links to TAP output, so other tools (e.g.
cirrus-flake-summarize) can link to particular lines
And, remove a "-f" from "wait" in test-apiv2; looks
like there's some version of bash used in some CI VM
that doesn't grok it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
1) 'podman system info' (in logcollector): has been silently
failing in special_testing_rootless, with:
logcollector.sh: line 16: podman: command not found
Use ./bin/podman instead of just podman; this is probably
the right thing to do in the general case anyway
2) logformatter: highlight 'panic:', seen in bindings test:
https://storage.googleapis.com/cirrus-ci-5385732420009984-fcae48/artifacts/containers/libpod/6693715108429824/html/integration_test.log.html
3) logformatter: handle Unicode bullet in front of 'Running',
seen in bindings test.
4) logformatter: turn down contrast on BATS 'ok' results,
for legibility
Signed-off-by: Ed Santiago <santiago@redhat.com>
This introduces a new cirrus helper script, logformatter.
Usage is:
[commands...] | logformatter TEST-NAME
It reformats its input into a readable, highlighed, linkable
form. Some features:
- boring stuff (timestamps, standard podman options) is
deemphasized
- important stuff (warnings, errors) is emphasized
- in-page links to the actual failures
- active links to source files
- jumps to bottom of page on load, because that's where
the errors are. (All errors are linked)
Add it to select test commands (integration, system) and
add a new artifacts_html, run in the 'always' block, which
uploads generated *.log.html into Cirrus; from there we
generate a live URL that can be viewed in browser.
Unfortunately, due to security concerns in Cirrus, it is
not currently possible to make the link a live one.
Kludge: add a line of dashes after Restoring images; without this,
the first test ("systemd PID 1") has no dashes before it, so
logformatter doesn't see it.
Signed-off-by: Ed Santiago <santiago@redhat.com>
