The step summary posted when a CI job fails is useless for the bindings
suite: it carries a bare failure count and never the failure itself. Two
reasons stack up.
github_log_summary.py picks between the ginkgo and bats parsers by
looking for "int-" in the file name, but int is not the only ginkgo
suite - bindings runs ginkgo as well, so its log is handed to the bats
parser, which finds nothing beyond the count in ginkgo's summary line.
logformatter decides the layout from the log contents, so do the same
here.
That alone still leaves the summary empty. Only ginkgo run under -p
reports a failure on the line logformatter reads the test status from,
and `make testbindings` does not pass -p, so every test in a bindings log
heads 'log-passed' and the stack trace div is the sole mark of a failure.
Key on that div as well. Not on the class by itself: logformatter gives
it to error-level logrus lines too, which passing tests emit, but those
are spans.
Three more things the script got wrong along the way, all of them kept
out of sight by the "|| true" on the workflow step. The workflow passes
it a glob but only argv[1] was ever read, so any further log was dropped
without a word. A job that produces no html log at all leaves that glob
unexpanded, which ended in a traceback. And logformatter reads its input
through ':utf8', which validates nothing, so a test that wrote raw bytes
leaves behind a log that the summary could not even decode.
Signed-off-by: rjgoyln <pt40419@gmail.com>
Adapt the logformatter script to run again to produce nicely formatted
html logs. Because github cannot upload and show raw html files we can
only add it to the log archive which must be downloaded and viewed
locally.
To improve the online experience however github does have the
GITHUB_STEP_SUMMARY logic which allows us to produce markdown which will
be shown on the summary page for a given test run.
The problem is even though github markdown supports some html is does
not support CSS and out custom style of the logformatter, in addition
the output size is limited so I wrote another script parsing the html
output again and turning it into plain test for only the failed tests
and then show this as plain text inside codeblocks in the markdown.
With this we have a short failure summary which should display all
failures at once in the run page so maintainers can see if the failed
log was just some flake or an actual problem with the PR.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
