The scenario for inducing this is as follows:
1. Start a container with a long stop timeout and a PID1 that
ignores SIGTERM
2. Use `podman stop` to stop that container
3. Simultaneously, in another terminal, kill -9 `pidof podman`
(the container is now in ContainerStateStopping)
4. Now kill that container's Conmon with SIGKILL.
5. No commands are able to move the container from Stopping to
Stopped now.
The cause is a logic bug in our exit-file handling logic. Conmon
being dead without an exit file causes no change to the state.
Add handling for this case that tries to clean up, including
stopping the container if it still seems to be running.
Fixes#19629
Addresses: https://issues.redhat.com/browse/ACCELFIX-250
Signed-off-by: Matt Heon <mheon@redhat.com>
Signed-off-by: tomsweeneyredhat <tsweeney@redhat.com>
Add a wait_for_ready() to one kube-play test, to make sure
container output has made it to the journal.
Probably does not fix#18501, but I think it might fix its
most common presentation.
Signed-off-by: Ed Santiago <santiago@redhat.com>
The long term goal was to provide the customer with a way to turn on the
preexec_hooks processing of scripts by having some kind of configuration
that could be read. I had tried putting it into containers.conf to
start, but that turned out to be unyieldly quickly, and time is of
the essence for this fix. That is mostly due to the fact that this
code is preexecution and in C, the containers.conf file is read in
Go much further down the stack.
After first trying this process using an ENVVAR, I have
thought it over and chatted with others, and will now look for a
/etc/containers/podman_preexec_hooks.txt file to exist. If the admin
had put one in there, we will then process the files in the
directories /usr/libexec/podman/pre-exec-hooks
and /etc/containers/pre-exec-hooks.
Thoughts/suggestions gratefully accepted. This will be a 8.8/9.2 ZeroDay
fix and will need to be backported to the v4.4.1-rhel branch.
Signed-off-by: tomsweeneyredhat <tsweeney@redhat.com>
When a userns is set we setup the network after the bind mounts, at the
point where resolv.conf is generated we do not yet know the subnet.
Just like the other dns servers for bridge networks we need to add the
ip later in completeNetworkSetup()
Addresses: https://bugzilla.redhat.com/show_bug.cgi?id=2182492 and
https://bugzilla.redhat.com/show_bug.cgi?id=2182491
This is targeted to RHEL 8.8 and 9.2 ZeroDay
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Signed-off-by: tomsweeneyredhat <tsweeney@redhat.com>
skip in setup() if journald unavailable.
To be pedantic, this is overkill: some quadlet tests pass
because they don't run journald. Too bad.
Also skip a play-kube test that requires journal
Signed-off-by: Ed Santiago <santiago@redhat.com>
As described in #17777, the `restart` on-failure action did not behave
correctly when the health check is being run by a transient systemd
unit. It ran just fine when being executed outside such a unit, for
instance, manually or, as done in the system tests, in a scripted
fashion.
There were two issue causing the `restart` on-failure action to
misbehave:
1) The transient systemd units used the default `KillMode=cgroup` which
will nuke all processes in the specific cgroup including the recently
restarted container/conmon once the main `podman healthcheck run`
process exits.
2) Podman attempted to remove the transient systemd unit and timer
during restart. That is perfectly fine when manually restarting the
container but not when the restart itself is being executed inside
such a transient unit. Ultimately, Podman tried to shoot itself in
the foot.
Fix both issues by moving the restart logic in the cleanup process.
Instead of restarting the container, the `healthcheck run` will just
stop the container and the cleanup process will restart the container
once it has turned unhealthy.
Backport of commit 9563415430.
Fixes: #17777
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2180125
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2180126
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Only enforce the passthrough log driver for Quadlet. Commit 68fbebf
introduced a regression on the `podman-kube@` template as `podman logs`
stopped working and settings from containers.conf were ignored.
Fixes: #17482
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Primarily, add skip_if_journald_unavailable because RHEL.
Secondarily, reverse a flipped actual/expect assertion
that made it difficult to understand the RHEL failure.
Signed-off-by: Ed Santiago <santiago@redhat.com>
Commit 2f29639bd3 added a UX improvement to cleanup/teardown when
running the specified YAML has failed. However, the teardown happens
unconditionally such that rerunning the same YAML file will teardown the
previously created workload instead of just failing with a name-conflict
error (e.g., "pod already exists"). The regression popped up testing
the Ansible system role with Podman v4.4.0.
For now, do not teardown at all on error to quickly fix this regression
for the upcoming Podman v4.4.1 release. The UX improvement is still
desired but must be conditional and only happen on newly created
resources, which probably requires moving it down to the backend.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Reserved annotations are used internally by Podman and would effect
nothing when run with Kubernetes so we should not be generating these
annotations.
Fixes: https://github.com/containers/podman/issues/17105
Signed-off-by: Daniel J Walsh <dwalsh@redhat.com>
...and instrument with 'podman ps'es for debugging failures.
Test flakes pretty regularly in Fedora gating. If the increased
timeout doesn't help, at least we should be able to see if the
container is stopping or failed or something.
Signed-off-by: Ed Santiago <santiago@redhat.com>
to not give a false sense of security since these are not a security
mechanism but a hook to run arbitrary code before executing a
command.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Until Podman v4.3, privileged rootfull containers would expose all the
host devices to the container while rootless ones would exclude
`/dev/ptmx` and `/dev/tty*`.
When 5a2405ae1b ("Don't mount /dev/tty* inside privileged containers
running systemd") landed, rootfull containers started excluding all the
`/dev/tty*` devices when the container would be running in systemd
mode, reducing the disparity between rootless and rootfull containers
when running in this mode.
However, this commit regressed some legitimate use cases: exposing
non-virtual-terminal tty devices (modems, arduinos, serial
consoles, ...) to the container, and the regression was addressed in
f4c81b0aa5 ("Only prevent VTs to be mounted inside privileged
systemd containers").
This now calls into question why all tty devices were historically
prevented from being shared to the rootless non-privileged containers.
A look at the podman git history reveals that the code was introduced
as part of ba430bfe5e ("podman v2 remove bloat v2"), and obviously
was copy-pasted from some other code I couldn't find.
In any case, we can easily guess that this check was put for the same
reason 5a2405ae1b was introduced: to prevent breaking the host
environment's consoles. This also means that excluding *all* tty
devices is overbearing, and should instead be limited to just virtual
terminals like we do on the rootfull path.
This is what this commit does, thus making the rootless codepath behave
like the rootfull one when in systemd mode.
This leaves `/dev/ptmx` as the main difference between the two
codepath. Based on the blog post from the then-runC maintainer[1] and
this Red Hat bug[2], I believe that this is intentional and a needed
difference for the rootless path.
Closes: #16925
Suggested-by: Fabian Holler <mail@fholler.de>
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
[1]: https://www.cyphar.com/blog/post/20160627-rootless-containers-with-runc
[2]: https://bugzilla.redhat.com/show_bug.cgi?id=501718
Increase the loop range from 5 to 20 to make sure we give the service
enough time to transition to inactive. Other tests have the same range
with 0.5 seconds sleeps, so I expect the new value to be sufficient and
consistent.
Fixes: #17093
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Make sure that the specs of containers generated by `kube play` are
correctly completed. They have not before which surfaced in default
environment variables not being set.
Fixes: #17016
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
The kube-play test file was a rat's nest of long complicated
yaml strings all differing only slightly. Clean it up, by
adding a helper function with optional parameters. The
helper is ugly, but the actual test code (the important
stuff) is cleaner.
Signed-off-by: Ed Santiago <santiago@redhat.com>
While manually playing with --service-container, I encountered a number
of too verbose logs. For instance, there's no need to error-log when
the service-container has already been stopped.
For testing, add a new kube test with a multi-pod YAML which will
implicitly show that #17024 is now working.
Fixes: #17024
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
While mounting virtual console devices in a systemd container is a
recipe for disaster (I experienced it first hand), mounting serial
console devices, modems, and others should still be done by default
for privileged systemd-based containers.
v2, addressing the review from @fho:
- use backticks in the regular expression to remove backslashes
- pre-compile the regex at the package level
- drop IsVirtualTerminalDevice (not needed for a one-liner)
v3, addressing the review from @fho and @rhatdan:
- re-introduce a private function for matching the device names
- use path.Match rather than a regex not to slow down startup time
Closes#16925.
Fixes: 5a2405ae1b ("Don't mount /dev/tty* inside privileged...")
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
If the run errors, cobra does not execute post runs. It is a somehow
known issue (https://github.com/spf13/cobra/issues/914) but problematic
for Podmand as the runtime is shutdown during post run.
Since some commands overwrite the post run and a general lack in cobra
of post runs on errors, move the shutting down the engines directly into
Execute. Fixing the issue may fix a number of flakes.
Note that the shutdowns are NOPs for the remote client.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
follow-up to 6886e80b45
when "podman -rm -f" is used on a container in "stopping" state, also
make sure it is terminated before removing it from the local storage.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
if /sys is bind mounted from the host then also add an explicit mount
for /sys/fs/cgroup so that 'ro' is honored.
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
Do not allow for removing the service container unless all associated
pods have been removed. Previously, the service container could be
removed when all pods have exited which can lead to a number of issues.
Now, the service container is treated like an infra container and can
only be removed along with the pods.
Also make sure that a pod is unlinked from the service container once
it's being removed.
Fixes: #16964
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Red Hat registry is too unreliable. (As of this writing
in January 2023, quay.io is not much better, but this is
a new flake. Ubi has been flaking for a year or more).
Instead of UBI, use the new systemd-image added to system tests
in #16814. Since this reduces the number of cached images,
a few unrelated tests (image count) need to be tweaked.
And, sigh, Fedora systemd colorizes boot messages by default,
causing a failure where we don't see an expected Reached Target
message. I don't want to rely on ASCII formatting codes, so
I've updated the build-systemd-image script so it disables
systemd colors, and have built a new systemd-image:20230106.
Made a few small usability improvements to the script as well.
Closes: #16695
Signed-off-by: Ed Santiago <santiago@redhat.com>
When you use podman logs with --until and --follow it should exit after
the requested until time and not keep hanging forever.
This fixes the behavior for the k8s-file backend.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
When you use podman logs with --until and --follow it should exit after
the requested until time and not keep hanging forever.
To make this work I reworked the code to use the better journald event
reading code for logs as well. this correctly uses the sd_journal API
without having to compare the cursors to find the EOF.
The same problems exists for the k8s-file driver, I will fix this in the
next commit.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
The `containerCouldBeLogging` bool should not be false by default, when
--since is used we seek in the journal and can miss the start event so
that bool would stay false forever. This means that a running container
is not followed even when it should.
To fix this we can just set the `containerCouldBeLogging` bool based on
the current contianer state.
Fixes#16950
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
do not allow removing containers that are in the stopping state,
otherwise it can lead to a race condition where a "podman rm" removes
the container from the storage while another process is stopping the
same container.
Closes: https://bugzilla.redhat.com/show_bug.cgi?id=2155828
Signed-off-by: Giuseppe Scrivano <gscrivan@redhat.com>
