spiegel_podman/libpod
Paul Holzinger d6ae7427f7 fix network cleanup flake in play kube
When using service containers and play kube we create a complicated set
of dependencies.

First in a pod all conmon/container cgroups are part of one slice, that
slice will be removed when the entire pod is stopped resulting in
systemd killing all processes that were part in it.

Now the issue here is around the working of stopPodIfNeeded() and
stopIfOnlyInfraRemains(), once a container is cleaned up it will check
if the pod should be stopped depending on the pod ExitPolicy. If this is
the case it wil stop all containers in that pod. However in our flaky
test we calle podman pod kill which logically killed all containers
already. Thus the logic now thinks on cleanup it must stop the pod and
calls into pod.stopWithTimeout(). Then there we try to stop but because
all containers are already stopped it just throws errors and never gets
to the point were it would call Cleanup(). So the code does not do
cleanup and eventually calls removePodCgroup() which will cause all
conmon and other podman cleanup processes of this pod to be killed.

Thus the podman container cleanup process was likely killed while
actually trying to the the proper cleanup which leaves us in a bad
state.

Following commands such as podman pod rm will try to the cleanup again
as they see it was not completed but then fail as they are unable to
recover from the partial cleanup state.

Long term network cleanup needs to be more robust and ideally should be
idempotent to handle cases were cleanup was killed in the middle.

Fixes #21569

Signed-off-by: Paul Holzinger <pholzing@redhat.com>
2024-07-31 14:37:46 -04:00
..
define fix race conditions in start/attach logic 2024-07-12 15:11:34 +02:00
driver Bump Go module to v5 2024-02-08 09:35:39 -05:00
events Make podman update changes persistent 2024-04-17 08:23:50 -04:00
layers Initial checkin from CRI-O repo 2017-11-01 11:24:59 -04:00
linkmode chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
lock libpod: use fileutils.(Le|E)xists 2024-04-19 09:52:14 +02:00
logs Bump Go module to v5 2024-02-08 09:35:39 -05:00
plugin libpod: use fileutils.(Le|E)xists 2024-04-19 09:52:14 +02:00
shutdown libpod: shutdown Stop waits for handlers completion 2024-07-15 11:41:28 +02:00
boltdb_state.go libpod: use fileutils.(Le|E)xists 2024-04-19 09:52:14 +02:00
boltdb_state_internal.go Ignore result of EvalSymlinks on ENOENT 2024-07-11 09:39:56 -04:00
common_test.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
container_api.go fix race conditions in start/attach logic 2024-07-12 15:11:34 +02:00
container_commit.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_config.go libpod: fix comment 2024-06-21 10:07:55 +02:00
container_copy_common.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_copy_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_copy_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_exec.go Use persist dir for oom file 2024-02-12 09:13:39 -05:00
container_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_graph.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_graph_test.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_inspect.go Add support for updating restart policy 2024-04-17 08:23:51 -04:00
container_inspect_freebsd.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_inspect_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_internal.go fix race conditions in start/attach logic 2024-07-12 15:11:34 +02:00
container_internal_common.go restore: fix missing network setup 2024-06-24 18:52:02 +02:00
container_internal_freebsd.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
container_internal_linux.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
container_internal_linux_test.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_internal_test.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_linux.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_log.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_log_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_log_unsupported.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_path_resolution.go chore: fix function names in comment 2024-04-08 11:36:50 +08:00
container_path_resolution_test.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_stat_common.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_stat_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_stat_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_top_freebsd.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
container_top_linux.c libpod/container_top_linux.c: fix missing header 2024-06-27 10:50:17 +02:00
container_top_linux.go podman top: join the container userns 2024-06-26 11:10:56 +02:00
container_top_unsupported.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
container_validate.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
diff.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
doc.go document that using libpod package directly is not supported 2022-03-23 19:05:29 +01:00
events.go libpod: make healthcheck events more efficient 2024-04-19 17:24:44 +02:00
healthcheck.go libpod: correctly capture healthcheck output 2024-07-19 15:16:55 +02:00
healthcheck_linux.go libpod: do not leak systemd hc startup unit timer 2024-06-04 18:03:46 +02:00
healthcheck_nosystemd_linux.go libpod: do not leak systemd hc startup unit timer 2024-06-04 18:03:46 +02:00
healthcheck_unsupported.go libpod: do not leak systemd hc startup unit timer 2024-06-04 18:03:46 +02:00
info.go Add rootless network command to podman info 2024-03-26 09:56:48 -04:00
info_freebsd.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
info_linux.go rootless: drop function ReadMappingsProc 2024-04-10 11:55:35 +02:00
info_test.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
kube.go libpod: fix 'podman kube generate' on FreeBSD 2024-06-05 10:38:30 +01:00
mounts_linux.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
networking_common.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
networking_freebsd.go use new c/common pasta2 setup logic to fix dns 2024-03-19 12:09:31 +01:00
networking_linux.go use new c/common pasta2 setup logic to fix dns 2024-03-19 12:09:31 +01:00
networking_linux_test.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
networking_machine.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
networking_pasta_linux.go use new c/common pasta2 setup logic to fix dns 2024-03-19 12:09:31 +01:00
networking_slirp4netns.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
oci.go Use persist dir for oom file 2024-02-12 09:13:39 -05:00
oci_conmon.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
oci_conmon_attach_common.go fix race conditions in start/attach logic 2024-07-12 15:11:34 +02:00
oci_conmon_attach_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
oci_conmon_attach_linux.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
oci_conmon_common.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
oci_conmon_exec_common.go Use persist dir for oom file 2024-02-12 09:13:39 -05:00
oci_conmon_exec_freebsd.go Cease using deprecated runc userlookup 2024-02-02 11:02:43 -05:00
oci_conmon_exec_linux.go Cease using deprecated runc userlookup 2024-02-02 11:02:43 -05:00
oci_conmon_freebsd.go libpod: intermediate mount if UID not mapped into the userns 2024-06-21 18:01:26 +02:00
oci_conmon_linux.go libpod: intermediate mount if UID not mapped into the userns 2024-06-21 18:01:26 +02:00
oci_missing.go Use persist dir for oom file 2024-02-12 09:13:39 -05:00
oci_util.go libpod: bind ports before network setup 2024-07-31 14:24:07 -04:00
options.go podman: add new hidden flag --pull-option 2024-06-12 15:48:36 +02:00
pod.go podman pod stats: fix race when ctr process exits 2024-07-22 10:30:42 +02:00
pod_api.go fix network cleanup flake in play kube 2024-07-31 14:37:46 -04:00
pod_internal.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
pod_internal_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
pod_internal_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
pod_status.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
pod_top_freebsd.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
pod_top_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
reset.go libpod: cleanup default cache on system reset 2024-05-29 11:10:55 +02:00
rlimit_int64.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
rlimit_uint64.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime.go libpod: cleanup store at shutdown 2024-07-15 11:41:28 +02:00
runtime_cstorage.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_ctr.go libpod: first delete container then cidfile 2024-07-05 10:27:42 +02:00
runtime_ctr_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime_ctr_linux.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime_freebsd.go Detect unhandled reboots and require user intervention 2024-04-05 10:07:42 -04:00
runtime_img.go chore: fix function names in comment 2024-04-24 12:07:38 +08:00
runtime_linux.go Detect unhandled reboots and require user intervention 2024-04-05 10:07:42 -04:00
runtime_migrate_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_migrate_unsupported.go Remove Libpod special-init conditions 2024-01-12 09:19:34 -05:00
runtime_pod.go Replace golang.org/x/exp/slices with slices from std 2024-04-23 11:16:40 +02:00
runtime_pod_common.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_pod_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime_pod_linux.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_pre_go1.20.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime_renumber.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_test.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
runtime_volume.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
runtime_volume_common.go libpod: use fileutils.(Le|E)xists 2024-04-19 09:52:14 +02:00
runtime_worker.go create runtime's worker queue before queuing any job 2024-07-09 11:15:29 +02:00
service.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
sqlite_state.go libpod: add hidden env to set sqlite timeout 2024-07-22 12:59:00 +02:00
sqlite_state_internal.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
state.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
state_test.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
stats_common.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
stats_freebsd.go Fix some comments 2024-04-13 15:20:19 +08:00
stats_linux.go Fix some comments 2024-04-13 15:20:19 +08:00
storage.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
util.go Ignore result of EvalSymlinks on ENOENT 2024-07-11 09:39:56 -04:00
util_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
util_linux.go libpod: use fileutils.(Le|E)xists 2024-04-19 09:52:14 +02:00
util_linux_test.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
volume.go Only stop chowning volumes once they're not empty 2024-05-22 17:47:01 -04:00
volume_inspect.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
volume_internal.go Only stop chowning volumes once they're not empty 2024-05-22 17:47:01 -04:00
volume_internal_common.go Bump Go module to v5 2024-02-08 09:35:39 -05:00
volume_internal_freebsd.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00
volume_internal_linux.go chore: delete obsolete // +build lines 2024-01-04 11:53:38 +02:00