Until now the default could only be expressed by leaving the option off, which
reads as an omission rather than a choice and gives a script no way to say it
wants the archive transferred as podman save wrote it.
Accept none as a format meaning exactly that. It is taken on the API path too,
so both interfaces share one vocabulary, and it is treated as the absence of a
format throughout: nothing is compressed, a level attached to it is rejected the
same way a level with no format is, and the local user to user transfer has
nothing to warn about ignoring.
The remote client still leaves it off the request, so naming the default does
not make a transfer fail against a service that predates these options.
Signed-off-by: Scott Callaway <github@scottcallaway.co.uk>
When the source is a remote host the archive is produced there, so it has to be
compressed there too: compressing after copying it down would mean the
uncompressed archive had already crossed the network, which is the cost this is
meant to avoid.
The only thing we can do on that host is run a command, so the matching
compressor is invoked over ssh between the save and the copy. That is also why
the set of formats is limited to algorithms available as a command of the same
name.
Two details worth stating. A shell reports 127 when it cannot find the command,
which is worth reporting plainly as a host without the compressor installed;
anything else, a failure to connect included, must not be reported that way,
and a probe beforehand cannot make that distinction without also costing an
extra connection. And the compressor removes its input only once it succeeds
and may have written part of its output before giving up, so a failure cleans
up both paths.
Signed-off-by: Scott Callaway <github@scottcallaway.co.uk>
podman save writes docker-archive layers uncompressed, so podman image scp puts
the whole archive on the wire as is. Compressing it first takes a docker-archive
to around half its size or less, which is why people work around this today with
podman save | zstd, a manual copy and podman load on the far side. An
oci-archive keeps whatever compression its layers already have, so there is
little to gain there; the man page records the difference.
When the archive is produced locally it can be compressed on the way out: the
c/image compression package wraps the file as it is streamed into the ssh
connection feeding the remote podman image load. No second temporary file,
nothing buffered in full.
Nothing is needed on the destination. podman load detects the compression from
the stream and decompresses it itself, for docker-archive via c/image's
AutoDecompress and for oci-archive via c/storage's DecompressStream. Both are
covered, since which one runs depends on --format.
Signed-off-by: Scott Callaway <github@scottcallaway.co.uk>
Groundwork for compressing the transfer archive: the options themselves, the
set of algorithms that may be requested, and the check ExecuteTransfer runs
before it does anything else. Nothing acts on them yet.
The set of formats is deliberately narrower than what c/image knows. Every
entry has to satisfy three things: podman load has to detect and decompress it
from the stream alone, c/image has to be able to compress it (it only
decompresses bzip2 and xz), and a command line compressor of the same name has
to exist for the case where the archive is produced on a remote host. gzip and
zstd qualify, and they match the vocabulary --compression-format already uses
on podman push.
One table drives the accepted formats, their level ranges, and the list
offered on the command line, so there is nothing to keep in sync.
A transfer between two users on the same machine never crosses a network, so a
requested format is reported as ignored there rather than refused.
The validation is worded without flag names because it also runs on the API
path, where the caller never passed a flag.
Signed-off-by: Scott Callaway <github@scottcallaway.co.uk>
Three small things in one place, all groundwork rather than behaviour:
The path SaveToRemote gets back from the remote mktemp still carries the
trailing newline ssh.Exec hands over with the rest of the raw output. That is
harmless while the path is only ever the last thing on a command line, but it
is a trap for anything that appends to it. Trim it.
The host, identity, port and user were restated in full for every command.
State them once and let each command copy the value and add its own argv.
Removing a file on the far end had one caller and was about to have more, so
give it a name. It also gains -f, since a caller cleaning up after a failure
cannot know which of the paths it is removing were created.
Lastly, the ssh operations a transfer performs are gathered into one value the
exported entry points pass in. Nothing about the options changes: SaveToRemote
and LoadToRemote keep taking their options struct, and the body moves to an
unexported function taking that same struct plus the operations to run it with.
That is what lets a test assert the sequence of remote commands, and what is
streamed to them, without a host to run against.
Signed-off-by: Scott Callaway <github@scottcallaway.co.uk>
The podman module paths are moving from github.com/containers/podman to
go.podman.io/podman. This will help with future mobility.
Signed-off-by: Brent Baude <bbaude@redhat.com>
Add --format (oci-archive, docker-archive) to pass through to podman save.
Default is unchanged (no --format) so podman save uses its own default.
Document that scp is not storage-to-storage and only archive formats
are supported.
Fixes: https://github.com/containers/podman/issues/28183
Fixes: https://issues.redhat.com/browse/RUN-4403
Signed-off-by: Jan Rodák <hony.com@seznam.cz>
There is exactly one caller, with two code paths, and each only
needs _half_ of the function - and they really only share the
parentFlags and Quiet logic. It's easier to do things
directly.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
When we have the _precise_ knowledge of where the parameter boundaries
are, and an API that allows us to express that, just _do that_ instead
of completely unnecessarily worrying about spaces in parameter values.
Also, this allows us to format the code to make the option and value
correspondence much easier to see.
Signed-off-by: Miloslav Trmač <mitr@redhat.com>
Tremendous amount of changes in here, but all should amount to
the same thing: changing Go import paths from v5 to v6.
Also bumped go.mod to github.com/containers/podman/v6 and updated
version to v6.0.0-dev.
Signed-off-by: Matt Heon <mheon@redhat.com>
Using golang.org/x/tools/gopls/internal/analysis/modernize/cmd/modernize
+ some manual cleanup in libpod/lock/shm/shm_lock_test.go as it
generated an unused variable
+ restored one removed comment
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
podman artifact add now supports two new command line switches.
--type string that describes the type of artifact
--annotation string slice in the form of key=val
These new options allow users to "tag" information in on their artifacts
for any number of purposes down the line
RUN-2446
Signed-off-by: Brent Baude <bbaude@redhat.com>
Prior to this commit, many scp functions existed without option structs, which would make extending functionality (adding new options) impossible without breaking changes, or without adding redundant wrapper functions.
This commit adds in new option types for various scp related functions, and changes those functions' signatures to use the new options.
This commit also modifies the `ImageEngine.Scp()` function's interface to use the new opts.
The commit also renames the existing `ImageScpOptions` entity type to `ScpTransferImageOptions`. This is because the previous `ImageScpOptions` was inaccurate, as it is not the actual options for `ImageEngine.Scp()`. `ImageEngine.Scp()` should instead receive `ImageScpOptions`.
This commit should not change any behavior, however it will break the existing functions' signatures.
Signed-off-by: Zachary Hanham <z.hanham00@gmail.com>
The default location for temporary files created by mktemp may not
have enough space for an image. Use the new SSH functions which
support an input reader to make the code simpler, more reliable,
and more efficient.
[NO NEW TESTS NEEDED]
Signed-off-by: Gordon Messmer <gordon.messmer@gmail.com>
SSH uses 22 as default so it is really not necessary to require the
port. The backend code already does this but the parsing in the
frontend always tried to parse the port.
[NO NEW TESTS NEEDED] This would require actual remote host ssh setup in
CI so it is not possible to be check but I verified it locally.
Fixes https://issues.redhat.com/browse/RHEL-17776
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Moving from Go module v4 to v5 prepares us for public releases.
Move done using gomove [1] as with the v3 and v4 moves.
[1] https://github.com/KSubedi/gomove
Signed-off-by: Matt Heon <mheon@redhat.com>
We now no longer write containers.conf, instead system connections and
farms are written to a new file called podman-connections.conf.
This is a major rework and I had to change a lot of things to get this
to compile again with my c/common changes.
It is a breaking change for users as connections/farms added before this
commit can now no longer be removed or modified directly. However because
the logic keeps reading from containers.conf the old connections can
still be used to connect to a remote host.
Signed-off-by: Paul Holzinger <pholzing@redhat.com>
Cut is a cleaner & more performant api relative to SplitN(_, _, 2) added in go 1.18
Previously applied this refactoring to buildah:
https://github.com/containers/buildah/pull/5239
Signed-off-by: Philip Dubé <philip@peerdb.io>
Support a new concept in containers.conf called "modules". A "module"
is a containers.conf file located at a specific directory. More than
one module can be loaded in the specified order, following existing
override semantics.
There are three directories to load modules from:
- $CONFIG_HOME/containers/containers.conf.modules
- /etc/containers/containers.conf.modules
- /usr/share/containers/containers.conf.modules
With CONFIG_HOME pointing to $HOME/.config or, if set, $XDG_CONFIG_HOME.
Absolute paths will be loaded as is, relative paths will be resolved
relative to the three directories above allowing for admin configs
(/etc/) to override system configs (/usr/share/) and user configs
($CONFIG_HOME) to override admin configs.
Pulls in containers/common/pull/1599.
Signed-off-by: Valentin Rothberg <vrothberg@redhat.com>
Package `io/ioutil` was deprecated in golang 1.16, preventing podman from
building under Fedora 37. Fortunately, functionality identical
replacements are provided by the packages `io` and `os`. Replace all
usage of all `io/ioutil` symbols with appropriate substitutions
according to the golang docs.
Signed-off-by: Chris Evich <cevich@redhat.com>
implement new ssh interface into podman
this completely redesigns the entire functionality of podman image scp,
podman system connection add, and podman --remote. All references to golang.org/x/crypto/ssh
have been moved to common as have native ssh/scp execs and the new usage of the sftp package.
this PR adds a global flag, --ssh to podman which has two valid inputs `golang` and `native` where golang is the default.
Users should not notice any difference in their everyday workflows if they continue using the golang option. UNLESS they have been using an improperly verified ssh key, this will now fail. This is because podman was incorrectly using the
ssh callback method to IGNORE the ssh known hosts file which is very insecure and golang tells you not yo use this in production.
The native paths allows for immense flexibility, with a new containers.conf field `SSH_CONFIG` that specifies a specific ssh config file to be used in all operations. Else the users ~/.ssh/config file will be used.
podman --remote currently only uses the golang path, given its deep interconnection with dialing multiple clients and urls.
My goal after this PR is to go back and abstract the idea of podman --remote from golang's dialed clients, as it should not be so intrinsically connected. Overall, this is a v1 of a long process of offering native ssh, and one that covers some good ground with podman system connection add and podman image scp.
Signed-off-by: Charlie Doern <cdoern@redhat.com>
We now use the golang error wrapping format specifier `%w` instead of
the deprecated github.com/pkg/errors package.
[NO NEW TESTS NEEDED]
Signed-off-by: Sascha Grunert <sgrunert@redhat.com>
add support for podman-remote image scp as well as direct access via the API. This entailed
a full rework of the layering of image scp functions as well as the usual API plugging and type creation
also, implemented podman image scp tagging. which makes the syntax much more readable and allows users t tag the new image
they are loading to the local/remote machine:
allow users to pass a "new name" for the image they are transferring
`podman tag` as implemented creates a new image im `image list` when tagging, so this does the same
meaning that when transferring images with tags, podman on the remote machine/user will load two images
ex: `podman image scp computer1::alpine computer2::foobar` creates alpine:latest and localhost/foobar on the remote host
implementing tags means removal of the flexible syntax. In the currently released podman image scp, the user can either specify
`podman image scp source::img dest::` or `podman image scp dest:: source::img`. However, with tags this task becomes really hard to check
which is the image (src) and which is the new tag (dst). Removal of that streamlines the arg parsing process
Signed-off-by: Charlie Doern <cdoern@redhat.com>
Filtering is missing in both compat API and libpod API, while docker
has filtering functinality. This commit enables filtering option using
name and id in both libpod and http API.
Signed-off-by: Jakub Guzik <jakubmguzik@gmail.com>
