With the new dial-stdio command the dialer is split
from `Client` function in order to access it directly.
This breaks the custom connhelpers functionality
as support for connhelpers is a feature of the default
dialer. If client defines a custom dialer then only
it is used without extra modifications. This means
that remote driver dialer needs to detect the
connhelpers on its own.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 8f576e5790)
Measure the transfer size and duration of context transfers for various
categories of local source transfers from the progress stream that's
returned during the build.
Local source transfers are split into one of four categories:
* context
* dockerfile
* dockerignore
* namedcontext
Named contexts that are different names will be categorized under the
same metric.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Because of the way buildkit cache works if you have request
with external cache, if some vertices from the request have
already been evaluated and are available in the shared graph
BuildKit will not load cache keys from external source for such
vertices. This may mean that children of such vertices will
not load cache because there isn't a cache path through the parent.
To work around it, wait before child definition is loaded before
evaluating the parent.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The `BUILDX_EXPERIMENTAL` check is removed from the docker otel
collector. We'll send metrics to the OTLP endpoint for docker desktop if
it is present and enabled regardless of experimental status.
The user-facing `OTEL` endpoints for enabling the metric reporting for
external use is still hidden behind the experimental flag. We'll likely
remove the experimental flag for this feature for v0.14.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This adds a build duration metric for the build command with attributes
related to the buildx driver, the error type (if any), and which options
were used to perform the build from a subset of the options.
This also refactors some of the utility methods used by the git tool to
determine filepaths into its own separate package so they can be reused
in another place.
Also adds a test to ensure the resource is initialized correctly and
doesn't error. The otel handler logging message is suppressed on buildx
invocations so we never see the error if there's a problem with the
schema url. It's so easy to mess up the schema url when upgrading OTEL
that we need a proper test to make sure we haven't broken the
functionality.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This allows the buildx CLI to act a proxy to the configured instance.
It allows external code to use buildx itself as a driver for connecting
to buildkitd instances.
Instance and node selection should follow the same semantics as as
`buildx build`, including taking into account the `BUILDX_BUILDER` env
var and the `--builder` global flag.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Add the service instance id to the resource attributes to prevent
downstream OTEL processors and exporters from thinking that the CLI
invocations are a single process that keeps restarting. The unique id
can be removed through downstream aggregation to prevent cardinality
issues, but we need some way to tell OTEL that it shouldn't reset the
counters.
Move the check for the experimental flag to its own package and then use
that invocation to prevent creating exporters so metrics are disabled
completely. This makes it so we don't have to check for the experimental
flag in every place we add metrics until we decide to make metrics
stable in general.
This also moves the OTEL initialization to a `util/metricutil` package
to be more consistent with the existing util naming and to differentiate
it from the upstream `metric` name. Using both `metrics` and `metric` as
import names was confusing since `metric` was an upstream dependency and
`metrics` was a local utility. `metricutil` matches with the existing
utilities and makes clear that it isn't a spelling mistake.
The record version metric has been removed since we weren't planning on
keeping that metric anyway and most of the information is now included
in the instrumentation library name and version. That function is
included as a utility in the `otel/sdk/metric` package to retrieve the
appropriate meter from the meter provider.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
- release notes: https://github.com/spf13/cobra/releases/tag/v1.8.0
- full diff: https://github.com/spf13/cobra/compare/v1.7.0...v1.8.0
Release notes highlights:
Features
- Support usage as plugin for tools like kubectl - this means that programs
that utilize a "plugin-like" structure have much better support and usage
(like for completions, command paths, etc.)
- Move documentation sources to site/content
- Add 'one required flag' group - this includes a new MarkFlagsOneRequired API
for flags which can be used to mark a flag group as required and cause command
failure if at least one is not used when invoked.
- Customizable error message prefix - This adds the SetErrPrefix and ErrPrefix
APIs on the Command struct to allow for setting a custom prefix for errors
- feat: add getters for flag completions
- Feature: allow running persistent run hooks of all parents
- Improve API to get flag completion function
Bug fixes
- Fix typo in fish completions
- Fix grammar: 'allows to'
- powershell: escape variable with curly brackets
- Don't complete --help flag when flag parsing disabled
- Replace all non-alphanumerics in active help env var program prefix
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- cli-plugins: move socket code into common package
- cli-plugins: don't use abstract sockets on macOS
- fixes CLI leaving behind plugin socket mount-points
- socket: return from loop after EOF
full diff: https://github.com/docker/cli/compare/v25.0.0-rc.1...v25.0.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
See https://github.com/docker/cli/pull/4599 and
https://github.com/docker/cli/pull/4769.
Since we switched to using the cobra command context instead of
`appcontext`, we need to set up the signal handling that was being
provided by `appcontext`, as well as configuring the context with
the OTEL tracing utilities also used by `appcontext`.
This commit introduces `cobrautil.ConfigureContext` which implements
the pre-existing signal handling logic from `appcontext` and cancels
the command's context when a signal is received, as well as doing
the relevant OTEL config.
Signed-off-by: Laura Brehm <laurabrehm@hey.com>
Prior to this commit, experimental flags were not distinguishable from
regular flags in `--help`
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Introduce a meter provider to the buildx cli that will send metrics to
the otel-collector included in docker desktop if enabled.
This will send usage metrics to the desktop application but also send
metrics to a user-provided otlp receiver endpoint through the standard
environment variables.
This introduces a single metric which is the cli count for build and
bake along with the command name and a few additional attributes.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Set local dirs metadata if relative to VCS directory so
dockerfile path is tracked accurately in case vcs information
is not fulfilled.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
*client.SolveOpt in driver code is only used by build code.
For a clear separation of concerns, move it to an internal
struct type only accessible by BuildWithResultHandler func.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
This moves the builder creation logic to the builder package as we
plan to allow builder creation in the build ui of Docker Desktop so we
avoid drifting with the same logic in command package.
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Using this command resolves remote based on remote
tracking branch of the curently selected branch and
should be more precise in case we can't predict if user
uses origin to mark upstream or their fork.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
This regression was introduced in 616fb3e55c,
with the node resolution refactor.
The core issue here is just that we would unconditionally set the
solve opt's platform to the default current platform, which was
incorrect. We can prevent this easily by having a special case for the
default case, like we had before, and then not setting the platforms
field on this (which keeping the resolution behavior which was
introduced).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Fixesdocker/cli#4648
Make it easier to specify IPv6 addresses in the '--add-host' option by
permitting 'host=ip' in addition to 'host:ip', and allowing square
brackets around the address.
For example:
--add-host=hostname:127.0.0.1
--add-host=hostname:::1
--add-host=hostname=::1
--add-host=hostname=[::1]
Signed-off-by: Rob Murray <rob.murray@docker.com>
This adds status reporting for the remote driver so it shows the length
of time it is spending waiting for the connection. If the service is
already present, this logger isn't shown but it should help provide a
message to show the user why the build is stalled.
A timeout of 20 seconds has been added to the bootstrap.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
This patch reworks and updates the node resolution logic for selecting a
node from a builder.
The new implementation reworks the logic to make use of containerd's
platforms.Matcher interface instead of manually associated strings, and
additionally provides a few behavioural changes over the original
implementation, namely platforms can be matched with non-strict
semantics. e.g. i386 builds can be scheduled on an amd64 node, arm/v6
builds can be scheduled on an arm/v7 node.
We also add a new collection of tests for tracking regressions and
making the intended behaviour clearer.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The `docker-container` driver relies on the default config file location
for buildkit when writing the configuration file. When run in a rootless
version of docker (dind), the default location is different.
Instead of trying to figure out where the appropriate default location
is, this just writes the files to the same location and sets the
`--config` parameter explicitly. This flag is placed first so a
user-specified config option in `--buildkitd-flags` will take precedence
over the implicit config parameter.
This also fixes the `--config` option with the rootless image.
Previously, the config directory was being copied in a way that rendered
`/etc` unreadable and the configuration file wasn't readable either. It
also wasn't copied to the correct place. Now, `--config` is used to
specify the directory, `/etc` isn't included in the copied archive (so
the permissions aren't overwritten), and the directory is set as world
readable to be readable from the rootless buildkit process`.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Previously, we would fail while trying to use the global "target" field
when using a matrix. The contents of the matrix really don't matter for
this.
What was happening was that we would copy the "target" property into the
child evaluation context, so that when it was updated on the parent, it
wouldn't propagate to the child. The correct behavior here is to avoid
copying variables from the target evaluation context if it is the
root.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This adds the ability to source additional local build definition files when
sourcing Bake files via a remote url.
Prefixing a file with 'cwd://' will source a bake file on the local
machine, instead of the remote location.
Local files will be read/have precedence before remote files.
Usage:
```
docker buildx bake https://github.com/example/upstream.git --file cwd://docker-bake.override.hcl --print
```
This will source a default file from the example/upstream repository,
and also source a build definition from the local machine.
Also moves remote and local files reading logic to a func
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: Cameron Adams <pnzreba@gmail.com>
Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com>
we don't currently support reading a remote Dockerfile with a local
context when doing a remote invocation because we automatically derive
the dockerfile from the context atm. To avoid mistakenly reading a local
Dockerfile, we check if the Dockerfile exists locally and if so, we
error out.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
also update config to show all issues at once and
set same exlucde rules as buildkit ones.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Adds a new HostGatewayIP entry in the Driver interface so we can
move Features and HostGatewayIP handling with cache to the very
specific driver (docker).
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
If the boot command hung or took a long time, it blocked any read
operations (such as `buildx ls`).
When the boot happens, we no longer need to hold the file lock so we can
release it. Releasing multiple times is legal and causes whichever
release that is second to be a no-op so the defer is kept to ensure the
lock is released even when an error condition happens.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Without this, we attempt to overwrite the previous display in the lines
above which have been filled with logs from the monitor (see when used
with `--invoke`).
Signed-off-by: Justin Chadwell <me@jedevc.com>
This extracts the same logic for parsing annotations from the imagetools
create command, and allows the same flags to be attached to the build
command.
These annotations are then merged into all provided exporters.
Signed-off-by: Justin Chadwell <me@jedevc.com>
go1.20.8 (released 2023-09-06) includes two security fixes to the html/template
package, as well as bug fixes to the compiler, the go command, the runtime,
and the crypto/tls, go/types, net/http, and path/filepath packages. See the
Go 1.20.8 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.8+label%3ACherryPickApproved
full diff: https://github.com/golang/go/compare/go1.20.7...go1.20.8
From the security mailing:
[security] Go 1.21.1 and Go 1.20.8 are released
Hello gophers,
We have just released Go versions 1.21.1 and 1.20.8, minor point releases.
These minor releases include 4 security fixes following the security policy:
- cmd/go: go.mod toolchain directive allows arbitrary execution
The go.mod toolchain directive, introduced in Go 1.21, could be leveraged to
execute scripts and binaries relative to the root of the module when the "go"
command was executed within the module. This applies to modules downloaded using
the "go" command from the module proxy, as well as modules downloaded directly
using VCS software.
Thanks to Juho Nurminen of Mattermost for reporting this issue.
This is CVE-2023-39320 and Go issue https://go.dev/issue/62198.
- html/template: improper handling of HTML-like comments within script contexts
The html/template package did not properly handle HMTL-like "<!--" and "-->"
comment tokens, nor hashbang "#!" comment tokens, in <script> contexts. This may
cause the template parser to improperly interpret the contents of <script>
contexts, causing actions to be improperly escaped. This could be leveraged to
perform an XSS attack.
Thanks to Takeshi Kaneko (GMO Cybersecurity by Ierae, Inc.) for reporting this
issue.
This is CVE-2023-39318 and Go issue https://go.dev/issue/62196.
- html/template: improper handling of special tags within script contexts
The html/template package did not apply the proper rules for handling occurrences
of "<script", "<!--", and "</script" within JS literals in <script> contexts.
This may cause the template parser to improperly consider script contexts to be
terminated early, causing actions to be improperly escaped. This could be
leveraged to perform an XSS attack.
Thanks to Takeshi Kaneko (GMO Cybersecurity by Ierae, Inc.) for reporting this
issue.
This is CVE-2023-39319 and Go issue https://go.dev/issue/62197.
- crypto/tls: panic when processing post-handshake message on QUIC connections
Processing an incomplete post-handshake message for a QUIC connection caused a panic.
Thanks to Marten Seemann for reporting this issue.
This is CVE-2023-39321 and CVE-2023-39322 and Go issue https://go.dev/issue/62266.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- Parse service device count to int if possible
- introduce ResourceResolver to accept remote resources
- use include.env_file to resolve variables in included compose.yaml file
- remove potential dependencies to disabled services in ForServices
- ability to convert a mapping (back) to KEY=VALUE strings
- load: include details about included files on Project
- include disabled services
- local environment to override included .env
- load: move env var profile detection to option
- add support for multi-document yaml
full diff: https://github.com/compose-spec/compose-go/compare/v1.17.0...v1.18.3
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Also, tidy up the error printing, so that now we always print out the
"current commit information was not captured by the build" message,
instead of just for not locating the git binary.
Before:
WARNING: buildx: git was not found in the system. Current commit information was not captured by the build
After:
WARNING: current commit information was not captured by the build: git was not found in the system: <error message>
Signed-off-by: Justin Chadwell <me@jedevc.com>
Fixes the race condition where two boots are executed simultaneously
across multiple processes.
We initially check to see if the container exists, but if during
container creation we get a name conflict, we don't treat this error as
a hard failure, and instead move immediately into waiting for the node
to boot.
Signed-off-by: Justin Chadwell <me@jedevc.com>
On drivers that do not support multi-platform builds (the default
`docker` driver), we do not support building attestations (unless using
the containerd store).
We need to check this feature before attempting to build using
attestations.
Also adds a test to ensure that attestations can be pushed to registries
at all, and that it adequately fails on the docker driver.
Signed-off-by: Justin Chadwell <me@jedevc.com>
An integration test for `docker buildx version` has been created. The
integration test checks that there is one line output, the output is
composed of three sections, and that these sections could feasibly be
the package path, version, and revision information.
The intention of the checks is to find obvious errors in the output like
the package path not existing or the version and revision being swapped.
It is not intended to assert that these values must be certain values
because it is assumed these values may vary depending on the build
process for buildx.
Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
Includes a fix for CVE-2023-29409
go1.20.7 (released 2023-08-01) includes a security fix to the crypto/tls
package, as well as bug fixes to the assembler and the compiler. See the
Go 1.20.7 milestone on our issue tracker for details:
- https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved
- full diff: https://github.com/golang/go/compare/go1.20.6...go1.20.7
From the mailing list announcement:
[security] Go 1.20.7 and Go 1.19.12 are released
Hello gophers,
We have just released Go versions 1.20.7 and 1.19.12, minor point releases.
These minor releases include 1 security fixes following the security policy:
- crypto/tls: restrict RSA keys in certificates to <= 8192 bits
Extremely large RSA keys in certificate chains can cause a client/server
to expend significant CPU time verifying signatures. Limit this by
restricting the size of RSA keys transmitted during handshakes to <=
8192 bits.
Based on a survey of publicly trusted RSA keys, there are currently only
three certificates in circulation with keys larger than this, and all
three appear to be test certificates that are not actively deployed. It
is possible there are larger keys in use in private PKIs, but we target
the web PKI, so causing breakage here in the interests of increasing the
default safety of users of crypto/tls seems reasonable.
Thanks to Mateusz Poliwczak for reporting this issue.
View the release notes for more information:
https://go.dev/doc/devel/release#go1.20.7
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
No other parts of the progress rendering modify the inputs, so we should
avoid this as well.
This actually fixes an edge case in pushWithMoby which writes the same
VertexStatus multiple times, modifying the timestamps and similar.
However, if the operation takes long enough the small time difference
can accumulate, and move the Start time far into the past.
Signed-off-by: Justin Chadwell <me@jedevc.com>
- http2: properly discard data received after request/response body is closed
- http2: don't reuse connections that are experiencing errors
- internal/socks: permit authenticating with an empty password
full diff: https://github.com/golang/net/compare/v0.8.0...v0.10.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This native implementation uses GRPC level waiting, instead of starting
a busy loop. We also a manual max backoff of one second to improve
responsiveness.
Signed-off-by: Justin Chadwell <me@jedevc.com>
#7 [internal] load metadata for docker.io/library/golang:1.20.6-buster
#7 ERROR: docker.io/library/golang:1.20.6-buster: not found
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Use buildkit's parsing of entitlements to make sure that accepted
values match what's accepted by BuildKit.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This attempts to make it clearer that the --cgroup-parent option is only used
for the containers used during build. Instead of mentioning "build container",
I opted for using "RUN instructions" (to match the --network description),
although this may not be ideal (as it assumes the "Dockerfile" front-end, which
of course may not be the case).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This means that we can run our docker-container tests in parallel again,
which can help speed up our test runs by a *significant* amount.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Although a builder from the store cannot be created unless
it has a valid name, this is not the case for a Docker context.
We should skip name validation when checking a node from the
store and fall back to finding one from Docker context instead.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
We need to resolve the strip the cwd:// prefix before attempting to
resolve the dockerfile. Otherwise, we'll get the cwd:// prefix in the
dockerfile name, which isn't stripped out later.
Signed-off-by: Justin Chadwell <me@jedevc.com>
ReadLocalFiles should allow passing the stdin file as an argument, which
allows us to read from dockerCli.Stdin() to be consistent with other
commands in the same package.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This commit allows specifying a JSON array to the long-form arg, entrypoint and
envvars.
Non-JSON-array value can still be specified. Buildx treats the value as a JSON
array only when it can be parsed as a JSON array.
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
When running `--invoke` against images that have `Cmd` set, the interactions with `Entrypoint` start to cause issues like the following:
/usr/local/bin/bash: /usr/local/bin/bash: cannot execute binary file
Or:
sh: can't open 'bash': No such file or directory
This patch fixes those by explicitly setting `Cmd` to be empty if it is unspecified and `Entrypoint` is being set, which matches `docker`'s behavior:
$ docker image inspect --format '{{ json .Config.Entrypoint }} + {{ json .Config.Cmd }}' bash
["docker-entrypoint.sh"] + ["bash"]
$ docker create --name foo --entrypoint bash bash
$ docker container inspect --format '{{ json .Config.Entrypoint }} + {{ json .Config.Cmd }}' foo
["bash"] + null
$ docker rm foo
$ docker create --name foo bash ls
$ docker container inspect --format '{{ json .Config.Entrypoint }} + {{ json .Config.Cmd }}' foo
["docker-entrypoint.sh"] + ["ls"]
(There are still some weird edge cases in the interaction between the `InvokeConfig` and the original image config, but this fixes the most irritating for me and the rest are going to be deeper changes that are possibly less acceptable. 😅)
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
Fixed in 12b6a3ad9a, but now we have
regression tests! So we can add a check that we don't break this
behavior again.
Signed-off-by: Justin Chadwell <me@jedevc.com>
- Adds a new GitServeHTTP function to start an http server to serve a
target git repository.
- Adds a new GitDir helper method to get the path to the .git
directory
- Updates the GitAdd method to take a variable number of files
Signed-off-by: Justin Chadwell <me@jedevc.com>
Now, we always perform the full solve request in the main gateway call.
This ensures that progress works properly, and makes the lifetime
semantics much clearer.
NewResultContext abstracts the details of a successful/failed build, to
always return a single ResultContext, even though the details of how a
gateway is created is different:
- For a failed build, we can just keep the gateway open.
- For a successful build, we immediately open another gateway and
re-evaluate the build definition in that gateway. This should give an
instant cache hit (since the build was just successful).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Subrequests have been included in docker/dockerfile:1.5 labs, so we can
update the fallback to point to this release.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Container driver wrote manifest digest that had a
mismatch with --iidfile output.
When --iidfile was set the --metadata-file was not
written.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
There is no reason why one can't make a docker tarball or
load to Docker instance from remote driver.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Replace invalid characters in the name of the base directory of
certificates using the name of the registry which can contain
for example a colon when the port is specified.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
This was missing, since the driver property can only be fully populated
after loading nodes from disk. So we add logic to load the nodes, and
check for an error, which ensures that the "docker" driver is always
correctly present in the progress description.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Replace rules are not inherited by consumers of buildx as a module, and as
such would default to use the v0.26.2 version. Removing the replace rules
also removes various (indirect) dependencies (although brings in some new
packages from k8s itself).
The "azure" and "gcp" authentication packages in k8s.io/go-client are now
no longer functional, so removing those imports.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When running in standalone mode, the --debug flag passed to docker
cannot be passed. The docker cli also supports a DEBUG env var, however,
in standalone mode this won't be consumed.
This patch reads the contents of the DEBUG environment variable, and
enables debugging logs when it's been set.
Signed-off-by: Justin Chadwell <me@jedevc.com>
When the user supply static creds, we must not enrich them with a
session token which is unrelated.
Signed-off-by: Bertrand Paquet <bertrand.paquet@gmail.com>
This doesn't require pulling in a new direct dependency, we already use
docker/distribution throughout imagetools and build.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This ensures that `target.attest=["type=sbom,<value>"]` can be
appropriately merged when `--sbom=true` or `--set
target.attest=type=sbom`.
To merge, we simply naively take the last valid value.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Before this change, there were two bugs:
- the HCL was not valid. in hcl, argument names can't be quoted
- the target argument should be a real target
Signed-off-by: Nick Santos <nick.santos@docker.com>
When using imagetools create and combining multiple sources
we should check the media type of each manifest and set
the right media type for the manifest list.
If there is a mismatch we set OCI index as best effort.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
This fixes a possible race where messages printed directly after calls
to NewPrinter may appear before the printer starts. With this change, we
delay all of the logs until after.
Signed-off-by: Justin Chadwell <me@jedevc.com>
kind labels should only be manually added by maintainers after
confirming the issue. Otherwise, it can be tricky to sort through all
the issues when filtering for legitimate bugs.
Signed-off-by: Justin Chadwell <me@jedevc.com>
--detach shouldn't be the default yet, since it still has the potential
to leak some sessions in odd edge cases, and is slightly more painful to
debug.
For now, we should set the local controller as the default, with the
idea that we can change it back in the future.
Signed-off-by: Justin Chadwell <me@jedevc.com>
compose-go v1.13.0 supports the new additional_contexts to allow passing
additional build context during build, so we should map this to bake's
contexts property.
Signed-off-by: Justin Chadwell <me@jedevc.com>
compose-go v1.13.0 supports the new dockerfile_inline to allow including
a dockerfile inline in the compose definition, so we should map this to
bake's dockerfile-inline property.
Signed-off-by: Justin Chadwell <me@jedevc.com>
When resolving remote contexts locally in bake, then we need to ensure
that we properly unpack the contents of that context to the root
directory, instead of leaving it in the subdirectory.
Otherwise, any files will be found in the wrong location. Along with
this change, we also need a change to the dockerfile location lookup to
ensure that it is found at the root instead of in the subdirectory.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Print more understandable messages on error:
- When ps fails because the monitor doesn't attach to any session, print "no
attaching session" instead of "unknown ref".
- Avoid disconnect silently fails when the monitor doesn't attach to any
session. Print "no attaching session" error instead.
- Fix error message of "attach"'s arguments. ("server name must be passed" ->
"ID of session or process must be passed")
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
This adds an env var which can be used to pass in a path to a file to
read a buildkit source poliy from.
This is applied to any build is executed with the env set.
It is also applied to bakes (which are calling build behind the scenes).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This commit moves monitor commands to `monior/commands` package.
Commands still need access to the `monitor` object and buildx controller so this
commit enables this via `Monitor` interface stored in `monitor/types`.
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
This will allow result printing to work with the remote controller
(though this currently causes a panic, to be fixed in a follow-up).
Signed-off-by: Justin Chadwell <me@jedevc.com>
* Return errors from creating the `NodeGroup`
* Ensure that `b.NodeGroup != nil` before reading from
it during validation
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
logrus info messages aren't particularly in-theme with the rest of the
progress output (and are also frustratingly racy). The progress output
is a lot neater, so we refactor it into that.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Refactor the progress printer creation to the caller-side of the
controller api. Then, instead of passing around status channels (and
progressMode strings), we can simply pass around the higher level
interface progress.Writer.
This has a couple of benefits:
- A simplified interface to the controller
- Allows us to correctly extract warnings out of the controller, so that
they can be displayed correctly from the client side.
Some extra work is required to make sure that we can pass a
progress.Printer into the debug monitor. If we want to keep it
persistent, then we need a way to temporarily suspend output from it,
otherwise it will continue printing as the monitor is prompting for
input from the user, and forwarding output from debug containers.
To handle this, we add two methods to the printer, `Pause` and
`Unpause`. `Pause` acts similarly to `Wait`, closing the printer, and
cleanly shutting down the display - however, the printer does not
terminate, and can later be resumed by a call to `Unpause`. This
provides a neater interface to the caller, instead of needing to
continually reconstruct printers for every single time we want to
produce progress output.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We had some duplicated code between the basic runBuild and
launchControllerAndRunBuild.
This patch refactors out the common logic (since it's only really like
to keep growing), and has runBuild call into either the controller or
directly start the build depending on whether BUILDX_EXPERIMENTAL is
set.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This ensures that the code used to capture and evaluated a result is
only executed when built through the controller. Otherwise, no build
result should be recorded.
This ensures that new code added to capture and store the build result
for debugging isn't used when BUILDX_EXPERIMENTAL is not set.
Signed-off-by: Justin Chadwell <me@jedevc.com>
In 566f41b598, we added a check to ensure
that we avoid resolving http URLs for Dockerfile. However, we have
another circumstance we should not resolve the path in - if the context
is a remote context, the dockerfile is resolved in that context (see
build.go#LoadInputs for more information).
Therefore, we should only resolve the dockerfile to a local directory if
the context is also resolved to a local directory.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Dockerfiles can be HTTP URLs as well as local paths 🤦
We just copy the same logic we use for resolving context paths, and
apply it here as well.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The updateContext function may make modifications to the build inputs,
creating either an SSH URL, or an SSH llb.State. In these cases, we need
to ensure that we appropriately expose the client's default agent.
Previously, we would only expose it if the remote context was a git URL,
however, we need to also ensure that if the input was used to override
the context (in the case of ReadRemoteFiles), that we expose the agent
here as well.
Signed-off-by: Justin Chadwell <me@jedevc.com>
BuildKit's gitutil package behaves slightly differently than moby's
urlutil, so we should rely on BuildKit's gitutil when detecting URLs to
avoid cases of accidentally producing invalid build requests that can
confuse users.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This adds the following constraints to the new features:
- Explicit renaming with the `name` property is *only* permitted when
used with the `matrix` property.
- Group does not support either `name` or `matrix` (we may choose to
relax this constraint over time).
- All generated names must be unique.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Buildx currently returns the Docker Engine version for the docker driver
and it can be confusing. This is because BuildKit before 0.11 version does
not support the Info API that returns the BuildKit version.
This change resolves the BuildKit version from the Docker engine one.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Previously, we would explicitly error if all TLS parameters were not
available. However, it is a perfectly valid use case to connect to a
buildkit server that only provides TLS in one direction to verify the
server (which is possible today with buildctl).
To support this use case, we only need to error if only one of key or
cert is set, and the other is not - if both are unspecified, the client
will not present a certificate to the server.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, the name property could not be set in the body of a bake
target and could only be set for a label. This patch allows the body to
override the values of label fields, though the default is still the
label.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch allows high level clients to define an EvalContext method
which can derive a new context given a block and the base parent
context.
This allows users of the package to intercept evaluation before it
begins, and define additional variables and functions that are bound to
a single block.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, when directly modifying the args map when reading targets,
we could end up in a scenario where bake tests that compare arg maps
would fail if SOURCE_DATE_EPOCH was set in the environment.
This patch prevents this failure by setting the SOURCE_DATE_EPOCH at the
command level (which isn't injected into tests as well), ensuring that
we test correctly even when SOURCE_DATE_EPOCH is set in the environment.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This allows the build package code to become more generic, and also
ensures that when the environment variables are not propogated (in the
case of the remote controller), that we can still correctly set
SOURCE_DATE_EPOCH.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We can perform all attestation processing, handling how the sbom and
provenance arguments interact on the client, while applying defaults on
the server.
Additionally, this allows us to start pulling fields out of CommonOpts.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Now clients can access the result of the solve, specifically the image
id output. This is a useful refactor, as well as being required if we
want to allow bake to invoke through the controller api.
This also allows us to remove the quiet option from the API, since we
can compute the required progress type outside of the controller, and
can print the image id from the result of the solve.
As a follow-up, we should also be able to remove the image id file
output from the controller api, now that the client has access to it.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Strongly typing the API allows us to perform all command line parsing
fully on the client-side, where we have access to the client local
directory and all the client environment variables, which may not be
available on the remote server.
Additionally, the controller api starts to look a lot like
build.Options, so at some point in the future there may be an
oppportunity to merge the two, which would allow both build and bake to
execute through the controller, instead of needing to maintain multiple
code paths.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Linters 'deadcode' and 'varcheck' are deprecated since v1.49.0
and had been replaced by 'unused'
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
For certain cases we need to build with `--provenance=false`.
However not all build envs (especially in the OSS ethos) have the latest
buildx so just blanket setting `--provenance=false` will fail in these
cases.
Having an env var allows people to set the value without having to worry
about if the buildx version has the `--provenance` flag.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
To give us the option later down the road of producing recommended OCI
names in BuildKit (using com instead of vnd, woops), we need to update
Buildx to be able to process both.
Ideally, if a Buildx/BuildKit release hadn't been made we could just
switch over, but since we have, we'd need to support both (at least for
a while, eventually we could consider deprecating+removing the vnd
variant).
Signed-off-by: Justin Chadwell <me@jedevc.com>
This ensures that we should never accidentally connect to a server with
a mismatched version, while also allowing us to run multiple buildx
servers at a time.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch improves timeout logic for testing and creating a buildx
server. Instead of needing to have a custom loop, we can just use the
primitives provided by go and grpc for easier handling.
Additionally, without the explicit time.Sleeps, we defer to GRPCs
exponential backoff algorithm which should provide substantially better
results.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This signal may be sent using an external tool such as pkill, and since
we can handle it neatly, we should.
Signed-off-by: Justin Chadwell <me@jedevc.com>
When exiting, we should ideally always print a message, and give details
as to exactly what error we received.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We don't know if other builds might be running, etc, so we should allow
the server to decide when to exit.
Signed-off-by: Justin Chadwell <me@jedevc.com>
With changes to the lazy evaluation, the evaluation order is no longer
fixed - this means that we can follow long and confusing paths to get to
an error.
Because of the co-recursive nature of the lazy evaluation, we need to
take special care that the original HCL diagnostics are not discarded
and are preserved so that the original source of the error can be
detected. Preserving the full trace is not necessary, and probably not
useful to the user - all of the file that is not lazily loaded will be
eagerly loaded after all struct blocks are loaded - so the error would
be found regardless.
Signed-off-by: Justin Chadwell <me@jedevc.com>
With changes made to allow lazy evaluation, we were early exiting if an
undefined name was detected, either for a variable or a function.
This had two key implications:
1. The error messages changed, and became significantly less
informative.
For example, we went from:
> Unknown variable; There is no variable named "FO". Did you mean "FOO"?, and 1 other diagnostic(s)
To
> Invalid expression; undefined variable "FO"
2. Any issues in our function detection from funcCalls which cause JSON
functions to be erroneously detected cause invalid functions to be
resolved, which causes new name resolution errors.
To avoid the above problems, we can defer the error from an undefined
name until HCL evaluation - which produces the more informative errors,
and does not suffer from incorrectly detecting JSON functions.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This is now directly handled by our actions-toolkit. We keep
`.github/releases.json` file for now but will be removed on
next v0.11 release.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
We are currently using the GitHub API in our setup-buildx-action
to check for latest and tagged releases to make sure they exist
before download. But this requires using a token to avoid
rate-limit. It's fine for public runners but GHES runners don't
have the `github.token` populated automatically. They need to
create a PAT.
This PR will solve this issue by generating and pushing a
`releases.json` file in this repo when we publish a GitHub Release
that will then be fetched through `raw.githubusercontent.com`
endpoint on `setup-buildx-action` repo. This endpoint is better
served for our purpose with 5000 requests per hour compared to the
GitHub API endpoint that is limited to 60 requests per hour (unauth)
and 1000 request per hour when authenticated.
Also ignore .github/releases.json file on pull request event as an
action in a workflow run can't trigger a new workflow run anyway.
See https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#example-using-more-than-one-event
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Go 1.20 will be there soon, I think it's time to move our go.mod to
latest stable. We can then remove the compat in our vendor.Dockerfile
Downstream projects like compose or github.com/linuxkit/linuxkit should
not be affected.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
This change has been introduced in e7b5ee7518
but we should not check context builders when removing inactive
ones.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
We should avoid erroring with attestations support compatability errors
when a user has specified --provenance=false.
A user may wish to enable --provenance=false that works across buildkit
versions, but currently it will fail on old versions - this patch fixes
this, to silently ignore the provenance flag for this check if it's set
to disabled.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The buildx builder persists across Desktop restarts. So when we start up Jaeger, we should recommend that people configure it to persist across restarts as well.
Signed-off-by: Nick Santos <nick.santos@docker.com>
Delay loading the attestation data immediately, and only compute it upon
request. We do this using a deferred function which allows to define the
computation in the same place as before, but perform the computation
later.
With this patch, we ensure that the attestation data is only pulled from
the remote if it is actually referenced in the format string -
otherwise, we can skip it, for improved performance.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This refactor ensures that the attestations are not output in the JSON
output for "{{ json . }}", and additionally allows future refactors to
dynamically load the attestation contents, ensuring faster performance
when attestations are not used in the output.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Using the syncronization primitive, we can avoid needing to create a
separate wait group.
This allows us to sidestep the issue where the wait group could be
completed, but the build invocation functions had not terminated - if
one of the functions was to terminate with an error, then it was
possible to encounter a race condition, where the result handling code
would begin executing, despite an error.
The refactor to use a separate error group which more elegantly handles
the concept of function returns and errors, ensures that we can't
encounter this issue.
Signed-off-by: Justin Chadwell <me@jedevc.com>
All build options should be passed everywhere - the frontend and the
backend of buildkit should both be able to see all attestations, as well
as all other opts: e.g. epoch settings, and no-cache.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We allow any valid image reference format for the oci-layout, not just
limiting to name@digest, we additionally allow images of the form
name:tag@digest now.
The name of the reference is used to find the local directory to lookup
the store in, while the tag and digest are attached to a random identity
to generate the dummy reference sent to the oci-layout context.
This separation of the target to replace and the value to replace it
with ensures that any tag or digest set in the client is properly sent
across to the server. The tag is used when a digest was not specified,
and it is resolved in the context of the local directory before being
sent, using the same helpers as we use for the local cache expoter.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch adds support for block-based interpolation, so that
properties of blocks can be referenced in the current block and across
other blocks.
Previously, order-of-evaluation did not matter for blocks, and could be
evaluated in any order. However, now that blocks can refer to each
other, we split out this dynamic evaluation order into a separate
resolveBlock function.
Additionally, we need to support partial block evaluations - if block A
refers to property X of block B, when we should only evaluate property
X, and not the entire block. This ensures that we can safely evaluate
blocks that refer to other properties within themselves, and allows
sequences that would otherwise be co-recursive. We take special care in
this logic to ensure that each property is evaluated once *and only*
once - this could otherwise present inconsistencies with stateful
functions, and could risk inconsistent results.
Signed-off-by: Justin Chadwell <me@jedevc.com>
- drops the replace-rule for github.com/aws/aws-sdk-go-v2/config (as it no longer was replacing anything)
- drops the replace-rules for docker/cli and docker/docker (at least as long as we continue using tagged releases)
- removes the github.com/docker/docker/pkg/stringid package (which was redundant)
full diff: 9624ab4710..5993b526de
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This looks like an oversight, all of the other options have similar
checks. This can interfere with generated provenance where "add-hosts"
will be marked as an argument to the build, even though it's not
actually being utilized.
Signed-off-by: Justin Chadwell <me@jedevc.com>
I believe the documentation is misleading in saying that specifying a scope in a gha cache declaration will `ensure each branch gets its own cache`. I believe this should say that each build or each image will get its own cache.
Signed-off-by: Mark Hildreth <mark.hildreth@gravityclimate.com>
This patch adds more complete documentation for the various exporters
available to buildx.
Co-Authored-By: David Karlsson <david.karlsson@docker.com>
Signed-off-by: Justin Chadwell <me@jedevc.com>
The error handling for the cast to client.TracerDelegate was incorrect,
and previously, a client would unconditionally append an opt.
This results in the scenario that while the ClientOpt was not nil, the
tracer delegate in the ClientOpt was, which isn't an error case
explicitly handled by buildkit.
Signed-off-by: Justin Chadwell <me@jedevc.com>
The NewPrinter function is mostly borrowed from buildkit. However, at
some point, it seems that the implementations drifted.
This patch updates buildx to be more similar in behavior to it's
buildkit counterpart, specifically, it will explicitly fail if a TTY
output is requested using "--progress=tty", but the output is not
available.
To gracefully fallback to plain progress in this scenario,
"--progress=plain" is required.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This moves the detection of the docker daemon's security options into
the driver from the factory, handling them in a similar way to how we do
cgroups.
Because of recent changes that modify error detection in driver
creation, this attempt to contact the docker daemon during builder
creation meant that a docker-container builder could not be created
without access to the docker socket. This patch resolves this, by
defering the Info call to the driver, when the container is actually
created.
Signed-off-by: Justin Chadwell <me@jedevc.com>
We should be able to detect nolint comments that point to linters that
are disabled (such as with the removed structcheck).
Signed-off-by: Justin Chadwell <me@jedevc.com>
This fleshes out the documentation to include information along with
examples and detailed use cases for the cache backends. The general
format and style follows from the style of the build driver docs.
Eventually, these docs will be included on docs.docker.com along with
the rest of the build docs.
Signed-off-by: Justin Chadwell <me@jedevc.com>
If user does not specify image certain container parameters, we can load
them from the exporter metadata.
Additionally, we introduce a new "default" value for the --invoke flag,
that keeps all of the default parameters (since cobra does not have an
easy way of accepting an optional flag argument).
Signed-off-by: Justin Chadwell <me@jedevc.com>
Groups that contained other groups were not recursively resolved by
ReadTargets, which prevented output from --print from being useable as a
self-contained bake file.
This patch ensures that all groups that are referenced inside the bake
file are actually defined under the groups field. This has required a
substantial refactor, as previously only a single group was returned
from ReadTargets, notably, returning a map of groups, instead of a
slice.
This does introduce a small behavior change to the behavior of --print -
while previously, passing a group name to bake would return all the
targets of that group back as the default group, now only the name of
that group will be inserted into the default group, keeping the original
group intact. The impact of this can be observed in some of the changes
to the bake_test.go file.
Signed-off-by: Justin Chadwell <me@jedevc.com>
As buildkit now uses progress groups for the COPY --link instruction
we need to ensure that we additionally prefix the progress group name,
or the target name will be left off in bake commands with more than one
target.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Append an [experimental] tag to the end of each experimental command to
highlight that these are experimental options. Square brackets are used
instead of parentheses as parentheses are already in use to highlight
examples and defaults.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, the help information for buildx indicated that users could
create a new instance of the docker driver - which is explicitly
something we don't support, driver of this form are automatically
derived from the available list of docker contexts.
This patch ensures that don't have AllowsInstance set will not appear in
the help text, and additionally provide a new more specific error
message instead of the generic "failed to find driver". This should help
point users in the correct direction.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Print flag can be used to make additional information
requests about the build and print their results.
Currently Dockerfile supports: outline, targets, subrequests.describe
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The previous definition was the same as the docker images prune command
and referenced dangling images, which isn't what the command does. This
commit brings the command description more inline with the buildctl
definition.
Additionally, add some more description of what the various flags do in
our reference pages.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, editing nodes to contain a new set of driver options or
config files was unsupported, and silently dropping them. In this patch,
we update with these, as well as add a new warning message that any new
options may not taken into account until the builder restarts (which
may apply to the flags, platforms and endpoints as well).
Signed-off-by: Justin Chadwell <me@jedevc.com>
full diff: f1615facb1...418ca3b4d4
relevant changes;
- cli/command: remove unused args from ResolveDefaultContext()
- consider empty DOCKER_HOST and DOCKER_CONTEXT env-vars equivalent to "not set"
- cli: set timeout connection ping on sockets as well
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Previously, when specifying the filter option with the until value, no
cache would be cleaned, preventing users from clearing by time. This bug
arises from passing the until field through into buildkit, where, on
filtering, a non-existent field returns false for a match.
The fix is simple, as we build up our list of filters to pass to
buildkit, we skip over the until key, so create a valid list of filters
for buildkit.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch reorders+refactors the runCreate function to ensure that we
can detect and notify the user in the scenario that the user attempts to
combine multiple drivers in a single builder, which is an unsupported
scenario.
Previously, we would just overwrite the previous builder with the new
driver, potentially invalidating the already existing nodes.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This patch modifies the existing combining code in imagetools create to
provide better support for multiple repositories down the road.
Specifically, the code should no longer rely on a single repository
being used for all sources and tags, and should resolve descriptors in
their relevant repositories.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Create a dedicated folder for information on drivers, and write a new
index.md with content adapted from the README, and a new feature
comparisons table.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This builds on the added warnings from initialized builders, now
erroring the command, and additionally attempting to revert to the
previous configuration.
To preserve the previous configuration, we add a deep Copy() function to
the NodeGroup and Node so that we can easily restore it later if we
encounter a failure.
Signed-off-by: Justin Chadwell <me@jedevc.com>
also needs to update docker/docker to a60b458 (22.06 branch) otherwise
build breaks since docker/cli#3512 with:
# github.com/docker/cli/cli/flags
vendor/github.com/docker/cli/cli/flags/common.go:40:37: undefined: client.EnvOverrideCertPath
vendor/github.com/docker/cli/cli/flags/common.go:41:37: undefined: client.EnvTLSVerify
vendor/github.com/docker/cli/cli/flags/common.go:89:76: undefined: client.EnvOverrideHost
needs also to update github.com/spf13/cobra to v1.5.0 otherwise
build breaks with:
# github.com/docker/cli/cli-plugins/plugin
vendor/github.com/docker/cli/cli-plugins/plugin/plugin.go:130:4: unknown field 'HiddenDefaultCmd' in struct literal of type cobra.CompletionOptions
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Terraform includes a timestamp function to get the current time. go-cty
has imported a number of the timestamp functions to it's standard
library, however, this was one was not included.
This patch simply pulls in the TimestampFunc from Terraform's
internal/lang/funcs/datetime.go to allow easily fetching the current
time in bake.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Previously, errors within the driver config would not be reported to the
user until they tried to use the driver, even though they are easily
accessible from the node group info.
This patch reports these errors (but will not fail because of them,
since the data is already saved) - this should help improve
debuggability of some of the more complex drivers, and prevent error
messages being suppressed.
Signed-off-by: Justin Chadwell <me@jedevc.com>
This prevents the fall-through to the panic from division by zero in the
modulus below, and presents a neater error to the user.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Update modules:
go mod edit -require github.com/moby/buildkit@master
go mod tidy -compat=1.17 && ./hack/update-vendor
Signed-off-by: Justin Chadwell <me@jedevc.com>
Resolves the following message in golangci output:
> The linter 'golint' is deprecated (since v1.41.0) due to: The
repository of the linter has been archived by the owner. Replaced by
revive.
Additionally, fix a minor linting issue discovered by revive.
Signed-off-by: Justin Chadwell <me@jedevc.com>
BuildKit has supported external Dockerfile on remote contexts since
v0.5.0, included in Moby v19.03.0. The client side was the only missing
piece.
Signed-off-by: Cory Snider <csnider@mirantis.com>
This might break compatibility with projects using this module that
are still on go1.16, which is EOL, so probably ok to ignore:
github.com/docker/buildx/store imports
github.com/gofrs/flock tested by
github.com/gofrs/flock.test imports
gopkg.in/check.v1 loaded from gopkg.in/check.v1@v1.0.0-20200227125254-8fa46927fb4f,
but go 1.16 would select v1.0.0-20201130134442-10cb98267c6c
To upgrade to the versions selected by go 1.16:
go mod tidy -go=1.16 && go mod tidy -go=1.17
If reproducibility with go 1.16 is not needed:
go mod tidy -compat=1.17
For other options, see:
https://golang.org/doc/modules/pruning
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This patch completes the work started in creating a remote driver:
- Renames the env driver to the remote driver (an alternative suggestion
that should be more user-friendly)
- Adds support for TLS to encrypt connections with buildkitd
- Fixes outstanding review comments
- Reworks the buildx create command endpoint construction to be clearer
and include better support for this new driver.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Body.JustAttributes cannot distinguish between blocks and attributes for
JSON files, so the variable block could be included in the list of
attributes returned.
This patch ensures that JSON and HCL files behave the same way by
removing all known block types first, from the provided config schema
and then from a generated definitions schema.
Fixes#1051
Signed-off-by: Justin Chadwell <me@jedevc.com>
The --builder flag was being ignored by imagetools because of pointer
problems. Essentially, because the root cmds aren't parsed immediately,
we need to pass a pointer to the builder string so that it can be
updated before the RunE function gets called.
Signed-off-by: Justin Chadwell <me@jedevc.com>
Only change the image to the default rootless image when using the
--rootless option if the image has not already customized with the
--image option.
Fix#938
Signed-off-by: Doug Borg <dougborg@apple.com>
Kubernetes added the official `securityContext.seccompProfile` support in Kubernetes 1.19.
Seccomp is still disabled by default.
The legacy `container.seccomp.security.alpha.kubernetes.io/<PODNAME>` annotation has been deprecated and will be unsupported in Kubernetes 1.25.
https://kubernetes.io/docs/tutorials/security/seccomp/
A test cluster can be created with the following minikube command:
```
minikube start --feature-gates SeccompDefault=true --extra-config kubelet.seccomp-default=true
```
Related to moby/buildkit PR 2782
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
This fix adds a restriction `[a-zA-Z0-9_-]+`
for target name. This is pretty much the same as the
container name restriction in moby.
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Set `UsernsMode="host"` only when the daemon is running in userns-remapping mode.
Fix issue 561
The issue will be also fixed in moby/moby PR 43084 (Docker 20.10.13).
This buildx PR helps users of old releases of Docker.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Add --keep-daemon to the `rm` command option to preserve the buildkitd daemon after the buildx context is deleted.
Signed-off-by: Mayeul Blanzat <mayeul.blanzat@datadoghq.com>
containerd pusher can’t handle this case atm so we
need to make sure we always create a new resolver
for each name.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Before this only recorded errors instead of setting the span status,
which makes it harder to dig through.
Now an error that bubbles is reflected in the span status.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
This allows the parent cgroup to be customised, which allows resource
limits to be imposed on build containers separately from "user"
containers.
Signed-off-by: David Scott <dave@recoil.org>
For array fields, overrides are merged together
but override is not merged with the target. If merging
with target is desired we can add support for
overrides with += operator in the future.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
- ./_site/engine/reference/commandline/buildx_build/index.html
* linking to internal hash #--buildkitd-flags-flags that does not exist (line 904)
<a href="/engine/reference/commandline/buildx_create/#--buildkitd-flags-flags"></a>
- ./_site/engine/reference/commandline/buildx_create/index.html
* linking to internal hash #--buildkitd-flags-flags that does not exist (line 350)
<a href="#--buildkitd-flags-flags"></a>
* linking to internal hash #--config-file that does not exist (line 336)
<a href="#--config-file"></a>
* linking to internal hash #--config-file that does not exist (line 336)
<a href="/engine/reference/commandline/buildx_build/#--load"></a>
* linking to internal hash #--load that does not exist (line 369)
<a href="/engine/reference/commandline/buildx_build/#--load"></a>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The container-driver creates a Linux container (as there currently isn't a
Windows version of buildkitd). However, the defaults are platform specific.
Buildx was using the defaults from the buildkit `util/appdefault' package,
which resulted in Buildx running on a Windows client to create a Linux
container that used the Windows location, which causes it to fail:
invalid mount config for type "volume": invalid mount path: 'C:/ProgramData/buildkitd/.buildstate' mount path must be absolute
This patch hard-codes the destination to the default Linux path.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
- updated buildkit to current code in master via:
go mod edit -require github.com/moby/buildkit@master && go mod tidy && ./hack/update-vendor
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
Put the flag descriptions/examples under an "examples"
section (used at docs.docker.com), and rephrase the
headings to be more consistent with other pages in the
docker documentation.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use the usage output of `--help` for each subcommand, to make
sure all flags/options are included on the page, and to make
it easier to keep docs in sync.
Note that the usage output is only used when reading these
docs on GitHub; docs.docker.com only consumes the "description"
and "example" sections (when present), and generates flag information
and usage output from source
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
v1.1.1:
- Fix: yaml.v2 2.3.0 contained a unintended breaking change. This release reverts
to yaml.v2 v2.2.8 which has recent critical CVE fixes, but does not have the
breaking changes.
- Fix: correct internal formatting for go-md2man v2 (which caused man page
generation to be broken).
v1.1.0:
- Extend Go completions and revamp zsh comp
- Add completion for help command
- Complete subcommands when TraverseChildren is set
- Fix stderr printing functions
- fix: fish output redirection
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Addresses https://github.com/docker/setup-buildx-action/issues/45
Simple repro:
```
$ buildx create --platform linux/amd64 --use
$ buildx build - <<EOF
from scratch
EOF
```
Since https://github.com/docker/buildx/pull/370 a `platform` driver-opt was automatically inserted with the value specified by `--platform` flag on regardless of the type of driver, even though it was only used in the kubernetes driver. However, because the docker-container driver is pedantic about the options being passed, it errored out.
Another side-effect I suspect is that with the kubernetes driver it was now possible to specify the platforms in two different ways: `--driver-opt platform=...` and `--platform`.
This patch reverts completely the `platform` driver-opt and instead ensures the platforms information is passed onto the kubernetes driver via variables.
Signed-off-by: Tibor Vass <tibor@docker.com>
commit c41b006be1 updated the version of
docker/docker in go.mod, but possibly overlooked that there was still a
replace rule present. As a result the version was not actually updated.
This patch removes the replace rule, updating docker/docker to 9f28837c1d93
full diff: 4634ce647c...9f28837c1d
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
These annotations were picked up by the YAML docs generator, and shows up as
"minimum API version". I couldn't find a reference to these annotations in the
PR that added them, so thought it would be ok to remove
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/hashicorp/hcl/compare/v2.4.0...v2.6.0
v2.6.0
-------------------------
Enhancements:
- hcldec: Add a new Spec, ValidateSpec, which allows custom validation of values at decode-time.
Bugs Fixed:
- hclsyntax: Fix panic with combination of sequences and null arguments
- hclsyntax: Fix handling of unknown values and sequences
v2.5.1
-------------------------
- hclwrite: handle legacy dot access of numeric indexes. (#369)
- hclwrite: Fix panic for dotted full splat (foo.*)
v2.5.0
-------------------------
- hclwrite: Generate multi-line objects and maps.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
full diff: https://github.com/gofrs/flock/compare/v0.7.0...v0.7.3
v0.7.3
-------------------------
- Fix issues in the license file, update year.
v0.7.2
-------------------------
- Ensure we release file handle if we failed to take an exclusive lock
v0.7.1
-------------------------
- Fix linting issues and add goreportcard badge
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
v0.7.2
----------------
Fixes:
- solver: gracefully handle cache loading errors
- remotecache: only visit each item once when walking results
- cache: avoid possible nil dereference on error handling
- contenthash: allow security.capability in cache checksum
- contenthash: treat unix sockets as regular files
- push: fix race condition on pushing the same layers in parallel
- inline cache: fix handling of duplicate blobs in same image
- gateway: fix metadata getting lost on subsolve in external frontend
- filesync: avoid ignoring close error
- runc: update runc binary to v1.0.0-rc91
- buildctl-daemonless: allow max retries on socket connect for buildctl
- buildctl-daemonless: fix shell args expansion
- buildctl-daemonless: show log on startup timeout
v0.7.1
----------------
Fixes:
- git: use --force flag on fetch
- tar exporter: handle symlinks properly
- resolver: disable http2 for pushing
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
1.6.0
--------------------------
Fixes:
* end of line cleanup
* revert the entry concurrency bug fix whic leads to deadlock under some circumstances
* update dependency on go-windows-terminal-sequences to fix a crash with go 1.14
Features:
* add an option to the `TextFormatter` to completely disable fields quoting
1.5.0
--------------------------
Code quality:
* add golangci linter run on travis
Fixes:
* add mutex for hooks concurrent access on `Entry` data
* caller function field for go1.14
* fix build issue for gopherjs target
Feature:
* add an hooks/writer sub-package whose goal is to split output on different stream depending on the trace level
* add a `DisableHTMLEscape` option in the `JSONFormatter`
* add `ForceQuote` and `PadLevelText` options in the `TextFormatter`
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
I needed "split" specifically so I can do something like:
```hcl
variable PLATFORMS {
default = "linux/amd64"
}
target foo {
platforms = split(",", "${PLATFORMS}")
# other stuff
}
```
Where the existing "csvdecode" does not work for this because it parses
the string into a list of objects instead of a list of strings.
I went ahead and just added all the available new functions.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
The docker builder prune command has a shorthand `-f` flag for `--force`:
docker builder prune --help
Usage: docker builder prune
Remove build cache
Options:
-a, --all Remove all unused build cache, not just dangling ones
--filter filter Provide filter values (e.g. 'until=24h')
-f, --force Do not prompt for confirmation
--keep-storage bytes Amount of disk space to keep for cache
Given that `buildx` can be used as a drop-in replacement for the native build
commands, it should match the UI, and also have a shorthand flag.
This patch also updates the flag's description to be in line with the docker commandline
With this patch applied;
buildx prune --help
Remove build cache
Usage:
buildx prune [flags]
Flags:
-a, --all Remove all unused images, not just dangling ones
--filter filter Provide filter values (e.g. 'until=24h')
-f, --force Do not prompt for confirmation
-h, --help help for prune
--keep-storage bytes Amount of disk space to keep for cache
--verbose Provide a more verbose output
Global Flags:
--builder string Override the configured builder instance
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This allows all subcommands to use this flag.
Additionally reads the default value for the flag from the
`BUILDX_BUILDER` env var.
Precedence is:
CLI ARG > flag > env var > config file
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Although bake is for running multiple targets, --set required a single
target name for overriding a property. This change allows matching
multiple targets for overrides.
Signed-off-by: Tibor Vass <tibor@docker.com>
When following this pattern:
buildx build --arg FOO
Where we want to pull `FOO` from env, currently we always set `FOO`
regardless if the `FOO` env var is even set.
This change makes it so that `FOO` would only be set if it has been set
in the env (even if it is set to empty).
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Getting the QEMU interpreters to work transparently inside containers seems to be a bit of a sticking point, with major distributions still on kernels and/or binfmt-support package versions that don't support the fix_binary flag yet. Give the reader a little more guidance what to look out for.
When booting `docker-container` driver, it will pull and run image
`moby/buildkit:buildx-stable-1`.
If current node cannot connect to dockerhub, driver `docker-container`
will always booting failure.
But user may already load the image manually or pull it from a priviate
registry.
Buildx should check local docker image store after pull failed.
Fixes: #199 issuecomment-561996661
Signed-off-by: Lu Jingxiao <lujingxiao@huawei.com>
Tested with `kind` and GKE.
Note: "nodes" shown in `docker buildx ls` are unrelated to Kubernetes "nodes".
Probably buildx should come up with an alternative term.
Usage:
$ kind create cluster
$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
$ docker buildx create --driver kubernetes --driver-opt replicas=3 --use
$ docker buildx build -t foo --load .
`--load` loads the image into the local Docker.
Driver opts:
- `image=IMAGE` - Sets the container image to be used for running buildkit.
- `namespace=NS` - Sets the Kubernetes namespace. Defaults to the current namespace.
- `replicas=N` - Sets the number of `Pod` replicas. Defaults to 1.
- `rootless=(true|false)` - Run the container as a non-root user without `securityContext.privileged`. Defaults to false.
- `loadbalance=(sticky|random)` - Load-balancing strategy. If set to "sticky", the pod is chosen using the hash of the context path. Defaults to "sticky"
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
Move a paragraph in README to clarify where it fits in the structure.
- Before the move, the paragraph seems to apply to the `--output=local` section when in fact it applies to the entire `--output` section. This is especially confusing for the sentence "if just the path is specified as a value, `buildx` will use the local exporter with this path as the destination".
- After the move, it is clear that the paragraph applies to `--output`
This works just like the `build` command where if you have `--build-arg
FOO`, it will read the variable from env and only set a value if the
variable is defined.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Using `buildx`as a docker CLI plugin requires using Docker 19.03. A limited set of functionality works with older versions of Docker when invoking the binary directly.
Using `buildx`with Docker requires Docker engine 19.03 or newer.
### Docker CE
> **Warning**
>
> Using an incompatible version of Docker may result in unexpected behavior,
> and will likely cause issues, especially when using Buildx builders with more
> recent versions of BuildKit.
`buildx` comes bundled with Docker CE starting with 19.03, but requires experimental mode to be enabled on the Docker CLI.
To enable it, `"experimental": "enabled"` can be added to the CLI configuration file `~/.docker/config.json`. An alternative is to set the `DOCKER_CLI_EXPERIMENTAL=enabled` environment variable.
## Windows and macOS
### Binary release
Docker Buildx is included in [Docker Desktop](https://docs.docker.com/desktop/)
for Windows and macOS.
Download the latest binary release from https://github.com/docker/buildx/releases/latest and copy it to `~/.docker/cli-plugins` folder with name `docker-buildx`.
## Linux packages
Change the permission to execute:
```sh
chmod a+x ~/.docker/cli-plugins/docker-buildx
Docker Engine package repositories contain Docker Buildx packages when installed according to the
[Docker Engine install documentation](https://docs.docker.com/engine/install/). Install the
`docker-buildx-plugin` package to install the Buildx plugin.
## Manual download
> **Important**
>
> This section is for unattended installation of the buildx component. These
> instructions are mostly suitable for testing purposes. We do not recommend
> installing buildx using manual download in production environments as they
> will not be updated automatically with security updates.
>
> On Windows and macOS, we recommend that you install [Docker Desktop](https://docs.docker.com/desktop/)
> instead. For Linux, we recommend that you follow the [instructions specific for your distribution](#linux-packages).
You can also download the latest binary from the [GitHub releases page](https://github.com/docker/buildx/releases/latest).
Rename the relevant binary and copy it to the destination matching your OS:
Buildx is a Docker CLI plugin that extends the `docker build` command with the full support of the features provided by [Moby BuildKit](https://github.com/moby/buildkit) builder toolkit. It provides the same user experience as `docker build` with many new features like creating scoped builder instances and building against multiple nodes concurrently.
Buildx is a Docker CLI plugin that extends the `docker build` command with the
full support of the features provided by [Moby BuildKit](https://github.com/moby/buildkit)
builder toolkit. It provides the same user experience as `docker build` with
many new features like creating scoped builder instances and building against
multiple nodes concurrently.
After installation, buildx can be accessed through the `docker buildx` command. `docker buildx build` is the command for starting a new build.
After installation, buildx can be accessed through the `docker buildx` command
with Docker 19.03. `docker buildx build` is the command for starting a new
build. With Docker versions older than 19.03 buildx binary can be called
directly to access the `docker buildx` subcommands.
```
```console
$ docker buildx build .
[+] Building 8.4s (23/32)
=> ...
```
Buildx will always build using the BuildKit engine and does not require
`DOCKER_BUILDKIT=1` environment variable for starting builds.
Buildx will always build using the BuildKit engine and does not require `DOCKER_BUILDKIT=1` environment variable for starting builds.
The `docker buildx build` command supports features available for `docker build`,
including features such as outputs configuration, inline build caching, and
specifying target platform. In addition, Buildx also supports new features that
are not yet available for regular `docker build` like building manifest lists,
distributed caching, and exporting build results to OCI image tarballs.
Buildx build command supports the features available for `docker build` including the new features in Docker 19.03 such as outputs configuration, inline build caching or specifying target platform. In addition, buildx supports new features not yet available for regular `docker build` like building manifest lists, distributed caching, exporting build results to OCI image tarballs etc.
Buildx is flexible and can be run in different configurations that are exposed
through various "drivers". Each driver defines how and where a build should
run, and have different feature sets.
Buildx is supposed to be flexible and can be run in different configurations that are exposed through a driver concept. Currently, we support a "docker" driver that uses the BuildKit library bundled into the docker daemon binary, and a "docker-container" driver that automatically launches BuildKit inside a Docker container. We plan to add more drivers in the future, for example, one that would allow running buildx inside an (unprivileged) container.
The user experience of using buildx is very similar across drivers, but there are some features that are not currently supported by the "docker" driver, because the BuildKit library bundled into docker daemon currently uses a different storage component. In contrast, all images built with "docker" driver are automatically added to the "docker images" view by default, whereas when using other drivers the method for outputting an image needs to be selected with `--output`.
We currently support the following drivers:
- The `docker` driver ([guide](docs/manuals/drivers/docker.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
- The `docker-container` driver ([guide](docs/manuals/drivers/docker-container.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
- The `kubernetes` driver ([guide](docs/manuals/drivers/kubernetes.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
- The `remote` driver ([guide](docs/manuals/drivers/remote.md))
For more information on drivers, see the [drivers guide](docs/manuals/drivers/index.md).
## Working with builder instances
By default, buildx will initially use the "docker" driver if it is supported, providing a very similar user experience to the native `docker build`. But using a local shared daemon is only one way to build your applications.
By default, buildx will initially use the `docker` driver if it is supported,
providing a very similar user experience to the native `docker build`. Note that
you must use a local shared daemon to build your applications.
Buildx allows you to create new instances of isolated builders. This can be used for getting a scoped environment for your CI builds that does not change the state of the shared daemon or for isolating the builds for different projects. You can create a new instance for a set of remote nodes, forming a build farm, and quickly switch between them.
Buildx allows you to create new instances of isolated builders. This can be
used for getting a scoped environment for your CI builds that does not change
the state of the shared daemon or for isolating the builds for different
projects. You can create a new instance for a set of remote nodes, forming a
build farm, and quickly switch between them.
New instances can be created with `docker buildx create` command. This will create a new builder instance with a single node based on your current configuration. To use a remote node you can specify the `DOCKER_HOST` or remote context name while creating the new builder. After creating a new instance you can manage its lifecycle with the `inspect`, `stop` and `rm` commands and list all available builders with `ls`. After creating a new builder you can also append new nodes to it.
You can create new instances using the [`docker buildx create`](docs/reference/buildx_create.md)
command. This creates a new builder instance with a single node based on your
current configuration.
To switch between different builders use `docker buildx use <name>`. After running this command the build commands would automatically keep using this builder.
To use a remote node you can specify the `DOCKER_HOST` or the remote context name
while creating the new builder. After creating a new instance, you can manage its
lifecycle using the [`docker buildx inspect`](docs/reference/buildx_inspect.md),
[`docker buildx stop`](docs/reference/buildx_stop.md), and
[`docker buildx rm`](docs/reference/buildx_rm.md) commands. To list all
available builders, use [`buildx ls`](docs/reference/buildx_ls.md). After
creating a new builder you can also append new nodes to it.
Docker 19.03 also features a new `docker context` command that can be used for giving names for remote Docker API endpoints. Buildx integrates with `docker context` so that all of your contexts automatically get a default builder instance. While creating a new builder instance or when adding a node to it you can also set the context name as the target.
To switch between different builders, use [`docker buildx use <name>`](docs/reference/buildx_use.md).
After running this command, the build commands will automatically use this
builder.
Docker also features a [`docker context`](https://docs.docker.com/engine/reference/commandline/context/)
command that can be used for giving names for remote Docker API endpoints.
Buildx integrates with `docker context` so that all of your contexts
automatically get a default builder instance. While creating a new builder
instance or when adding a node to it you can also set the context name as the
target.
## Building multi-platform images
BuildKit is designed to work well for building for multiple platforms and not only for the architecture and operating system that the user invoking the build happens to run.
BuildKit is designed to work well for building for multiple platforms and not
only for the architecture and operating system that the user invoking the build
happens to run.
When invoking a build, the `--platform` flag can be used to specify the target platform for the build output, (e.g. linux/amd64, linux/arm64, darwin/amd64). When the current builder instance is backed by the "docker-container" driver, multiple platforms can be specified together. In this case, a manifest list will be built, containing images for all of the specified architectures. When this image is used in `docker run` or `docker service`, Docker will pick the correct image based on the node’s platform.
When you invoke a build, you can set the `--platform` flag to specify the target
platform for the build output, (for example, `linux/amd64`, `linux/arm64`, or
`darwin/amd64`).
Multi-platform images can be built by mainly three different strategies that are all supported by buildx and Dockerfiles. You can use the QEMU emulation support in the kernel, build on multiple native nodes using the same builder instance or use a stage in Dockerfile to cross-compile to different architectures.
When the current builder instance is backed by the `docker-container` or
`kubernetes` driver, you can specify multiple platforms together. In this case,
it builds a manifest list which contains images for all specified architectures.
When you use this image in [`docker run`](https://docs.docker.com/engine/reference/commandline/run/)
or [`docker service`](https://docs.docker.com/engine/reference/commandline/service/),
Docker picks the correct image based on the node's platform.
QEMU is the easiest way to get started if your node already supports it (e.g. if you are using Docker Desktop). It requires no changes to your Dockerfile and BuildKit will automatically detect the secondary architectures that are available. When BuildKit needs to run a binary for a different architecture it will automatically load it through a binary registered in the binfmt_misc handler.
You can build multi-platform images using three different strategies that are
supported by Buildx and Dockerfiles:
Using multiple native nodes provides better support for more complicated cases not handled by QEMU and generally have better performance. Additional nodes can be added to the builder instance with `--append` flag.
1.Using the QEMU emulation support in the kernel
2. Building on multiple native nodes using the same builder instance
3. Using a stage in Dockerfile to cross-compile to different architectures
QEMU is the easiest way to get started if your node already supports it (for
example. if you are using Docker Desktop). It requires no changes to your
Dockerfile and BuildKit automatically detects the secondary architectures that
are available. When BuildKit needs to run a binary for a different architecture,
it automatically loads it through a binary registered in the `binfmt_misc`
handler.
For QEMU binaries registered with `binfmt_misc` on the host OS to work
transparently inside containers they must be registered with the `fix_binary`
flag. This requires a kernel >= 4.8 and binfmt-support >= 2.1.7. You can check
for proper registration by checking if `F` is among the flags in
`/proc/sys/fs/binfmt_misc/qemu-*`. While Docker Desktop comes preconfigured
with `binfmt_misc` support for additional platforms, for other installations
it likely needs to be installed using [`tonistiigi/binfmt`](https://github.com/tonistiigi/binfmt)
image.
```console
$ docker run --privileged --rm tonistiigi/binfmt --install all
```
# assuming contexts node-amd64 and node-arm64 exist in "docker context ls"
Using multiple native nodes provide better support for more complicated cases
that are not handled by QEMU and generally have better performance. You can
add additional nodes to the builder instance using the `--append` flag.
Assuming contexts `node-amd64` and `node-arm64` exist in `docker context ls`;
Finally, depending on your project, the language that you use may have good support for cross-compilation. In that case, multi-stage builds in Dockerfiles can be effectively used to build binaries for the platform specified with `--platform` using the native architecture of the build node. List of build arguments like `BUILDPLATFORM` and `TARGETPLATFORM` are available automatically inside your Dockerfile and can be leveraged by the processes running as part of your build.
Finally, depending on your project, the language that you use may have good
support for cross-compilation. In that case, multi-stage builds in Dockerfiles
can be effectively used to build binaries for the platform specified with
`--platform` using the native architecture of the build node. A list of build
arguments like `BUILDPLATFORM` and `TARGETPLATFORM` is available automatically
inside your Dockerfile and can be leveraged by the processes running as part
of your build.
```
```dockerfile
# syntax=docker/dockerfile:1
FROM --platform=$BUILDPLATFORM golang:alpine AS build
ARG TARGETPLATFORM
ARG BUILDPLATFORM
@@ -141,537 +304,12 @@ FROM alpine
COPY --from=build /log /log
```
You can also use [`tonistiigi/xx`](https://github.com/tonistiigi/xx) Dockerfile
cross-compilation helpers for more advanced use-cases.
## High-level build options
Buildx also aims to provide support for higher level build concepts that go beyond invoking a single build command. We want to support building all the images in your application together and let the users define project specific reusable build flows that can then be easily invoked by anyone.
BuildKit has great support for efficiently handling multiple concurrent build requests and deduplicating work. While build commands can be combined with general-purpose command runners (eg. make), these tools generally invoke builds in sequence and therefore can’t leverage the full potential of BuildKit parallelization or combine BuildKit’s output for the user. For this use case we have added a command called `docker buildx bake`.
Currently, the bake command supports building images from compose files, similar to `compose build` but allowing all the services to be built concurrently as part of a single request.
There is also support for custom build rules from HCL/JSON files allowing better code reuse and different target groups. The design of bake is in very early stages and we are looking for feedback from users.
# Documentation
### `buildx build [OPTIONS] PATH | URL | -`
The `buildx build` command starts a build using BuildKit. This command is similar to the UI of `docker build` command and takes the same flags and arguments.
Options:
| Flag | Description |
| --- | --- |
| --add-host [] | Add a custom host-to-IP mapping (host:ip)
| --allow [] | Allow extra privileged entitlement, e.g. network.host, security.insecure
| --progress string | Set type of progress output (auto, plain, tty). Use plain to show container output (default "auto")
| --pull | Always attempt to pull a newer version of the image
| --push | Shorthand for --output=type=registry
| --secret [] | Secret file to expose to the build: id=mysecret,src=/local/secret
| --ssh [] | SSH agent socket or keys to expose to the build (format: default|<id>[=<socket>|<key>[,<key>]])
| --tag [] | Name and optionally a tag in the 'name:tag' format
| --target string | Set the target build stage to build.
For documentation on most of these flags refer to `docker build` documentation in https://docs.docker.com/engine/reference/commandline/build/ . In here we’ll document a subset of the new flags.
#### ` --platform=value[,value]`
Set the target platform for the build. All `FROM` commands inside the Dockerfile without their own `--platform` flag will pull base images for this platform and this value will also be the platform of the resulting image. The default value will be the current platform of the buildkit daemon.
When using `docker-container` driver with `buildx`, this flag can accept multiple values as an input separated by a comma. With multiple values the result will be built for all of the specified platforms and joined together into a single manifest list.
If the`Dockerfile` needs to invoke the `RUN` command, the builder needs runtime support for the specified platform. In a clean setup, you can only execute `RUN` commands for your system architecture. If your kernel supports binfmt_misc https://en.wikipedia.org/wiki/Binfmt_misc launchers for secondary architectures buildx will pick them up automatically. Docker desktop releases come with binfmt_misc automatically configured for `arm64` and `arm` architectures. You can see what runtime platforms your current builder instance supports by running `docker buildx inspect --bootstrap`.
Inside a `Dockerfile`, you can access the current platform value through `TARGETPLATFORM` build argument. Please refer to `docker build` documentation for the full description of automatic platform argument variants https://docs.docker.com/engine/reference/builder/#automatic-platform-args-in-the-global-scope .
The formatting for the platform specifier is defined in https://github.com/containerd/containerd/blob/v1.2.6/platforms/platforms.go#L63 .
Sets the export action for the build result. In `docker build` all builds finish by creating a container image and exporting it to `docker images`. `buildx` makes this step configurable allowing results to be exported directly to the client, oci image tarballs, registry etc.
Supported exported types are:
##### `local`
The `local` export type writes all result files to a directory on the client. The new files will be owned by the current user. On multi-platform builds, all results will be put in subdirectories by their platform.
Attribute key:
-`dest` - destination directory where files will be written
##### `tar`
The `tar` export type writes all result files as a single tarball on the client. On multi-platform builds all results will be put in subdirectories by their platform.
Attribute key:
-`dest` - destination path where tarball will be written. “-” writes to stdout.
##### `oci`
The `oci` export type writes the result image or manifest list as an OCI image layout tarball https://github.com/opencontainers/image-spec/blob/master/image-layout.md on the client.
Attribute key:
-`dest` - destination path where tarball will be written. “-” writes to stdout.
##### `docker`
The `docker` export type writes the single-platform result image as a Docker image specification tarball https://github.com/moby/moby/blob/master/image/spec/v1.2.md on the client. Tarballs created by this exporter are also OCI compatible.
Currently, multi-platform images cannot be exported with the `docker` export type. The most common usecase for multi-platform images is to directly push to a registry (see [`registry`](#registry)).
Attribute keys:
-`dest` - destination path where tarball will be written. If not specified the tar will be loaded automatically to the current docker instance.
-`context` - name for the docker context where to import the result
##### `image`
The `image` exporter writes the build result as an image or a manifest list. When using `docker` driver the image will appear in `docker images`. Optionally image can be automatically pushed to a registry by specifying attributes.
Attribute keys:
-`name` - name (references) for the new image.
-`push` - boolean to automatically push the image.
##### `registry`
The `registry` exporter is a shortcut for `type=image,push=true`.
Buildx with `docker` driver currently only supports local, tarball exporter and image exporter. `docker-container` driver supports all the exporters.
If just the path is specified as a value, `buildx` will use the local exporter with this path as the destination. If the value is “-”, `buildx` will use `tar` exporter and write to `stdout`.
Shorthand for [`--output=type=registry`](#registry). Will automatically push the build result to registry.
#### `--load`
Shorthand for [`--output=type=docker`](#docker). Will automatically load the single-platform build result to `docker images`.
#### `--cache-from=[NAME|type=TYPE[,KEY=VALUE]]`
Use an external cache source for a build. Supported types are `registry` and `local`. The `registry` source can import cache from a cache manifest or (special) image configuration on the registry. The `local` source can import cache from local files previously exported with `--cache-to`.
If no type is specified, `registry` exporter is used with a specified reference.
`docker` driver currently only supports importing build cache from the registry.
Export build cache to an external cache destination. Supported types are `registry`, `local` and `inline`. Registry exports build cache to a cache manifest in the registry, local exports cache to a local directory on the client and inline writes the cache metadata into the image configuration.
`docker` driver currently only supports exporting inline cache metadata to image configuration. Alternatively, `--build-arg BUILDKIT_INLINE_CACHE=1` can be used to trigger inline cache exporter.
Attribute key:
- `mode` - Specifies how many layers are exported with the cache. “min” on only exports layers already in the final build build stage, “max” exports layers for all stages. Metadata is always exported for the whole build.
Allow extra privileged entitlement. List of entitlements:
- `network.host` - Allows executions with host networking.
- `security.insecure` - Allows executions without sandbox. See [related Dockerfile extensions](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---securityinsecuresandbox).
For entitlements to be enabled, the `buildkitd` daemon also needs to allow them with `--allow-insecure-entitlement` (see [`create --buildkitd-flags`](#--buildkitd-flags-flags))
Create makes a new builder instance pointing to a docker context or endpoint, where context is the name of a context from `docker context ls` and endpoint is the address for docker socket (eg. `DOCKER_HOST` value).
By default, the current docker configuration is used for determining the context/endpoint value.
Builder instances are isolated environments where builds can be invoked. All docker contexts also get the default builder instance.
Options:
| Flag | Description |
| --- | --- |
| --append | Append a node to builder instead of changing it
| --buildkitd-flags string | Flags for buildkitd daemon
| --config string | BuildKit config file
| --driver string | Driver to use (eg. docker-container)
| --driver-opt stringArray | Options for the driver
| --leave | Remove a node from builder instead of changing it
| --name string | Builder instance name
| --node string | Create/modify node with given name
| --platform stringArray | Fixed platforms for current node
| --use | Set the current builder instance
#### `--append`
Changes the action of the command to appends a new node to an existing builder specified by `--name`. Buildx will choose an appropriate node for a build based on the platforms it supports.
Adds flags when starting the buildkitd daemon. They take precedence over the configuration file specified by [`--config`](#--config-file). See `buildkitd --help` for the available flags.
Specifies the configuration file for the buildkitd daemon to use. The configuration can be overridden by [`--buildkitd-flags`](#--buildkitd-flags-flags). See an [example buildkitd configuration file](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md).
#### `--driver DRIVER`
Sets the builder driver to be used. There are two available drivers, each have their own specificities.
- `docker` - Uses the builder that is built into the docker daemon. With this driver, the [`--load`](#--load) flag is implied by default on `buildx build`. However, building multi-platform images or exporting cache is not currently supported.
- `docker-container` - Uses a buildkit container that will be spawned via docker. With this driver, both building multi-platform images and exporting cache are supported. However, images built will not automatically appear in `docker images` (see [`build --load`](#--load)).
#### `--driver-opt OPTIONS`
Passes additional driver-specific options. Details for each driver:
- `docker` - No driver options
- `docker-container`
- `image` - Sets the container image to be used for running buildkit.
- `network` - Sets the network mode for running the buildkit container.
Changes the action of the command to removes a node from a builder. The builder needs to be specified with `--name` and node that is removed is set with `--node`.
Specifies the name of the builder to be created or modified. If none is specified, one will be automatically generated.
#### `--node NODE`
Specifies the name of the node to be created or modified. If none is specified, it is the name of the builder it belongs to, with an index number suffix.
#### `--platform PLATFORMS`
Sets the platforms supported by the node. It expects a comma-separated list of platforms of the form OS/architecture/variant. The node will also automatically detect the platforms it supports, but manual values take priority over the detected ones and can be used when multiple nodes support building for the same platform.
Automatically switches the current builder to the newly created one. Equivalent to running `docker buildx use $(docker buildx create ...)`.
### `buildx use NAME`
Switches the current builder instance. Build commands invoked after this command will run on a specified builder. Alternatively, a context name can be used to switch to the default builder of that context.
### `buildx inspect [NAME]`
Shows information about the current or specified builder.
Ensures that the builder is running before inspecting it. If the driver is `docker-container`, then `--bootstrap` starts the buildkit container and waits until it is operational. Bootstrapping is automatically done during build, it is thus not necessary. The same BuildKit container is used during the lifetime of the associated builder node (as displayed in `buildx ls`).
### `buildx ls`
Lists all builder instances and the nodes for each instance
Each builder has one or more nodes associated with it. The current builder’s name is marked with a `*`.
### `buildx stop [NAME]`
Stops the specified or current builder. This will not prevent buildx build to restart the builder. The implementation of stop depends on the driver.
### `buildx rm [NAME]`
Removes the specified or current builder. It is a no-op attempting to remove the default builder.
### `buildx bake [OPTIONS] [TARGET...]`
Bake is a high-level build command.
Each specified target will run in parallel as part of the build.
Options:
| Flag | Description |
| --- | --- |
| -f, --file stringArray | Build definition file
| --no-cache | Do not use cache when building the image
| --print | Print the options without building
| --progress string | Set type of progress output (auto, plain, tty). Use plain to show container output (default "auto")
| --pull | Always attempt to pull a newer version of the image
| --set stringArray | Override target value (eg: target.key=value)
#### `-f, --file FILE`
Specifies the bake definition file. The file can be a Docker Compose, JSON or HCL file. If multiple files are specified they are all read and configurations are combined. By default, if no files are specified, the following are parsed:
docker-compose.yml
docker-compose.yaml
docker-bake.json
docker-bake.override.json
docker-bake.hcl
docker-bake.override.hcl
#### `--no-cache`
Same as `build --no-cache`. Do not use cache when building the image.
#### `--print`
Prints the resulting options of the targets desired to be built, in a JSON format, without starting a build.
```
$ docker buildx bake -f docker-bake.hcl --print db
{
"target": {
"db": {
"context": "./",
"dockerfile": "Dockerfile",
"tags": [
"docker.io/tiborvass/db"
]
}
}
}
```
#### `--progress`
Same as `build --progress`. Set type of progress output (auto, plain, tty). Use plain to show container output (default "auto").
In addition to compose files, bake supports a JSON and an equivalent HCL file format for defining build groups and targets.
A target reflects a single docker build invocation with the same options that you would specify for `docker build`. A group is a grouping of targets.
Multiple files can include the same target and final build options will be determined by merging them together.
In the case of compose files, each service corresponds to a target.
A group can specify its list of targets with the `targets` option. A target can inherit build options by setting the `inherits` option to the list of targets or groups to inherit from.
Note: Design of bake command is work in progress, the user experience may change based on feedback.
Imagetools contains commands for working with manifest lists in the registry. These commands are useful for inspecting multi-platform build results.
Create creates a new manifest list based on source manifests. The source manifests can be manifest lists or single platform distribution manifests and must already exist in the registry where the new manifest is created. If only one source is specified create performs a carbon copy.
Options:
| Flag | Description |
| --- | --- |
| --append | Append to existing manifest
| --dry-run | Show final image instead of pushing
| -f, --file stringArray | Read source descriptor from file
| -t, --tag stringArray | Set reference for new image
#### `--append`
Append appends the new sources to an existing manifest list in the destination.
#### `--dry-run`
Do not push the image, just show it.
#### `-f, --file FILE`
Reads source from files. A source can be a manifest digest, manifest reference or a JSON of OCI descriptor object.
Raw prints the original JSON bytes instead of the formatted output.
# Setting buildx as default builder in Docker 19.03+
Running `docker buildx install` sets up `docker builder` command as an alias to `docker buildx`. This results in the ability to have `docker build` use the current buildx builder.
To remove this alias, you can run `docker buildx uninstall`.
See [High-level builds with Bake](https://docs.docker.com/build/bake/) for more details.
returnnil,errors.Errorf("could not create a builder instance with TLS data loaded from environment. Please use `docker context create <context-name>` to create a context for current environment and then create a builder instance with context set to <context-name>")
}
ep,err=dockerutil.GetCurrentEndpoint(dockerCli)
iferr!=nil{
returnnil,err
}
setEp=false
}
buildkitdConfigFile:=opts.BuildkitdConfigFile
ifbuildkitdConfigFile==""{
// if buildkit daemon config is not provided, check if the default one
returnerrors.Errorf("could not create a builder instance with TLS data loaded from environment. Please use `docker context create <context-name>` to create a context for current environment and then create a builder instance with `docker buildx create <context-name>`")
flags.StringVar(&options.InvokeFlag,"invoke","","Launch a monitor with executing specified command")
flags.StringVar(&options.OnFlag,"on","error","When to launch the monitor ([always, error])")
flags.StringVar(&controlOptions.Root,"root","","Specify root directory of server to connect for the monitor")
flags.BoolVar(&controlOptions.Detach,"detach",runtime.GOOS=="linux","Detach buildx server for the monitor (supported only on linux)")
flags.StringVar(&controlOptions.ServerConfig,"server-config","","Specify buildx server config file for the monitor (used only when launching new server)")
flags.StringVar(&progressMode,"progress","auto",`Set type of progress output ("auto", "plain", "tty") for the monitor. Use plain to show container output`)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
