191 Commits

Author SHA1 Message Date
Jonathan A. Sternberg
bda968ad5d
metrics: add build command duration metric
This adds a build duration metric for the build command with attributes
related to the buildx driver, the error type (if any), and which options
were used to perform the build from a subset of the options.

This also refactors some of the utility methods used by the git tool to
determine filepaths into its own separate package so they can be reused
in another place.

Also adds a test to ensure the resource is initialized correctly and
doesn't error. The otel handler logging message is suppressed on buildx
invocations so we never see the error if there's a problem with the
schema url. It's so easy to mess up the schema url when upgrading OTEL
that we need a proper test to make sure we haven't broken the
functionality.

Signed-off-by: Jonathan A. Sternberg <jonathan.sternberg@docker.com>
2024-02-14 15:58:52 -06:00
CrazyMax
02bc4e8992
build: set local dirs as frontend attributes
Set local dirs metadata if relative to VCS directory so
dockerfile path is tracked accurately in case vcs information
is not fulfilled.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
2023-12-21 11:38:11 +01:00
CrazyMax
9d8ac1ce2d
build: move solveOpt to local struct type
*client.SolveOpt in driver code is only used by build code.
For a clear separation of concerns, move it to an internal
struct type only accessible by BuildWithResultHandler func.

Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
2023-12-11 19:53:05 +01:00
Justin Chadwell
aa0aeac297 build: move solve opt out of duplicate map
This was more error prone, as opposed to the approach used prior to
616fb3e55cbc85647026f6e409af17e1011a85c4.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-11-28 10:10:50 +00:00
Justin Chadwell
616fb3e55c
build: rework node resolution
This patch reworks and updates the node resolution logic for selecting a
node from a builder.

The new implementation reworks the logic to make use of containerd's
platforms.Matcher interface instead of manually associated strings, and
additionally provides a few behavioural changes over the original
implementation, namely platforms can be matched with non-strict
semantics. e.g. i386 builds can be scheduled on an amd64 node, arm/v6
builds can be scheduled on an arm/v7 node.

We also add a new collection of tests for tracking regressions and
making the intended behaviour clearer.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-11-14 15:08:30 +01:00
Tonis Tiigi
7683ef9137
build: wait from child targets to complete before session release
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-11-07 14:41:33 -08:00
CrazyMax
de9d253f45
bake: local dockerfile support for remote definition
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-10-19 17:41:05 +02:00
CrazyMax
503d483731
fix lint issues after update
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-10-19 12:31:24 +02:00
Silvin Lubecki
22e9e3342b Revert "Introduce a client interface, to abstract from buildkit client."
This reverts commit 2aa6d52b065d0cb30b5a6b9d24d948e3f32defce.

Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
2023-10-11 22:13:23 +02:00
Silvin Lubecki
2aa6d52b06 Introduce a client interface, to abstract from buildkit client.
This will allow further enhancement like caching remote driver responses.

Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
2023-10-06 14:24:04 +02:00
CrazyMax
6c77b76b7b
Merge pull request #2038 from crazy-max/localstate-group
build: support local state group
2023-09-30 06:01:14 +02:00
CrazyMax
3dcb03452c
build: support local state group
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-09-30 01:08:06 +02:00
CrazyMax
9b7d30c9a0
build: opt to set build ref
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-09-30 01:03:00 +02:00
CrazyMax
2134a1e104
build: set target to local state
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-09-30 01:03:00 +02:00
CrazyMax
cc6957d1cc
build: move localstate logic
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-09-30 01:02:45 +02:00
CrazyMax
e018f8b6fb
Merge pull request #1994 from crazy-max/load-progress
build: sublogger to show docker load progress output
2023-09-13 08:21:44 +02:00
CrazyMax
03bedfb3c3
build: sublogger to show docker load progress output
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-09-12 22:24:29 +02:00
David Karlsson
bf5a70023c build: improve error messages for docker driver
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
Co-authored-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: David Karlsson <35727626+dvdksn@users.noreply.github.com>
2023-09-12 09:39:51 +02:00
Sebastiaan van Stijn
bafdc63b8c
migrate to github.com/distribution/reference v0.5.0
The "reference" package was moved to a separate module, which was extracted
from b9b19409cf

Also update compose-go, which also switched to distribution/reference;

full diff: https://github.com/compose-spec/compose-go/compare/v1.18.3...v1.18.4

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2023-09-06 17:41:35 +02:00
Justin Chadwell
a43d9a67c7 git: fix error wrapping to ensure internal errors are propogated
Also, tidy up the error printing, so that now we always print out the
"current commit information was not captured by the build" message,
instead of just for not locating the git binary.

Before:

	WARNING: buildx: git was not found in the system. Current commit information was not captured by the build

After:

	WARNING: current commit information was not captured by the build: git was not found in the system: <error message>

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-08-22 12:17:17 +01:00
Justin Chadwell
e5cee892ed
Merge pull request #1965 from mqasimsarfraz/qasim/oci-annotations 2023-08-08 14:36:26 +01:00
Justin Chadwell
e206c585bb build: error on attests on non-multiplatform driver
On drivers that do not support multi-platform builds (the default
`docker` driver), we do not support building attestations (unless using
the containerd store).

We need to check this feature before attempting to build using
attestations.

Also adds a test to ensure that attestations can be pushed to registries
at all, and that it adequately fails on the docker driver.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-08-04 15:28:32 +01:00
Qasim Sarfraz
18894a8e3a allow annotations for OCI image index
Signed-off-by: Qasim Sarfraz <qasimsarfraz@microsoft.com>
2023-08-03 12:08:04 +02:00
Justin Chadwell
1d177f00d2 chore: tidy splitToDriverPairs to avoid unneccessary int
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-07-19 14:43:50 +01:00
CrazyMax
8cbb7a9319
build: fix host-gateway handling
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-06-24 17:47:48 +02:00
CrazyMax
c2500ea2d8
build: prefer local image resolution for docker driver
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-06-21 11:05:06 +02:00
Tonis Tiigi
1138789f20
avoid extra client for history API detection
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-06-12 17:39:09 -07:00
Justin Chadwell
47e34f2684
Merge pull request #1813 from jedevc/oci-export-on-containerd 2023-06-07 15:19:54 +02:00
CrazyMax
0a2f35970c
build: display build details link
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-06-06 18:54:31 +02:00
Justin Chadwell
7cef021a8a
Merge pull request #1804 from jedevc/fixup-solve 2023-06-06 09:12:10 +02:00
Justin Chadwell
183a73abae build: docker exporter should instead use oci with containerd
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 15:59:55 +01:00
Justin Chadwell
cd1648192e build: rename ResultContext to ResultHandle
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 09:50:46 +01:00
Justin Chadwell
8d822fb06c build: move main solve request into main gateway call
Now, we always perform the full solve request in the main gateway call.
This ensures that progress works properly, and makes the lifetime
semantics much clearer.

NewResultContext abstracts the details of a successful/failed build, to
always return a single ResultContext, even though the details of how a
gateway is created is different:
- For a failed build, we can just keep the gateway open.
- For a successful build, we immediately open another gateway and
  re-evaluate the build definition in that gateway. This should give an
  instant cache hit (since the build was just successful).

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-31 09:46:46 +01:00
CrazyMax
8c65e4fc1d
driver: add context to Features interface
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2023-05-30 20:13:18 +02:00
Justin Chadwell
2dae553d18 build: update outline fallback image
Subrequests have been included in docker/dockerfile:1.5 labs, so we can
update the fallback to point to this release.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-26 10:38:38 +01:00
Justin Chadwell
91c17f25fb build: tidy up print func
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-05-26 10:38:38 +01:00
Brian Goff
98efe7af10 Support for passing through raw buildkit policies
This adds an env var which can be used to pass in a path to a file to
read a buildkit source poliy from.

This is applied to any build is executed with the env set.
It is also applied to bakes (which are calling build behind the scenes).

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2023-05-03 23:07:44 +00:00
Justin Chadwell
75ed3e296b
Merge pull request #1737 from jedevc/improved-controller-progress
controller: refactor progress api
2023-04-26 10:17:00 +01:00
Tõnis Tiigi
28e6995f7c
Merge pull request #1735 from tonistiigi/localstate
build: support for saving local state by build ref
2023-04-25 15:17:08 -07:00
Justin Chadwell
2402607846 build: use gateway's solve context to allow cancelling getResultAt
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-21 10:52:03 +01:00
Kohei Tokunaga
b3340cc7ba
Enable to create container from error
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2023-04-15 15:36:33 +09:00
CrazyMax
200058b505
build: support for saving local state by build ref
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2023-04-12 21:37:03 -07:00
Justin Chadwell
589d4e4cf5 build: use buildkit's gitutil package to detect remote files
BuildKit's gitutil package behaves slightly differently than moby's
urlutil, so we should rely on BuildKit's gitutil when detecting URLs to
avoid cases of accidentally producing invalid build requests that can
confuse users.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-04-03 10:23:59 +01:00
Justin Chadwell
c6cdcb02cf build: replace removed buildkit BUILDINFO field
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-03-30 10:24:46 +01:00
Justin Chadwell
5d4223e4f8 build: move SOURCE_DATE_EPOCH parsing into option generation
This allows the build package code to become more generic, and also
ensures that when the environment variables are not propogated (in the
case of the remote controller), that we can still correctly set
SOURCE_DATE_EPOCH.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-03-14 10:21:32 +00:00
Kohei Tokunaga
e8f55a3cf7 monitor: Enable to exec into the container
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2023-03-02 19:30:29 +09:00
Justin Chadwell
c2e11196dd controller: handle attestation options across api
We can perform all attestation processing, handling how the sbom and
provenance arguments interact on the client, while applying defaults on
the server.

Additionally, this allows us to start pulling fields out of CommonOpts.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-02-23 15:46:37 +00:00
Justin Chadwell
0b8f0264b0 controller: move image id file writing to client
Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-02-23 15:46:37 +00:00
Justin Chadwell
5c31d855fd controller: return solve response through api
Now clients can access the result of the solve, specifically the image
id output. This is a useful refactor, as well as being required if we
want to allow bake to invoke through the controller api.

This also allows us to remove the quiet option from the API, since we
can compute the required progress type outside of the controller, and
can print the image id from the result of the solve.

As a follow-up, we should also be able to remove the image id file
output from the controller api, now that the client has access to it.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-02-23 15:46:37 +00:00
Justin Chadwell
90d7fb5e77 controller: strongly type the controller api
Strongly typing the API allows us to perform all command line parsing
fully on the client-side, where we have access to the client local
directory and all the client environment variables, which may not be
available on the remote server.

Additionally, the controller api starts to look a lot like
build.Options, so at some point in the future there may be an
oppportunity to merge the two, which would allow both build and bake to
execute through the controller, instead of needing to maintain multiple
code paths.

Signed-off-by: Justin Chadwell <me@jedevc.com>
2023-02-23 15:43:15 +00:00