Merge pull request #1233 from jedevc/drivers-guides

Enhanced driver guides

README.md

@@ -40,7 +40,7 @@ Key features:
   - [Using a custom registry configuration](docs/guides/custom-registry-config.md)
   - [OpenTelemetry support](docs/guides/opentelemetry.md)
   - [Registry mirror](docs/guides/registry-mirror.md)
-  - [Remote builder](docs/guides/remote-builder.md)
+  - [Drivers](docs/guides/drivers/index.md)
   - [Resource limiting](docs/guides/resource-limiting.md)
 - [Reference](docs/reference/buildx.md)
   - [`buildx bake`](docs/reference/buildx_bake.md)
@@ -185,27 +185,17 @@ specifying target platform. In addition, Buildx also supports new features that
 are not yet available for regular `docker build` like building manifest lists,
 distributed caching, and exporting build results to OCI image tarballs.
 
-Buildx is supposed to be flexible and can be run in different configurations
+Buildx is flexible and can be run in different configurations that are exposed
-that are exposed through a driver concept. Currently, we support:
+through various "drivers". Each driver defines how and where a build should
+run, and have different feature sets.
 
-- a [`docker` driver](docs/reference/buildx_create.md#docker-driver) that uses
+We currently support the following drivers:
-  the BuildKit library bundled into the Docker daemon binary,
+- The `docker` driver ([reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
-- a [`docker-container` driver](docs/reference/buildx_create.md#docker-container-driver)
+- The `docker-container` driver ([guide](./docker-container.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
-  that automatically launches BuildKit inside a Docker container,
+- The `kubernetes` driver ([guide](./kubernetes.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
-- a [`kubernetes` driver](docs/reference/buildx_create.md#kubernetes-driver) to
+- The `remote` driver ([guide](./remote.md))
-  spin up pods with defined BuildKit container image to build your images.
-- a [`remote` driver](docs/reference/buildx_create.md#remote-driver) to
-  connect to manually provisioned and managed buildkitd instances.
 
-We plan to add more drivers in the future.
+For more information on drivers, see the [drivers guide](docs/guides/drivers/index.md).
-
-The user experience of using buildx is very similar across drivers, but there
-are some features that are not currently supported by the `docker` driver,
-because the BuildKit library bundled into docker daemon currently uses a
-different storage component. In contrast, all images built with `docker` driver
-are automatically added to the `docker images` view by default, whereas when
-using other drivers the method for outputting an image needs to be selected
-with `--output`.
 
 ## Working with builder instances
 

docs/guides/drivers/docker-container.md

@@ -0,0 +1,75 @@
+# Docker container driver
+
+The buildx docker-container driver allows creation of a managed and
+customizable BuildKit environment inside a dedicated Docker container.
+
+Using the docker-container driver has a couple of advantages over the basic
+docker driver. Firstly, we can manually override the version of buildkit to
+use, meaning that we can access the latest and greatest features as soon as
+they're released, instead of waiting to upgrade to a newer version of Docker.
+Additionally, we can access more complex features like multi-architecture
+builds and the more advanced cache exporters, which are currently unsupported
+in the default docker driver.
+
+We can easily create a new builder that uses the docker-container driver:
+
+```console
+$ docker buildx create --name container --driver docker-container
+container
+```
+
+We should then be able to see it on our list of available builders:
+
+```console
+$ docker buildx ls
+NAME/NODE       DRIVER/ENDPOINT      STATUS   BUILDKIT PLATFORMS
+container       docker-container                       
+  container0    desktop-linux        inactive          
+default         docker                                 
+  default       default              running  20.10.17 linux/amd64, linux/386
+```
+
+If we trigger a build, the appropriate `moby/buildkit` image will be pulled
+from [Docker Hub](https://hub.docker.com/u/moby/buildkit), the image started,
+and our build submitted to our containerized build server.
+
+```console
+$ docker buildx build -t <image> --builder=container .
+WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load
+#1 [internal] booting buildkit
+#1 pulling image moby/buildkit:buildx-stable-1
+#1 pulling image moby/buildkit:buildx-stable-1 1.9s done
+#1 creating container buildx_buildkit_container0
+#1 creating container buildx_buildkit_container0 0.5s done
+#1 DONE 2.4s
+...
+```
+
+Note the warning "Build result will only remain in the build cache" - unlike
+the `docker` driver, the built image must be explicitly loaded into the local
+image store. We can use the `--load` flag for this:
+
+```console
+$ docker buildx build --load -t <image> --builder=container .
+...
+ => exporting to oci image format                                                                                                      7.7s
+ => => exporting layers                                                                                                                4.9s
+ => => exporting manifest sha256:4e4ca161fa338be2c303445411900ebbc5fc086153a0b846ac12996960b479d3                                      0.0s 
+ => => exporting config sha256:adf3eec768a14b6e183a1010cb96d91155a82fd722a1091440c88f3747f1f53f                                        0.0s 
+ => => sending tarball                                                                                                                 2.8s 
+ => importing to docker   
+```
+
+The image should then be available in the image store:
+
+```console
+$ docker image ls
+REPOSITORY                       TAG               IMAGE ID       CREATED             SIZE
+<image>                          latest            adf3eec768a1   2 minutes ago       197MB
+```
+
+## Further reading
+
+For more information on the docker-container driver, see the [buildx reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver).
+
+<!--- FIXME: for 0.9, make reference link relative --->

docs/guides/drivers/index.md

@@ -0,0 +1,41 @@
+# Buildx drivers overview
+
+The buildx client connects out to the BuildKit backend to execute builds -
+Buildx drivers allow fine-grained control over management of the backend, and
+supports several different options for where and how BuildKit should run.
+
+Currently, we support the following drivers:
+
+- The `docker` driver, that uses the BuildKit library bundled into the Docker
+  daemon.
+  ([reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
+- The `docker-container` driver, that launches a dedicated BuildKit container
+  using Docker, for access to advanced features.
+  ([guide](./docker-container.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
+- The `kubernetes` driver, that launches dedicated BuildKit pods in a
+  remote Kubernetes cluster, for scalable builds.
+  ([guide](./kubernetes.md), [reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver))
+- The `remote` driver, that allows directly connecting to a manually managed
+  BuildKit daemon, for more custom setups.
+  ([guide](./remote.md))
+
+<!--- FIXME: for 0.9, make links relative, and add reference link for remote --->
+
+To create a new builder that uses one of the above drivers, you can use the
+[`docker buildx create`](https://docs.docker.com/engine/reference/commandline/buildx_create/) command:
+
+```console
+$ docker buildx create --name=<builder-name> --driver=<driver> --driver-opt=<driver-options>
+```
+
+The build experience is very similar across drivers, however, there are some
+features that are not evenly supported across the board, notably, the `docker`
+driver does not include support for certain output/caching types.
+
+| Feature                       |    `docker`     | `docker-container` | `kubernetes` |        `remote`        |
+| :---------------------------- | :-------------: | :----------------: | :----------: | :--------------------: |
+| **Automatic `--load`**        |        ✅        |         ❌          |      ❌       |           ❌            |
+| **Cache export**              | ❔ (inline only) |         ✅          |      ✅       |           ✅            |
+| **Docker/OCI tarball output** |        ❌        |         ✅          |      ✅       |           ✅            |
+| **Multi-arch images**         |        ❌        |         ✅          |      ✅       |           ✅            |
+| **BuildKit configuration**    |        ❌        |         ✅          |      ✅       | ❔ (managed externally) |

docs/guides/drivers/kubernetes.md

@@ -1,4 +1,4 @@
-# Kubernetes builder
+# Kubernetes driver
 
 The buildx kubernetes driver allows connecting your local development or ci
 environments to your kubernetes cluster to allow access to more powerful
@@ -230,3 +230,9 @@ $ docker buildx create \
 ```
 
 This will create your pods without `securityContext.privileged`.
+
+## Further reading
+
+For more information on the kubernetes driver, see the [buildx reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver).
+
+<!--- FIXME: for 0.9, make reference link relative --->

docs/guides/drivers/remote.md

@@ -1,4 +1,4 @@
-# Remote builder
+# Remote driver
 
 The buildx remote driver allows for more complex custom build workloads that
 allow users to connect to external buildkit instances. This is useful for
@@ -174,3 +174,5 @@ $ docker buildx create \
   --driver remote \
   kube-pod://buildkitd-XXXXXXXXXX-xxxxx
 ```
+
+<!--- FIXME: for 0.9, add further reading section with link to reference --->