docker/buildx
1
0
Fork 0
mirror of https://gitea.com/Lydanne/buildx.git synced 2025-07-23 19:58:03 +08:00
Code Issues Packages Projects Releases 1 Wiki Activity

updated prose and structure for driver docs

Browse Source 
Signed-off-by: David Karlsson <david.karlsson@docker.com>
This commit is contained in:
David Karlsson
2022-09-29 11:13:20 +02:00
parent e98c252490
commit d030fcc076
5 changed files with 483 additions and 328 deletions
Download Patch File Download Diff File Expand all files Collapse all files

293
docs/guides/drivers/kubernetes.md
View File

@@ -1,89 +1,65 @@
# Kubernetes driver
The buildx kubernetes driver allows connecting your local development or ci
environments to your kubernetes cluster to allow access to more powerful
and varied compute resources.
The Buildx Kubernetes driver allows connecting your local development or CI
environments to your Kubernetes cluster to allow access to more powerful and
varied compute resources.
This guide assumes you already have an existing kubernetes cluster - if you don't already
have one, you can easily follow along by installing
[minikube](https://minikube.sigs.k8s.io/docs/).
## Synopsis
Before connecting buildx to your cluster, you may want to create a dedicated
namespace using `kubectl` to keep your buildx-managed resources separate. You
can call your namespace anything you want, or use the existing `default`
namespace, but we'll create a `buildkit` namespace for now:
```console
$ kubectl create namespace buildkit
```
Then create a new buildx builder:
Run the following command to create a new builder, named `container`, that uses
the Docker container driver:
```console
$ docker buildx create \
--bootstrap \
--name=kube \
--driver=kubernetes \
--driver-opt=namespace=buildkit
--driver-opt=[key=value,...]
```
This assumes that the kubernetes cluster you want to connect to is currently
accessible via the kubectl command, with the `KUBECONFIG` environment variable
[set appropriately](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable)
if neccessary.
The following table describes the available driver-specific options that you can
pass to `--driver-opt`:
You should now be able to see the builder in the list of buildx builders:
| Parameter | Value | Default | Description |
| ----------------- | ---------------- | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `image` | String | | Sets the image to use for running BuildKit. |
| `namespace` | String | Namespace in current Kubernetes context | Sets the Kubernetes namespace. |
| `replicas` | Integer | 1 | Sets the number of Pod replicas to create. See [scaling BuildKit][1] |
| `requests.cpu` | CPU units | | Sets the request CPU value specified in units of Kubernetes CPU. For example `requests.cpu=100m` or `requests.cpu=2` |
| `requests.memory` | Memory size | | Sets the request memory value specified in bytes or with a valid suffix. For example `requests.memory=500Mi` or `requests.memory=4G` |
| `limits.cpu` | CPU units | | Sets the limit CPU value specified in units of Kubernetes CPU. For example `requests.cpu=100m` or `requests.cpu=2` |
| `limits.memory` | Memory size | | Sets the limit memory value specified in bytes or with a valid suffix. For example `requests.memory=500Mi` or `requests.memory=4G` |
| `nodeselector` | CSV string | | Sets the pod's `nodeSelector` label(s). See [node assignment][2]. |
| `tolerations` | CSV string | | Configures the pod's taint toleration. See [node assignment][2]. |
| `rootless` | `true\|false` | `false` | Run the container as a non-root user. See [rootless mode][3]. |
| `loadbalance` | `sticky\|random` | `sticky` | Load-balancing strategy. If set to `sticky`, the pod is chosen using the hash of the context path. |
| `qemu.install` | `true\|false` | | Install QEMU emulation for multi platforms support. See [QEMU][4]. |
| `qemu.image` | String | `tonistiigi/binfmt:latest` | Sets the QEMU emulation image. See [QEMU][4]. |
```console
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
kube kubernetes
kube0-6977cdcb75-k9h9m running linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default * docker
default default running linux/amd64, linux/386
```
[1]: #scaling-buildkit
[2]: #node-assignment
[3]: #rootless-mode
[4]: #qemu
The buildx driver creates the neccessary resources on your cluster in the
specified namespace (in this case, `buildkit`), while keeping your
driver configuration locally. You can see the running pods with:
## Scaling BuildKit
```console
$ kubectl -n buildkit get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
kube0 1/1 1 1 32s
$ kubectl -n buildkit get pods
NAME READY STATUS RESTARTS AGE
kube0-6977cdcb75-k9h9m 1/1 Running 0 32s
```
You can use your new builder by including the `--builder` flag when running
buildx commands. For example (replacing `<user>` and `<image>` with your Docker
Hub username and desired image output respectively):
```console
$ docker buildx build . \
--builder=kube \
-t <user>/<image> \
--push
```
## Scaling Buildkit
One of the main advantages of the kubernetes builder is that you can easily
scale your builder up and down to handle increased build load. These controls
are exposed via the following options:
One of the main advantages of the Kubernetes driver is that you can scale the
number of builder replicas up and down to handle increased build load. Scaling
is configurable using the following driver options:
- `replicas=N`
- This scales the number of buildkit pods to the desired size. By default,
only a single pod will be created, but increasing this allows taking of
advantage of multiple nodes in your cluster.
This scales the number of BuildKit pods to the desired size. By default, it
only creates a single pod. Increasing the number of replicas lets you take
advantage of multiple nodes in your cluster.
- `requests.cpu`, `requests.memory`, `limits.cpu`, `limits.memory`
- These options allow requesting and limiting the resources available to each
buildkit pod according to the official kubernetes documentation
[here](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/).
For example, to create 4 replica buildkit pods:
These options allow requesting and limiting the resources available to each
BuildKit pod according to the official Kubernetes documentation
[here](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/).
For example, to create 4 replica BuildKit pods:
```console
$ docker buildx create \
@@ -93,7 +69,7 @@ $ docker buildx create \
--driver-opt=namespace=buildkit,replicas=4
```
Listing the pods, we get:
Listing the pods, you get this:
```console
$ kubectl -n buildkit get deployments
@@ -107,28 +83,54 @@ kube0-6977cdcb75-rkc6b 1/1 Running 0 8s
kube0-6977cdcb75-vb4ks 1/1 Running 0 8s
kube0-6977cdcb75-z4fzs 1/1 Running 0 8s
```
Additionally, you can use the `loadbalance=(sticky|random)` option to control
the load-balancing behavior when there are multiple replicas. While `random`
should selects random nodes from the available pool, which should provide
better balancing across all replicas, `sticky` (the default) attempts to
connect the same build performed multiple times to the same node each time,
ensuring better local cache utilization.
For more information on scalability, see the options for [buildx create](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver-opt).
Additionally, you can use the `loadbalance=(sticky|random)` option to control
the load-balancing behavior when there are multiple replicas. `random` selects
random nodes from the node pool, providing an even workload distribution across
replicas. `sticky` (the default) attempts to connect the same build performed
multiple times to the same node each time, ensuring better use of local cache.
For more information on scalability, see the options for
[buildx create](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver-opt).
## Node assignment
The Kubernetes driver allows you to control the scheduling of BuildKit pods
using the `nodeSelector` and `tolerations` driver options.
The value of the `nodeSelector` parameter is a comma-separated string of
key-value pairs, where the key is the node label and the value is the label
text. For example: `"nodeselector=kubernetes.io/arch=arm64"`
The `tolerations` parameter is a semicolon-separated list of taints. It accepts
the same values as the Kubernetes manifest. Each `tolerations` entry specifies a
taint key and the value, operator, or effect. For example:
`"tolerations=key=foo,value=bar;key=foo2,operator=exists;key=foo3,effect=NoSchedule"`
The syntax for these parameters is slightly different compared to other driver
options. You must wrap both `nodeSelector` and `tolerations` in double quotes.
For example:
```console
$ docker buildx create \
--bootstrap \
--name=kube \
--driver=kubernetes \
--driver-opt="nodeselector=label=value","tolerations=key=key1,value=value1"
```
## Multi-platform builds
The kubernetes buildx driver has support for creating [multi-platform images](https://docs.docker.com/build/building/multi-platform/),
for easily building for multiple platforms at once.
The Buildx Kubernetes driver has support for creating
[multi-platform images](https://docs.docker.com/build/building/multi-platform/),
either using QEMU or by leveraging the native architecture of nodes.
### QEMU
Like the other containerized driver `docker-container`, the kubernetes driver
also supports using [QEMU](https://www.qemu.org/) (user mode) to build
non-native platforms. If using a default setup like above, no extra setup
should be needed, you should just be able to start building for other
architectures, by including the `--platform` flag.
Like the `docker-container` driver, the Kubernetes driver also supports using
[QEMU](https://www.qemu.org/) (user mode) to build images for non-native
platforms. Include the `--platform` flag and specify which platforms you want to
output to.
For example, to build a Linux image for `amd64` and `arm64`:
@@ -141,13 +143,14 @@ $ docker buildx build . \
```
> **Warning**
> QEMU performs full-system emulation of non-native platforms, which is *much*
>
> QEMU performs full-system emulation of non-native platforms, which is much
> slower than native builds. Compute-heavy tasks like compilation and
> compression/decompression will likely take a large performance hit.
Note, if you're using a custom buildkit image using the `image=<image>` driver
option, or invoking non-native binaries from within your build, you may need to
explicitly enable QEMU using the `qemu.install` option during driver creation:
Using a custom BuildKit image or invoking non-native binaries in builds may
require that you explicitly turn on QEMU using the `qemu.install` option when
creating the builder:
```console
$ docker buildx create \
@@ -159,12 +162,12 @@ $ docker buildx create \
### Native
If you have access to cluster nodes of different architectures, we can
configure the kubernetes driver to take advantage of these for native builds.
To do this, we need to use the `--append` feature of `docker buildx create`.
If you have access to cluster nodes of different architectures, the Kubernetes
driver can take advantage of these for native builds. To do this, use the
`--append` flag of `docker buildx create`.
To start, we can create our builder with explicit support for a single
architecture, `amd64`:
First, create your builder with explicit support for a single architecture, for
example `amd64`:
```console
$ docker buildx create \
@@ -176,13 +179,13 @@ $ docker buildx create \
--driver-opt=namespace=buildkit,nodeselector="kubernetes.io/arch=amd64"
```
This creates a buildx builder `kube` containing a single builder node `builder-amd64`.
Note that the buildx concept of a node is not the same as the kubernetes
concept of a node - the buildx node in this case could connect multiple
kubernetes nodes of the same architecture together.
This creates a Buildx builder named `kube`, containing a single builder node
`builder-amd64`. Note that the Buildx concept of a node isn't the same as the
Kubernetes concept of a node. A Buildx node in this case could connect multiple
Kubernetes nodes of the same architecture together.
With our `kube` driver created, we can now introduce another architecture into
the mix, for example, like before we can use `arm64`:
With the `kube` builder created, you can now introduce another architecture into
the mix using `--append`. For example, to add `arm64`:
```console
$ docker buildx create \
@@ -200,7 +203,7 @@ If you list builders now, you should be able to see both nodes present:
```console
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
kube kubernetes
kube kubernetes
builder-amd64 kubernetes:///kube?deployment=builder-amd64&kubeconfig= running linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/386
builder-arm64 kubernetes:///kube?deployment=builder-arm64&kubeconfig= running linux/arm64*
```
@@ -217,10 +220,11 @@ architectures that you want to support.
## Rootless mode
The kubernetes driver supports rootless mode. For more information on how
rootless mode works, and it's requirements, see [here](https://github.com/moby/buildkit/blob/master/docs/rootless.md).
The Kubernetes driver supports rootless mode. For more information on how
rootless mode works, and it's requirements, see
[here](https://github.com/moby/buildkit/blob/master/docs/rootless.md).
To enable it in your cluster, you can use the `rootless=true` driver option:
To turn it on in your cluster, you can use the `rootless=true` driver option:
```console
$ docker buildx create \
@@ -231,6 +235,89 @@ $ docker buildx create \
This will create your pods without `securityContext.privileged`.
Requires Kubernetes version 1.19 or later. Using Ubuntu as the host kernel is
recommended.
## Guide: Creating a Buildx builder in Kubernetes
This guide shows you how to:
- Create a namespace for your Buildx resources
- Create a Kubernetes builder.
- List the available builders
- Build an image using your Kubernetes builders
Prerequisites:
- You have an existing Kubernetes cluster. If you don't already have one, you
can follow along by installing [minikube](https://minikube.sigs.k8s.io/docs/).
- The cluster you want to connect to is accessible via the `kubectl` command,
with the `KUBECONFIG` environment variable
[set appropriately](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/#set-the-kubeconfig-environment-variable)
if necessary.
1. Create a `buildkit` namespace.
Creating a separate namespace helps keep your Buildx resources separate from
other resources in the cluster.
```console
$ kubectl create namespace buildkit
namespace/buildkit created
```
2. Create a new Buildx builder with the Kubernetes driver:
```console
# Remember to specify the namespace in driver options
$ docker buildx create \
--bootstrap \
--name=kube \
--driver=kubernetes \
```
3. List available Buildx builders using `docker buildx ls`
```console
$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS PLATFORMS
kube kubernetes
kube0-6977cdcb75-k9h9m running linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default * docker
default default running linux/amd64, linux/386
```
4. Inspect the running pods created by the Buildx driver with `kubectl`.
```console
$ kubectl -n buildkit get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
kube0 1/1 1 1 32s
$ kubectl -n buildkit get pods
NAME READY STATUS RESTARTS AGE
kube0-6977cdcb75-k9h9m 1/1 Running 0 32s
```
The buildx driver creates the necessary resources on your cluster in the
specified namespace (in this case, `buildkit`), while keeping your driver
configuration locally.
5. Use your new builder by including the `--builder` flag when running buildx
commands. For example: :
```console
# Replace <registry> with your Docker username
# and <image> with the name of the image you want to build
docker buildx build . \
--builder=kube \
-t <registry>/<image> \
--push
```
That's it! You've now built an image from a Kubernetes pod, using Buildx!
## Further reading
For more information on the kubernetes driver, see the [buildx reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver).
For more information on the Kubernetes driver, see the
[buildx reference](https://docs.docker.com/engine/reference/commandline/buildx_create/#driver).