Compare commits

...

46 Commits

Author SHA1 Message Date
Tõnis Tiigi
6db68d0295 Merge pull request #155 from tiborvass/vendor-buildkit
vendor: update buildkit to docker-19.03 (ae10b292)
2019-09-27 10:36:16 -07:00
Tibor Vass
abe8ba769e vendor: update buildkit to docker-19.03 (ae10b292)
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-09-27 17:18:25 +00:00
Tõnis Tiigi
96fb17b711 Merge pull request #154 from tiborvass/fix-149
build: fix scoping issue in closure inside loop
2019-09-26 11:32:04 -07:00
Tibor Vass
63e5633d62 build: fix scoping issue in closure inside loop
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-09-26 18:01:29 +00:00
Tibor Vass
299d41660b Merge pull request #153 from tonistiigi/stdin-dockerfile
build: fix stdin dockerfile filename
2019-09-26 10:53:28 -07:00
Tibor Vass
1ec87b7beb Merge pull request #152 from tonistiigi/stream-input
build: use correct in-memory input
2019-09-26 10:45:55 -07:00
Tonis Tiigi
0475107882 build: fix stdin dockerfile filename
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-09-26 09:17:04 -07:00
Tonis Tiigi
75f8d7ebb5 build: use correct in-memory input
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-09-26 09:10:39 -07:00
Tibor Vass
7c97854b6f Merge pull request #144 from droopy4096/master
Add FOSSA checks to Jenkins CI
2019-09-17 14:56:00 -07:00
Dmytro Makovey
5f4d4a87f7 Add FOSSA checks to Jenkins CI
Signed-off-by: Dmytro Makovey <dmytro.makovey@docker.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-09-17 21:27:29 +00:00
Tõnis Tiigi
c1ce7300d5 Merge pull request #146 from gfrancesco/master
README typo
2019-09-17 10:19:34 -07:00
gfrancesco
e118c4d8e9 UPD: Readme typo 2019-09-17 18:13:16 +02:00
Tibor Vass
5fe779703d Merge pull request #134 from tonistiigi/group-merge
bake: merge targets on same groups
2019-09-05 17:15:01 -07:00
Tonis Tiigi
15a5a42eb1 bake: merge targets on same groups
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-19 15:48:42 -07:00
Tõnis Tiigi
5b974158f9 Merge pull request #131 from gracenoah/patch-1
Fix some quotes in the readme
2019-08-14 12:16:13 -07:00
gracenoah
1c0a7f14e8 Fix some quotes in the readme 2019-08-13 14:27:10 +02:00
Tibor Vass
7ec8912591 Merge pull request #125 from tiborvass/docs-allow
Document build --allow
2019-08-01 18:18:00 -07:00
Tibor Vass
83da6a3378 docs: crosslink buildkitd-flags and config flags in create
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-01 17:56:05 -07:00
Tibor Vass
cad02a4681 docs: document build --allow
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-01 17:56:05 -07:00
Tõnis Tiigi
c967f1d570 Merge pull request #124 from tiborvass/update-docs
Update docs
2019-08-01 16:41:26 -07:00
Tibor Vass
be3efc979b docs: add documentation for --buildkitd-flags, --config, --driver-opt on create
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-01 16:15:11 -07:00
Tibor Vass
5c5f54c6d6 docs: Update install instructions with Docker CE 19.03
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-08-01 15:23:02 -07:00
Tibor Vass
6f8f04e1f8 Merge pull request #122 from tonistiigi/custom-image
driver: allow setting driver opts
2019-08-01 11:41:49 -07:00
Tonis Tiigi
afd821010d docker-container: allow setting custom buildkit image
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-31 22:46:37 -07:00
Tonis Tiigi
bcc882cbf1 docker-container: allow using host network
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-31 17:42:49 -07:00
Tonis Tiigi
75b80c277f driver: allow setting driver opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-31 17:25:25 -07:00
Tibor Vass
096d1befc9 Merge pull request #104 from tonistiigi/entitlements
build: add allowed entitlements
2019-07-31 15:36:13 -07:00
Tibor Vass
2bf6187a88 Merge pull request #121 from tonistiigi/config
driver: allow setting buildkit config file
2019-07-31 15:21:17 -07:00
Tonis Tiigi
8ed8795268 driver: allow setting buildkit config file
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Co-Authored-By: Tibor Vass <tiborvass@users.noreply.github.com>
2019-07-31 15:08:26 -07:00
Tõnis Tiigi
6e32ea3418 Merge pull request #118 from tiborvass/bake-no-cache-pull
bake: honor --no-cache and --pull
2019-07-31 10:59:59 -07:00
Tibor Vass
8b2171f78a bake: honor --no-cache and --pull
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-07-30 19:39:01 -07:00
Tibor Vass
92f1234aaa Merge pull request #116 from tonistiigi/build-arg-default
build: load default build args from env
2019-07-30 19:20:09 -07:00
Tibor Vass
73645c8348 Merge pull request #117 from tonistiigi/compose-env
bake: replace env in compose files
2019-07-30 19:14:21 -07:00
Tonis Tiigi
662c0768cb bake: replace env in compose files
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-30 16:44:05 -07:00
Tonis Tiigi
43150ef849 build: load default build args from env
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-30 16:32:36 -07:00
Tibor Vass
3f18b659a0 Merge pull request #102 from tonistiigi/buildkitd-flags
driver: allow configuring buildkitd flags
2019-07-09 17:27:17 -07:00
Tonis Tiigi
6b81b0bed6 build: add allowed entitlements
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-08 15:59:53 -07:00
Tonis Tiigi
f0af89a204 driver: allow configuring buildkitd flags
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-08 15:29:43 -07:00
Tõnis Tiigi
550c2b9042 Merge pull request #100 from FernandoMiguel/patch-1
add chmod
2019-07-06 12:38:08 -07:00
Fernando Miguel
c8cda08209 add chmod 2019-07-05 12:14:40 +01:00
Tõnis Tiigi
2b03339235 Merge pull request #93 from zelahi/enable-fossa-scan
[TAR-853] ADDED .fossa file for fossa scans
2019-06-17 09:21:16 -07:00
zelahi
6e1fd0eab6 ADDED .fossa file for fossa scans 2019-06-14 10:49:12 -07:00
Tõnis Tiigi
5336e74bd4 Merge pull request #89 from khs1994/master
Fix Dockerfile format
2019-06-05 13:56:29 -07:00
Tõnis Tiigi
afeaed790f Merge pull request #86 from AkihiroSuda/driver-ls
Put driver names to create --help
2019-06-05 13:55:44 -07:00
khs1994
aed531a8a9 Fix Dockerfile format
Signed-off-by: Kang HuaiShuai <khs1994@khs1994.com>
2019-06-04 17:43:39 +08:00
Akihiro Suda
eee78c6c10 Put driver names to create --help
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2019-06-02 00:02:20 +09:00
48 changed files with 701 additions and 245 deletions

14
.fossa.yml Executable file
View File

@@ -0,0 +1,14 @@
# Generated by FOSSA CLI (https://github.com/fossas/fossa-cli)
# Visit https://fossa.com to learn more
version: 2
cli:
server: https://app.fossa.io
fetcher: custom
project: git@github.com:docker/buildx
analyze:
modules:
- name: github.com/docker/buildx/cmd/buildx
type: go
target: github.com/docker/buildx/cmd/buildx
path: cmd/buildx

29
Jenkinsfile vendored Normal file
View File

@@ -0,0 +1,29 @@
@Library('jps')
_
pipeline {
agent {
node {
label 'ubuntu-1804-overlay2'
}
}
options {
disableConcurrentBuilds()
}
stages {
stage("FOSSA Analyze") {
steps {
withCredentials([string(credentialsId: 'fossa-api-key', variable: 'FOSSA_API_KEY')]) {
withGithubStatus('FOSSA.scan') {
labelledShell returnStatus: false, returnStdout: true, label: "make fossa-analyze",
script:'make -f Makefile.fossa BRANCH_NAME=${BRANCH_NAME} fossa-analyze'
labelledShell returnStatus: false, returnStdout: true, label: "make fossa-test",
script: 'make -f Makefile.fossa BRANCH_NAME=${BRANCH_NAME} fossa-test'
}
}
}
}
}
}

18
Makefile.fossa Normal file
View File

@@ -0,0 +1,18 @@
REPO_PATH?=docker/buildx
BUILD_ANALYZER?=docker/fossa-analyzer
FOSSA_OPTS?=--option all-tags:true --option allow-unresolved:true --no-ansi
fossa-analyze:
docker run -i --rm -e FOSSA_API_KEY=$(FOSSA_API_KEY) \
-v $(CURDIR)/$*:/go/src/github.com/$(REPO_PATH) \
-w /go/src/github.com/$(REPO_PATH) \
-e GO111MODULE=on \
$(BUILD_ANALYZER) analyze $(FOSSA_OPTS) --branch $(BRANCH_NAME)
# This command is used to run the fossa test command
fossa-test:
docker run -i --rm -e FOSSA_API_KEY=$(FOSSA_API_KEY) \
-v $(CURDIR)/$*:/go/src/github.com/$(REPO_PATH) \
-w /go/src/github.com/$(REPO_PATH) \
-e GO111MODULE=on \
$(BUILD_ANALYZER) test --debug

120
README.md
View File

@@ -41,30 +41,21 @@ _buildx is Tech Preview_
# Installing
Using `buildx` as a docker CLI plugin requires using Docker 19.03.0 beta. A limited set of functionality works with older versions of Docker when invoking the binary directly.
Using `buildx` as a docker CLI plugin requires using Docker 19.03. A limited set of functionality works with older versions of Docker when invoking the binary directly.
### Docker Desktop (Edge)
### Docker CE
`buildx` is included with Docker Desktop Edge builds since 19.03.0-beta3.
For more information see https://docs.docker.com/docker-for-mac/edge-release-notes/
### Docker CE nightly builds
`buildx` comes bundled with the Docker CE nightly builds.
- Mac: https://download.docker.com/mac/static/nightly/
- Linux:
```
$ # uncomment next line to uninstall previous Docker CE installation if present
$ # apt purge docker-ce docker-ce-cli
$ curl -fsSL https://get.docker.com/ -o docker-install.sh
$ CHANNEL=nightly sh docker-install.sh
```
`buildx` comes bundled with Docker CE starting with 19.03, but requires experimental mode to be enabled on the Docker CLI.
To enable it, `"experimental": "enabled"` can be added to the CLI configuration file `~/.docker/config.json`. An alternative is to set the `DOCKER_CLI_EXPERIMENTAL=enabled` environment variable.
### Binary release
Download the latest binary release from https://github.com/docker/buildx/releases/latest and copy it to `~/.docker/cli-plugins` folder with name `docker-buildx`.
Change the permission to execute:
```sh
chmod a+x ~/.docker/cli-plugins/docker-buildx
```
After installing you can run `docker buildx` to see the new commands.
@@ -142,7 +133,7 @@ $ docker buildx build --platform linux/amd64,linux/arm64 .
Finally, depending on your project, the language that you use may have good support for cross-compilation. In that case, multi-stage builds in Dockerfiles can be effectively used to build binaries for the platform specified with `--platform` using the native architecture of the build node. List of build arguments like `BUILDPLATFORM` and `TARGETPLATFORM` are available automatically inside your Dockerfile and can be leveraged by the processes running as part of your build.
```
FROM --platform $BUILDPLATFORM golang:alpine AS build
FROM --platform=$BUILDPLATFORM golang:alpine AS build
ARG TARGETPLATFORM
ARG BUILDPLATFORM
RUN echo "I am running on $BUILDPLATFORM, building for $TARGETPLATFORM" > /log
@@ -174,6 +165,7 @@ Options:
| Flag | Description |
| --- | --- |
| --add-host [] | Add a custom host-to-IP mapping (host:ip)
| --allow [] | Allow extra privileged entitlement, e.g. network.host, security.insecure
| --build-arg [] | Set build-time variables
| --cache-from [] | External cache sources (eg. user/app:cache, type=local,src=path/to/dir)
| --cache-to [] | Cache export destinations (eg. user/app:cache, type=local,dest=path/to/dir)
@@ -295,7 +287,7 @@ Shorthand for [`--output=type=docker`](#docker). Will automatically load the sin
#### `--cache-from=[NAME|type=TYPE[,KEY=VALUE]]`
Use an external cache source for a build. Supported types are `registry` and `local`. The `registry` source can import cache from a cache manifest or (special) image configuration on the registry. The `local` source can export cache from local files previously exported with `--cache-to`.
Use an external cache source for a build. Supported types are `registry` and `local`. The `registry` source can import cache from a cache manifest or (special) image configuration on the registry. The `local` source can import cache from local files previously exported with `--cache-to`.
If no type is specified, `registry` exporter is used with a specified reference.
@@ -327,6 +319,20 @@ docker buildx build --cache-to=type=registry,ref=user/app .
docker buildx build --cache-to=type=local,dest=path/to/cache .
```
#### `--allow=ENTITLEMENT`
Allow extra privileged entitlement. List of entitlements:
- `network.host` - Allows executions with host networking.
- `security.insecure` - Allows executions without sandbox. See [related Dockerfile extensions](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md#run---securityinsecuresandbox).
For entitlements to be enabled, the `buildkitd` daemon also needs to allow them with `--allow-insecure-entitlement` (see [`create --buildkitd-flags`](#--buildkitd-flags-flags))
Example:
```
$ docker buildx create --use --name insecure-builder --buildkitd-flags '--allow-insecure-entitlement security.insecure'
$ docker buildx build --allow security.insecure .
```
### `buildx create [OPTIONS] [CONTEXT|ENDPOINT]`
@@ -340,21 +346,16 @@ Options:
| Flag | Description |
| --- | --- |
| --append | Append a node to builder instead of changing it
| --driver string | Driver to use (eg. docker-container)
| --leave | Remove a node from builder instead of changing it
| --name string | Builder instance name
| --node string | Create/modify node with given name
| --platform stringArray | Fixed platforms for current node
| --use | Set the current builder instance
#### `--driver DRIVER`
Sets the builder driver to be used. There are two available drivers, each have their own specificities.
- `docker` - Uses the builder that is built into the docker daemon. With this driver, the [`--load`](#--load) flag is implied by default on `buildx build`. However, building multi-platform images or exporting cache is not currently supported.
- `docker-container` - Uses a buildkit container that will be spawned via docker. With this driver, both building multi-platform images and exporting cache are supported. However, images built will not automatically appear in `docker images` (see [`build --load`](#--load)).
| --append | Append a node to builder instead of changing it
| --buildkitd-flags string | Flags for buildkitd daemon
| --config string | BuildKit config file
| --driver string | Driver to use (eg. docker-container)
| --driver-opt stringArray | Options for the driver
| --leave | Remove a node from builder instead of changing it
| --name string | Builder instance name
| --node string | Create/modify node with given name
| --platform stringArray | Fixed platforms for current node
| --use | Set the current builder instance
#### `--append`
@@ -368,6 +369,41 @@ $ docker buildx create --name eager_beaver --append mycontext2
eager_beaver
```
#### `--buildkitd-flags FLAGS`
Adds flags when starting the buildkitd daemon. They take precedence over the configuration file specified by [`--config`](#--config-file). See `buildkitd --help` for the available flags.
Example:
```
--buildkitd-flags '--debug --debugaddr 0.0.0.0:6666'
```
#### `--config FILE`
Specifies the configuration file for the buildkitd daemon to use. The configuration can be overridden by [`--buildkitd-flags`](#--buildkitd-flags-flags). See an [example buildkitd configuration file](https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md).
#### `--driver DRIVER`
Sets the builder driver to be used. There are two available drivers, each have their own specificities.
- `docker` - Uses the builder that is built into the docker daemon. With this driver, the [`--load`](#--load) flag is implied by default on `buildx build`. However, building multi-platform images or exporting cache is not currently supported.
- `docker-container` - Uses a buildkit container that will be spawned via docker. With this driver, both building multi-platform images and exporting cache are supported. However, images built will not automatically appear in `docker images` (see [`build --load`](#--load)).
#### `--driver-opt OPTIONS`
Passes additional driver-specific options. Details for each driver:
- `docker` - No driver options
- `docker-container`
- `image` - Sets the container image to be used for running buildkit.
- `network` - Sets the network mode for running the buildkit container.
- Example:
```
--driver docker-container --driver-opt image=moby/buildkit:master,network=host
```
#### `--leave`
Changes the action of the command to removes a node from a builder. The builder needs to be specified with `--name` and node that is removed is set with `--node`.
@@ -541,23 +577,23 @@ Note: Design of bake command is work in progress, the user experience may change
Example HCL defintion:
```
group default {
targets = [db, webapp-dev]
group "default" {
targets = ["db", "webapp-dev"]
}
target webapp-dev {
target "webapp-dev" {
dockerfile = "Dockerfile.webapp"
tags = ["docker.io/username/webapp"]
}
target webapp-release {
inherits = [webapp-dev]
platforms = [linux/amd64, linux/arm64]
target "webapp-release" {
inherits = ["webapp-dev"]
platforms = ["linux/amd64", "linux/arm64"]
}
target db {
target "db" {
dockerfile = "Dockerfile.db"
tags = [docker.io/username/db]
tags = ["docker.io/username/db"]
}
```

View File

@@ -77,7 +77,20 @@ func mergeConfig(c1, c2 Config) Config {
if c1.Group == nil {
c1.Group = map[string]Group{}
}
c1.Group[k] = g
if g1, exists := c1.Group[k]; exists {
nextTarget:
for _, t := range g.Targets {
for _, t2 := range g1.Targets {
if t == t2 {
continue nextTarget
}
}
g1.Targets = append(g1.Targets, t)
}
c1.Group[k] = g1
} else {
c1.Group[k] = g
}
}
for k, t := range c2.Target {
@@ -248,10 +261,10 @@ func (t *Target) normalize() {
t.Outputs = removeDupes(t.Outputs)
}
func TargetsToBuildOpt(m map[string]Target) (map[string]build.Options, error) {
func TargetsToBuildOpt(m map[string]Target, noCache, pull bool) (map[string]build.Options, error) {
m2 := make(map[string]build.Options, len(m))
for k, v := range m {
bo, err := toBuildOpt(v)
bo, err := toBuildOpt(v, noCache, pull)
if err != nil {
return nil, err
}
@@ -260,7 +273,7 @@ func TargetsToBuildOpt(m map[string]Target) (map[string]build.Options, error) {
return m2, nil
}
func toBuildOpt(t Target) (*build.Options, error) {
func toBuildOpt(t Target, noCache, pull bool) (*build.Options, error) {
if v := t.Context; v != nil && *v == "-" {
return nil, errors.Errorf("context from stdin not allowed in bake")
}
@@ -289,6 +302,8 @@ func toBuildOpt(t Target) (*build.Options, error) {
Tags: t.Tags,
BuildArgs: t.Args,
Labels: t.Labels,
NoCache: noCache,
Pull: pull,
}
platforms, err := platformutil.Parse(t.Platforms)

View File

@@ -59,11 +59,30 @@ services:
`), 0600)
require.NoError(t, err)
ctx := context.TODO()
fp2 := filepath.Join(tmpdir, "docker-compose2.yml")
err = ioutil.WriteFile(fp2, []byte(`
version: "3"
m, err := ReadTargets(ctx, []string{fp}, []string{"default"}, nil)
services:
newservice:
build: .
webapp:
build:
args:
buildno2: 12
`), 0600)
require.NoError(t, err)
ctx := context.TODO()
m, err := ReadTargets(ctx, []string{fp, fp2}, []string{"default"}, nil)
require.NoError(t, err)
require.Equal(t, 3, len(m))
_, ok := m["newservice"]
require.True(t, ok)
require.Equal(t, "Dockerfile.webapp", *m["webapp"].Dockerfile)
require.Equal(t, ".", *m["webapp"].Context)
require.Equal(t, "1", m["webapp"].Args["buildno"])
require.Equal(t, "12", m["webapp"].Args["buildno2"])
}

View File

@@ -2,7 +2,9 @@ package bake
import (
"fmt"
"os"
"reflect"
"strings"
"github.com/docker/cli/cli/compose/loader"
composetypes "github.com/docker/cli/cli/compose/types"
@@ -19,9 +21,22 @@ func parseCompose(dt []byte) (*composetypes.Config, error) {
Config: parsed,
},
},
Environment: envMap(os.Environ()),
})
}
func envMap(env []string) map[string]string {
result := make(map[string]string, len(env))
for _, s := range env {
kv := strings.SplitN(s, "=", 2)
if len(kv) != 2 {
continue
}
result[kv[0]] = kv[1]
}
return result
}
func ParseCompose(dt []byte) (*Config, error) {
cfg, err := parseCompose(dt)
if err != nil {
@@ -86,6 +101,8 @@ func toMap(in composetypes.MappingWithEquals) map[string]string {
for k, v := range in {
if v != nil {
m[k] = *v
} else {
m[k] = os.Getenv(k)
}
}
return m

View File

@@ -24,6 +24,7 @@ import (
"github.com/moby/buildkit/client"
"github.com/moby/buildkit/session"
"github.com/moby/buildkit/session/upload/uploadprovider"
"github.com/moby/buildkit/util/entitlements"
"github.com/opencontainers/go-digest"
specs "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
@@ -55,6 +56,7 @@ type Options struct {
CacheFrom []client.CacheOptionsEntry
CacheTo []client.CacheOptionsEntry
Allow []entitlements.Entitlement
// DockerTarget
}
@@ -324,11 +326,12 @@ func toSolveOpt(d driver.Driver, multiDriver bool, opt Options, dl dockerLoadCal
}
so := client.SolveOpt{
Frontend: "dockerfile.v0",
FrontendAttrs: map[string]string{},
LocalDirs: map[string]string{},
CacheExports: opt.CacheTo,
CacheImports: opt.CacheFrom,
Frontend: "dockerfile.v0",
FrontendAttrs: map[string]string{},
LocalDirs: map[string]string{},
CacheExports: opt.CacheTo,
CacheImports: opt.CacheFrom,
AllowedEntitlements: opt.Allow,
}
if multiDriver {
@@ -397,7 +400,7 @@ func toSolveOpt(d driver.Driver, multiDriver bool, opt Options, dl dockerLoadCal
return nil, nil, err
}
defers = append(defers, cancel)
opt.Exports[i].Output = w
opt.Exports[i].Output = wrapWriteCloser(w)
}
} else if !d.Features()[driver.DockerExporter] {
return nil, nil, notSupported(d, driver.DockerExporter)
@@ -454,6 +457,7 @@ func toSolveOpt(d driver.Driver, multiDriver bool, opt Options, dl dockerLoadCal
switch opt.NetworkMode {
case "host", "none":
so.FrontendAttrs["force-network-mode"] = opt.NetworkMode
so.AllowedEntitlements = append(so.AllowedEntitlements, entitlements.EntitlementNetworkHost)
case "", "default":
default:
return nil, nil, errors.Errorf("network mode %q not supported by buildkit", opt.NetworkMode)
@@ -537,7 +541,7 @@ func Build(ctx context.Context, drivers []DriverInfo, opt map[string]Options, do
multiTarget := len(opt) > 1
for k, opt := range opt {
err := func() error {
err := func(k string) error {
opt := opt
dps := m[k]
multiDriver := len(m[k]) > 1
@@ -681,7 +685,7 @@ func Build(ctx context.Context, drivers []DriverInfo, opt map[string]Options, do
}
return nil
}()
}(k)
if err != nil {
return nil, err
}
@@ -731,7 +735,7 @@ func LoadInputs(inp Inputs, target *client.SolveOpt) (func(), error) {
return nil, errStdinConflict
}
buf := bufio.NewReader(os.Stdin)
buf := bufio.NewReader(inp.InStream)
magic, err := buf.Peek(archiveHeaderSize * 2)
if err != nil && err != io.EOF {
return nil, errors.Wrap(err, "failed to peek context header from STDIN")
@@ -757,7 +761,7 @@ func LoadInputs(inp Inputs, target *client.SolveOpt) (func(), error) {
target.LocalDirs["context"] = inp.ContextPath
switch inp.DockerfilePath {
case "-":
dockerfileReader = os.Stdin
dockerfileReader = inp.InStream
case "":
dockerfileDir = inp.ContextPath
default:
@@ -780,6 +784,7 @@ func LoadInputs(inp Inputs, target *client.SolveOpt) (func(), error) {
return nil, err
}
toRemove = append(toRemove, dockerfileDir)
dockerfileName = "Dockerfile"
}
if dockerfileName == "" {

21
build/entitlements.go Normal file
View File

@@ -0,0 +1,21 @@
package build
import (
"github.com/moby/buildkit/util/entitlements"
"github.com/pkg/errors"
)
func ParseEntitlements(in []string) ([]entitlements.Entitlement, error) {
out := make([]entitlements.Entitlement, 0, len(in))
for _, v := range in {
switch v {
case "security.insecure":
out = append(out, entitlements.EntitlementSecurityInsecure)
case "network.host":
out = append(out, entitlements.EntitlementNetworkHost)
default:
return nil, errors.Errorf("invalid entitlement: %v", v)
}
}
return out, nil
}

View File

@@ -2,6 +2,7 @@ package build
import (
"encoding/csv"
"io"
"os"
"strings"
@@ -81,7 +82,7 @@ func ParseOutputs(inp []string) ([]client.ExportEntry, error) {
if _, err := console.ConsoleFromFile(os.Stdout); err == nil {
return nil, errors.Errorf("output file is required for %s exporter. refusing to write to console", out.Type)
}
out.Output = os.Stdout
out.Output = wrapWriteCloser(os.Stdout)
} else if dest != "" {
fi, err := os.Stat(dest)
if err != nil && !os.IsNotExist(err) {
@@ -94,7 +95,7 @@ func ParseOutputs(inp []string) ([]client.ExportEntry, error) {
if err != nil {
return nil, errors.Errorf("failed to open %s", err)
}
out.Output = f
out.Output = wrapWriteCloser(f)
}
delete(out.Attrs, "dest")
case "registry":
@@ -106,3 +107,9 @@ func ParseOutputs(inp []string) ([]client.ExportEntry, error) {
}
return outs, nil
}
func wrapWriteCloser(wc io.WriteCloser) func(map[string]string) (io.WriteCloser, error) {
return func(map[string]string) (io.WriteCloser, error) {
return wc, nil
}
}

View File

@@ -51,7 +51,7 @@ func runBake(dockerCli command.Cli, targets []string, in bakeOptions) error {
return nil
}
bo, err := bake.TargetsToBuildOpt(m)
bo, err := bake.TargetsToBuildOpt(m, in.noCache, in.pull)
if err != nil {
return err
}

View File

@@ -44,6 +44,8 @@ type buildOptions struct {
squash bool
quiet bool
allow []string
// hidden
// untrusted bool
// ulimits *opts.UlimitOpt
@@ -84,8 +86,8 @@ func runBuild(dockerCli command.Cli, in buildOptions) error {
InStream: os.Stdin,
},
Tags: in.tags,
Labels: listToMap(in.labels),
BuildArgs: listToMap(in.buildArgs),
Labels: listToMap(in.labels, false),
BuildArgs: listToMap(in.buildArgs, true),
Pull: in.pull,
NoCache: in.noCache,
Target: in.target,
@@ -167,6 +169,12 @@ func runBuild(dockerCli command.Cli, in buildOptions) error {
}
opts.CacheTo = cacheExports
allow, err := build.ParseEntitlements(in.allow)
if err != nil {
return err
}
opts.Allow = allow
return buildTargets(ctx, dockerCli, map[string]build.Options{"default": opts}, in.progress)
}
@@ -214,6 +222,8 @@ func buildCmd(dockerCli command.Cli) *cobra.Command {
flags.StringVar(&options.target, "target", "", "Set the target build stage to build.")
flags.StringSliceVar(&options.allow, "allow", []string{}, "Allow extra privileged entitlement, e.g. network.host, security.insecure")
// not implemented
flags.BoolVarP(&options.quiet, "quiet", "q", false, "Suppress the build output and print image ID on success")
flags.StringVar(&options.networkMode, "network", "default", "Set the networking mode for the RUN instructions during build")
@@ -282,12 +292,16 @@ func commonFlags(options *commonOptions, flags *pflag.FlagSet) {
flags.BoolVar(&options.pull, "pull", false, "Always attempt to pull a newer version of the image")
}
func listToMap(values []string) map[string]string {
func listToMap(values []string, defaultEnv bool) map[string]string {
result := make(map[string]string, len(values))
for _, value := range values {
kv := strings.SplitN(value, "=", 2)
if len(kv) == 1 {
result[kv[0]] = ""
if defaultEnv {
result[kv[0]] = os.Getenv(kv[0])
} else {
result[kv[0]] = ""
}
} else {
result[kv[0]] = kv[1]
}

View File

@@ -1,13 +1,16 @@
package commands
import (
"encoding/csv"
"fmt"
"os"
"strings"
"github.com/docker/buildx/driver"
"github.com/docker/buildx/store"
"github.com/docker/cli/cli"
"github.com/docker/cli/cli/command"
"github.com/google/shlex"
"github.com/moby/buildkit/util/appcontext"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
@@ -22,6 +25,9 @@ type createOptions struct {
actionAppend bool
actionLeave bool
use bool
flags string
configFile string
driverOpts []string
// upgrade bool // perform upgrade of the driver
}
@@ -107,6 +113,14 @@ func runCreate(dockerCli command.Cli, in createOptions, args []string) error {
ng.Driver = driverName
}
var flags []string
if in.flags != "" {
flags, err = shlex.Split(in.flags)
if err != nil {
return errors.Wrap(err, "failed to parse buildkit flags")
}
}
var ep string
if in.actionLeave {
if err := ng.Leave(in.nodeName); err != nil {
@@ -128,7 +142,11 @@ func runCreate(dockerCli command.Cli, in createOptions, args []string) error {
return err
}
}
if err := ng.Update(in.nodeName, ep, in.platform, len(args) > 0, in.actionAppend); err != nil {
m, err := csvToMap(in.driverOpts)
if err != nil {
return err
}
if err := ng.Update(in.nodeName, ep, in.platform, len(args) > 0, in.actionAppend, flags, in.configFile, m); err != nil {
return err
}
}
@@ -154,6 +172,11 @@ func runCreate(dockerCli command.Cli, in createOptions, args []string) error {
func createCmd(dockerCli command.Cli) *cobra.Command {
var options createOptions
var drivers []string
for s := range driver.GetFactories() {
drivers = append(drivers, s)
}
cmd := &cobra.Command{
Use: "create [OPTIONS] [CONTEXT|ENDPOINT]",
Short: "Create a new builder instance",
@@ -166,9 +189,12 @@ func createCmd(dockerCli command.Cli) *cobra.Command {
flags := cmd.Flags()
flags.StringVar(&options.name, "name", "", "Builder instance name")
flags.StringVar(&options.driver, "driver", "", "Driver to use (eg. docker-container)")
flags.StringVar(&options.driver, "driver", "", fmt.Sprintf("Driver to use (available: %v)", drivers))
flags.StringVar(&options.nodeName, "node", "", "Create/modify node with given name")
flags.StringVar(&options.flags, "buildkitd-flags", "", "Flags for buildkitd daemon")
flags.StringVar(&options.configFile, "config", "", "BuildKit config file")
flags.StringArrayVar(&options.platform, "platform", []string{}, "Fixed platforms for current node")
flags.StringArrayVar(&options.driverOpts, "driver-opt", []string{}, "Options for the driver")
flags.BoolVar(&options.actionAppend, "append", false, "Append a node to builder instead of changing it")
flags.BoolVar(&options.actionLeave, "leave", false, "Remove a node from builder instead of changing it")
@@ -178,3 +204,22 @@ func createCmd(dockerCli command.Cli) *cobra.Command {
return cmd
}
func csvToMap(in []string) (map[string]string, error) {
m := make(map[string]string, len(in))
for _, s := range in {
csvReader := csv.NewReader(strings.NewReader(s))
fields, err := csvReader.Read()
if err != nil {
return nil, err
}
for _, v := range fields {
p := strings.SplitN(v, "=", 2)
if len(p) != 2 {
return nil, errors.Errorf("invalid value %q, expecting k=v", v)
}
m[p[0]] = p[1]
}
}
return m, nil
}

View File

@@ -114,6 +114,9 @@ func runInspect(dockerCli command.Cli, in inspectOptions, args []string) error {
fmt.Fprintf(w, "Error:\t%s\n", err.Error())
} else {
fmt.Fprintf(w, "Status:\t%s\n", ngi.drivers[i].info.Status)
if len(n.Flags) > 0 {
fmt.Fprintf(w, "Flags:\t%s\n", strings.Join(n.Flags, " "))
}
fmt.Fprintf(w, "Platforms:\t%s\n", strings.Join(platformutil.Format(platformutil.Dedupe(append(n.Platforms, ngi.drivers[i].platforms...))), ", "))
}
}

View File

@@ -174,7 +174,7 @@ func driversForNodeGroup(ctx context.Context, dockerCli command.Cli, ng *store.N
// TODO: replace the following line with dockerclient.WithAPIVersionNegotiation option in clientForEndpoint
dockerapi.NegotiateAPIVersion(ctx)
d, err := driver.GetDriver(ctx, "buildx_buildkit_"+n.Name, f, dockerapi)
d, err := driver.GetDriver(ctx, "buildx_buildkit_"+n.Name, f, dockerapi, n.Flags, n.ConfigFile, n.DriverOpts)
if err != nil {
di.Err = err
return nil
@@ -251,7 +251,7 @@ func getDefaultDrivers(ctx context.Context, dockerCli command.Cli) ([]build.Driv
return driversForNodeGroup(ctx, dockerCli, ng)
}
d, err := driver.GetDriver(ctx, "buildx_buildkit_default", nil, dockerCli.Client())
d, err := driver.GetDriver(ctx, "buildx_buildkit_default", nil, dockerCli.Client(), nil, "", nil)
if err != nil {
return nil, err
}

View File

@@ -1,6 +1,8 @@
package docker
import (
"archive/tar"
"bytes"
"context"
"io"
"io/ioutil"
@@ -20,11 +22,13 @@ import (
"github.com/pkg/errors"
)
var buildkitImage = "moby/buildkit:master" // TODO: make this verified and configuratble
var defaultBuildkitImage = "moby/buildkit:buildx-stable-1" // TODO: make this verified
type Driver struct {
driver.InitConfig
factory driver.Factory
netMode string
image string
}
func (d *Driver) Bootstrap(ctx context.Context, l progress.Logger) error {
@@ -49,8 +53,12 @@ func (d *Driver) Bootstrap(ctx context.Context, l progress.Logger) error {
}
func (d *Driver) create(ctx context.Context, l progress.SubLogger) error {
if err := l.Wrap("pulling image "+buildkitImage, func() error {
rc, err := d.DockerAPI.ImageCreate(ctx, buildkitImage, types.ImageCreateOptions{})
imageName := defaultBuildkitImage
if d.image != "" {
imageName = d.image
}
if err := l.Wrap("pulling image "+imageName, func() error {
rc, err := d.DockerAPI.ImageCreate(ctx, imageName, types.ImageCreateOptions{})
if err != nil {
return err
}
@@ -59,15 +67,34 @@ func (d *Driver) create(ctx context.Context, l progress.SubLogger) error {
}); err != nil {
return err
}
cfg := &container.Config{
Image: imageName,
}
if d.InitConfig.BuildkitFlags != nil {
cfg.Cmd = d.InitConfig.BuildkitFlags
}
if err := l.Wrap("creating container "+d.Name, func() error {
_, err := d.DockerAPI.ContainerCreate(ctx, &container.Config{
Image: buildkitImage,
}, &container.HostConfig{
hc := &container.HostConfig{
Privileged: true,
}, &network.NetworkingConfig{}, d.Name)
}
if d.netMode != "" {
hc.NetworkMode = container.NetworkMode(d.netMode)
}
_, err := d.DockerAPI.ContainerCreate(ctx, cfg, hc, &network.NetworkingConfig{}, d.Name)
if err != nil {
return err
}
if f := d.InitConfig.ConfigFile; f != "" {
buf, err := readFileToTar(f)
if err != nil {
return err
}
if err := d.DockerAPI.CopyToContainer(ctx, d.Name, "/", buf, dockertypes.CopyToContainerOptions{}); err != nil {
return err
}
}
if err := d.start(ctx, l); err != nil {
return err
}
@@ -239,3 +266,26 @@ type demux struct {
func (d *demux) Read(dt []byte) (int, error) {
return d.Reader.Read(dt)
}
func readFileToTar(fn string) (*bytes.Buffer, error) {
buf := bytes.NewBuffer(nil)
tw := tar.NewWriter(buf)
dt, err := ioutil.ReadFile(fn)
if err != nil {
return nil, err
}
if err := tw.WriteHeader(&tar.Header{
Name: "/etc/buildkit/buildkitd.toml",
Size: int64(len(dt)),
Mode: 0644,
}); err != nil {
return nil, err
}
if _, err := tw.Write(dt); err != nil {
return nil, err
}
if err := tw.Close(); err != nil {
return nil, err
}
return buf, nil
}

View File

@@ -37,8 +37,22 @@ func (f *factory) New(ctx context.Context, cfg driver.InitConfig) (driver.Driver
if cfg.DockerAPI == nil {
return nil, errors.Errorf("%s driver requires docker API access", f.Name())
}
d := &Driver{factory: f, InitConfig: cfg}
for k, v := range cfg.DriverOpts {
switch k {
case "network":
d.netMode = v
if v == "host" {
d.InitConfig.BuildkitFlags = append(d.InitConfig.BuildkitFlags, "--allow-insecure-entitlement=network.host")
}
case "image":
d.image = v
default:
return nil, errors.Errorf("invalid driver option %s for docker-container driver", k)
}
}
return &Driver{factory: f, InitConfig: cfg}, nil
return d, nil
}
func (f *factory) AllowsInstances() bool {

View File

@@ -44,6 +44,9 @@ func (f *factory) New(ctx context.Context, cfg driver.InitConfig) (driver.Driver
if cfg.DockerAPI == nil {
return nil, errors.Errorf("docker driver requires docker API access")
}
if cfg.ConfigFile != "" {
return nil, errors.Errorf("setting config file is not supported for docker driver, use dockerd configuration file")
}
return &Driver{factory: f, InitConfig: cfg}, nil
}

View File

@@ -23,10 +23,11 @@ type BuildkitConfig struct {
type InitConfig struct {
// This object needs updates to be generic for different drivers
Name string
DockerAPI dockerclient.APIClient
BuildkitConfig BuildkitConfig
Meta map[string]interface{}
Name string
DockerAPI dockerclient.APIClient
BuildkitFlags []string
ConfigFile string
DriverOpts map[string]string
}
var drivers map[string]Factory
@@ -71,10 +72,13 @@ func GetFactory(name string, instanceRequired bool) Factory {
return nil
}
func GetDriver(ctx context.Context, name string, f Factory, api dockerclient.APIClient) (Driver, error) {
func GetDriver(ctx context.Context, name string, f Factory, api dockerclient.APIClient, flags []string, config string, do map[string]string) (Driver, error) {
ic := InitConfig{
DockerAPI: api,
Name: name,
DockerAPI: api,
Name: name,
BuildkitFlags: flags,
ConfigFile: config,
DriverOpts: do,
}
if f == nil {
var err error
@@ -85,3 +89,7 @@ func GetDriver(ctx context.Context, name string, f Factory, api dockerclient.API
}
return f.New(ctx, ic)
}
func GetFactories() map[string]Factory {
return drivers
}

5
go.mod
View File

@@ -14,7 +14,7 @@ require (
github.com/cenkalti/backoff v2.1.1+incompatible // indirect
github.com/cloudflare/cfssl v0.0.0-20181213083726-b94e044bb51e // indirect
github.com/containerd/console v0.0.0-20181022165439-0650fd9eeb50
github.com/containerd/containerd v1.3.0-0.20190426060238-3a3f0aac8819
github.com/containerd/containerd v1.3.0-0.20190507210959-7c1e88399ec0
github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448 // indirect
github.com/containerd/typeurl v0.0.0-20190228175220-2a93cfde8c20 // indirect
github.com/denisenkom/go-mssqldb v0.0.0-20190315220205-a8ed825ac853 // indirect
@@ -35,6 +35,7 @@ require (
github.com/gogo/protobuf v1.2.1 // indirect
github.com/google/certificate-transparency-go v1.0.21 // indirect
github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf // indirect
github.com/google/shlex v0.0.0-20150127133951-6f45313302b9
github.com/gorilla/mux v1.7.0 // indirect
github.com/hailocab/go-hostpool v0.0.0-20160125115350-e80d13ce29ed // indirect
github.com/hashicorp/go-version v1.1.0 // indirect
@@ -51,7 +52,7 @@ require (
github.com/mattn/go-sqlite3 v1.10.0 // indirect
github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
github.com/miekg/pkcs11 v0.0.0-20190322140431-074fd7a1ed19 // indirect
github.com/moby/buildkit v0.5.2-0.20190513182223-f238f1efb04f
github.com/moby/buildkit v0.6.2-0.20190921002054-ae10b292fefb
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
github.com/modern-go/reflect2 v1.0.1 // indirect
github.com/opencontainers/go-digest v1.0.0-rc1

26
go.sum
View File

@@ -39,20 +39,22 @@ github.com/containerd/cgroups v0.0.0-20190226200435-dbea6f2bd416/go.mod h1:X9rLE
github.com/containerd/console v0.0.0-20181022165439-0650fd9eeb50 h1:WMpHmC6AxwWb9hMqhudkqG7A/p14KiMnl6d3r1iUMjU=
github.com/containerd/console v0.0.0-20181022165439-0650fd9eeb50/go.mod h1:Tj/on1eG8kiEhd0+fhSDzsPAFESxzBBvdyEgyryXffw=
github.com/containerd/containerd v1.2.4/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA=
github.com/containerd/containerd v1.3.0-0.20190426060238-3a3f0aac8819 h1:otmq8xNIzAo+2SjPURbYZXVW+B6hZBAWJ+JApzCYWDk=
github.com/containerd/containerd v1.3.0-0.20190426060238-3a3f0aac8819/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA=
github.com/containerd/containerd v1.3.0-0.20190507210959-7c1e88399ec0 h1:enps1EZBEgR8QxwdrpsoSxcsCXWnMKchIQ/0dzC0eKw=
github.com/containerd/containerd v1.3.0-0.20190507210959-7c1e88399ec0/go.mod h1:bC6axHOhabU15QhwfG7w5PipXdVtMXFTttgp+kVtyUA=
github.com/containerd/continuity v0.0.0-20181001140422-bd77b46c8352/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc h1:TP+534wVlf61smEIq1nwLLAjQVEK2EADoW3CX9AuT+8=
github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/continuity v0.0.0-20190827140505-75bee3e2ccb6 h1:NmTXa/uVnDyp0TY5MKi197+3HWcnYWfnHGyaFthlnGw=
github.com/containerd/continuity v0.0.0-20190827140505-75bee3e2ccb6/go.mod h1:GL3xCUCBDV3CZiTSEKksMWbLE66hEyuu9qyDOOqM47Y=
github.com/containerd/fifo v0.0.0-20180307165137-3d5202aec260/go.mod h1:ODA38xgv3Kuk8dQz2ZQXpnv/UZZUHUCL7pnLehbXgQI=
github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448 h1:PUD50EuOMkXVcpBIA/R95d56duJR9VxhwncsFbNnxW4=
github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448/go.mod h1:ODA38xgv3Kuk8dQz2ZQXpnv/UZZUHUCL7pnLehbXgQI=
github.com/containerd/go-runc v0.0.0-20180907222934-5a6d9f37cfa3/go.mod h1:IV7qH3hrUgRmyYrtgEeGWJfWbgcHL9CSRruz2Vqcph0=
github.com/containerd/go-cni v0.0.0-20190610170741-5a4663dad645/go.mod h1:2wlRxCQdiBY+OcjNg5x8kI+5mEL1fGt25L4IzQHYJsM=
github.com/containerd/go-runc v0.0.0-20190911050354-e029b79d8cda/go.mod h1:IV7qH3hrUgRmyYrtgEeGWJfWbgcHL9CSRruz2Vqcph0=
github.com/containerd/ttrpc v0.0.0-20190411181408-699c4e40d1e7 h1:SKDlsIhYxNE1LO0xwuOR+3QWj3zRibVQu5jWIMQmOfU=
github.com/containerd/ttrpc v0.0.0-20190411181408-699c4e40d1e7/go.mod h1:PvCDdDGpgqzQIzDW1TphrGLssLDZp2GuS+X5DkEJB8o=
github.com/containerd/typeurl v0.0.0-20180627222232-a93fcdb778cd/go.mod h1:Cm3kwCdlkCfMSHURc+r6fwoGH6/F1hH3S4sg0rLFWPc=
github.com/containerd/typeurl v0.0.0-20190228175220-2a93cfde8c20 h1:14r0i3IeJj6zkNLigAJiv/TWSR8EY+pxIjv5tFiT+n8=
github.com/containerd/typeurl v0.0.0-20190228175220-2a93cfde8c20/go.mod h1:Cm3kwCdlkCfMSHURc+r6fwoGH6/F1hH3S4sg0rLFWPc=
github.com/containernetworking/cni v0.6.1-0.20180218032124-142cde0c766c/go.mod h1:LGwApLUm2FpoOfxTDEeq8T9ipbpZ61X79hmU3w8FmsY=
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
@@ -89,7 +91,7 @@ github.com/docker/go-metrics v0.0.0-20170502235133-d466d4f6fd96 h1:HVQ/BC7Ze+bcV
github.com/docker/go-metrics v0.0.0-20170502235133-d466d4f6fd96/go.mod h1:/u0gXw0Gay3ceNrsHubL3BtdOL2fHf93USgMTe0W5dI=
github.com/docker/go-units v0.3.1 h1:QAFdsA6jLCnglbqE6mUsHuPcJlntY94DkxHf4deHKIU=
github.com/docker/go-units v0.3.1/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
github.com/docker/libnetwork v0.0.0-20180913200009-36d3bed0e9f4/go.mod h1:93m0aTqz6z+g32wla4l4WxTrdtvBRmVzYRkYvasA5Z8=
github.com/docker/libnetwork v0.8.0-dev.2.0.20190604151032-3c26b4e7495e/go.mod h1:93m0aTqz6z+g32wla4l4WxTrdtvBRmVzYRkYvasA5Z8=
github.com/docker/libtrust v0.0.0-20150526203908-9cbd2a1374f4 h1:k8TfKGeAcDQFFQOGCQMRN04N4a9YrPlRMMKnzAuvM9Q=
github.com/docker/libtrust v0.0.0-20150526203908-9cbd2a1374f4/go.mod h1:cyGadeNEkKy96OOhEzfZl+yxihPEzKnqJwvfuSUqbZE=
github.com/erikstmartin/go-testdb v0.0.0-20160219214506-8d10e4a1bae5 h1:Yzb9+7DPaBjB8zlTR87/ElzFsnQfuHnVUVqpZZIcV5Y=
@@ -181,8 +183,8 @@ github.com/miekg/pkcs11 v0.0.0-20190322140431-074fd7a1ed19/go.mod h1:WCBAbTOdfhH
github.com/mitchellh/hashstructure v0.0.0-20170609045927-2bca23e0e452/go.mod h1:QjSHrPWS+BGUVBYkbTZWEnOh3G1DutKwClXU/ABz6AQ=
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
github.com/moby/buildkit v0.5.2-0.20190513182223-f238f1efb04f h1:gNiytoQ3/71JJKWbLPtxoInde5kBRpvqH6EbUf12JDU=
github.com/moby/buildkit v0.5.2-0.20190513182223-f238f1efb04f/go.mod h1:RozsxlEOjPuX/zKKrd0ZS76dlVNZu4qAff+vj7hR/qs=
github.com/moby/buildkit v0.6.2-0.20190921002054-ae10b292fefb h1:enyviD1ZOxgo62sGpT2yQY1uTtruq84wYJPjFJwsbH0=
github.com/moby/buildkit v0.6.2-0.20190921002054-ae10b292fefb/go.mod h1:JKVImCzxztxvULr5P6ZiBfA/B2P+ZpR6UHxOXQn4KiU=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd h1:TRLaZ9cD/w8PVh93nsPXa1VrQ6jlwL5oN8l14QlcNfg=
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd/go.mod h1:6dJC0mAP4ikYIbvyc7fijjWJddQyLn8Ig3JB5CqoB9Q=
github.com/modern-go/reflect2 v1.0.1 h1:9f412s+6RmYXLWZSEzVVgPGK7C2PphHj5RJrvfx9AWI=
@@ -197,8 +199,8 @@ github.com/opencontainers/go-digest v1.0.0-rc1/go.mod h1:cMLVZDEM3+U2I4VmLI6N8jQ
github.com/opencontainers/image-spec v1.0.1 h1:JMemWkRwHx4Zj+fVxWoMCFm/8sYGGrUVojFA6h/TRcI=
github.com/opencontainers/image-spec v1.0.1/go.mod h1:BtxoFyWECRxE4U/7sNtV5W15zMzWCbyJoFRP3s7yZA0=
github.com/opencontainers/runc v1.0.0-rc6/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opencontainers/runc v1.0.1-0.20190307181833-2b18fe1d885e h1:+uPGJuuDl61O9GKN/rLHkUCf597mpxmJI06RqMQX81A=
github.com/opencontainers/runc v1.0.1-0.20190307181833-2b18fe1d885e/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opencontainers/runc v1.0.0-rc8 h1:dDCFes8Hj1r/i5qnypONo5jdOme/8HWZC/aNDyhECt0=
github.com/opencontainers/runc v1.0.0-rc8/go.mod h1:qT5XzbpPznkRYVz/mWwUaVBUv2rmF59PVA73FjuZG0U=
github.com/opencontainers/runtime-spec v0.0.0-20180909173843-eba862dc2470/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
github.com/opencontainers/runtime-spec v1.0.1 h1:wY4pOY8fBdSIvs9+IDHC55thBuEulhzfSgKeC1yFvzQ=
github.com/opencontainers/runtime-spec v1.0.1/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
@@ -246,8 +248,8 @@ github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2 h1:b6uOv7YOFK0
github.com/syndtr/gocapability v0.0.0-20180916011248-d98352740cb2/go.mod h1:hkRG7XYTFWNJGYcbNJQlaLq0fg1yr4J4t/NcTQtrfww=
github.com/theupdateframework/notary v0.6.1 h1:7wshjstgS9x9F5LuB1L5mBI2xNMObWqjz+cjWoom6l0=
github.com/theupdateframework/notary v0.6.1/go.mod h1:MOfgIfmox8s7/7fduvB2xyPPMJCrjRLRizA8OFwpnKY=
github.com/tonistiigi/fsutil v0.0.0-20190327153851-3bbb99cdbd76 h1:eGfgYrNUSD448sa4mxH6nQpyZfN39QH0mLB7QaKIjus=
github.com/tonistiigi/fsutil v0.0.0-20190327153851-3bbb99cdbd76/go.mod h1:pzh7kdwkDRh+Bx8J30uqaKJ1M4QrSH/um8fcIXeM8rc=
github.com/tonistiigi/fsutil v0.0.0-20190819224149-3d2716dd0a4d h1:HJg27yqwTV7vFG9dWPDbUi373o/bmSDYGN9mZgVwdH0=
github.com/tonistiigi/fsutil v0.0.0-20190819224149-3d2716dd0a4d/go.mod h1:pzh7kdwkDRh+Bx8J30uqaKJ1M4QrSH/um8fcIXeM8rc=
github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea h1:SXhTLE6pb6eld/v/cCndK0AMpt1wiVFb/YYmqB3/QG0=
github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea/go.mod h1:WPnis/6cRcDZSUvVmezrxJPkiO87ThFYsoUiMwWNDJk=
github.com/tonistiigi/vt100 v0.0.0-20190402012908-ad4c4a574305 h1:y/1cL5AL2oRcfzz8CAHHhR6kDDfIOT0WEyH5k40sccM=

View File

@@ -16,9 +16,12 @@ type NodeGroup struct {
}
type Node struct {
Name string
Endpoint string
Platforms []specs.Platform
Name string
Endpoint string
Platforms []specs.Platform
Flags []string
ConfigFile string
DriverOpts map[string]string
}
func (ng *NodeGroup) Leave(name string) error {
@@ -33,7 +36,7 @@ func (ng *NodeGroup) Leave(name string) error {
return nil
}
func (ng *NodeGroup) Update(name, endpoint string, platforms []string, endpointsSet bool, actionAppend bool) error {
func (ng *NodeGroup) Update(name, endpoint string, platforms []string, endpointsSet bool, actionAppend bool, flags []string, configFile string, do map[string]string) error {
i := ng.findNode(name)
if i == -1 && !actionAppend {
if len(ng.Nodes) > 0 {
@@ -55,6 +58,9 @@ func (ng *NodeGroup) Update(name, endpoint string, platforms []string, endpoints
if len(platforms) > 0 {
n.Platforms = pp
}
if flags != nil {
n.Flags = flags
}
ng.Nodes[i] = n
if err := ng.validateDuplicates(endpoint, i); err != nil {
return err
@@ -72,9 +78,12 @@ func (ng *NodeGroup) Update(name, endpoint string, platforms []string, endpoints
}
n := Node{
Name: name,
Endpoint: endpoint,
Platforms: pp,
Name: name,
Endpoint: endpoint,
Platforms: pp,
ConfigFile: configFile,
Flags: flags,
DriverOpts: do,
}
ng.Nodes = append(ng.Nodes, n)

View File

@@ -11,16 +11,16 @@ func TestNodeGroupUpdate(t *testing.T) {
t.Parallel()
ng := &NodeGroup{}
err := ng.Update("foo", "foo0", []string{"linux/amd64"}, true, false)
err := ng.Update("foo", "foo0", []string{"linux/amd64"}, true, false, []string{"--debug"}, "", nil)
require.NoError(t, err)
err = ng.Update("foo1", "foo1", []string{"linux/arm64", "linux/arm/v7"}, true, true)
err = ng.Update("foo1", "foo1", []string{"linux/arm64", "linux/arm/v7"}, true, true, nil, "", nil)
require.NoError(t, err)
require.Equal(t, len(ng.Nodes), 2)
// update
err = ng.Update("foo", "foo2", []string{"linux/amd64", "linux/arm"}, true, false)
err = ng.Update("foo", "foo2", []string{"linux/amd64", "linux/arm"}, true, false, nil, "", nil)
require.NoError(t, err)
require.Equal(t, len(ng.Nodes), 2)
@@ -28,9 +28,11 @@ func TestNodeGroupUpdate(t *testing.T) {
require.Equal(t, []string{"linux/arm64"}, platformutil.Format(ng.Nodes[1].Platforms))
require.Equal(t, "foo2", ng.Nodes[0].Endpoint)
require.Equal(t, []string{"--debug"}, ng.Nodes[0].Flags)
require.Equal(t, []string(nil), ng.Nodes[1].Flags)
// duplicate endpoint
err = ng.Update("foo1", "foo2", nil, true, false)
err = ng.Update("foo1", "foo2", nil, true, false, nil, "", nil)
require.Error(t, err)
require.Contains(t, err.Error(), "duplicate endpoint")

View File

@@ -1,4 +1,4 @@
![containerd banner](https://raw.githubusercontent.com/cncf/artwork/master/containerd/horizontal/color/containerd-horizontal-color.png)
![containerd banner](https://raw.githubusercontent.com/cncf/artwork/master/projects/containerd/horizontal/color/containerd-horizontal-color.png)
[![GoDoc](https://godoc.org/github.com/containerd/containerd?status.svg)](https://godoc.org/github.com/containerd/containerd)
[![Build Status](https://travis-ci.org/containerd/containerd.svg?branch=master)](https://travis-ci.org/containerd/containerd)

View File

@@ -197,10 +197,7 @@ func onUntarJSON(r io.Reader, j interface{}) error {
if err != nil {
return err
}
if err := json.Unmarshal(b, j); err != nil {
return err
}
return nil
return json.Unmarshal(b, j)
}
func onUntarBlob(ctx context.Context, r io.Reader, store content.Ingester, size int64, ref string) (digest.Digest, error) {

View File

@@ -111,7 +111,18 @@ func unmount(target string, flags int) error {
// UnmountAll repeatedly unmounts the given mount point until there
// are no mounts remaining (EINVAL is returned by mount), which is
// useful for undoing a stack of mounts on the same mount point.
// UnmountAll all is noop when the first argument is an empty string.
// This is done when the containerd client did not specify any rootfs
// mounts (e.g. because the rootfs is managed outside containerd)
// UnmountAll is noop when the mount path does not exist.
func UnmountAll(mount string, flags int) error {
if mount == "" {
return nil
}
if _, err := os.Stat(mount); os.IsNotExist(err) {
return nil
}
for {
if err := unmount(mount, flags); err != nil {
// EINVAL is returned if the target is not a

View File

@@ -88,7 +88,7 @@ func appendDistributionSourceLabel(originLabel, repo string) string {
}
repos = append(repos, repo)
// use emtpy string to present duplicate items
// use empty string to present duplicate items
for i := 1; i < len(repos); i++ {
tmp, j := repos[i], i-1
for ; j >= 0 && repos[j] >= tmp; j-- {

View File

@@ -18,10 +18,10 @@ package docker
import (
"context"
"io"
"net/http"
"net/url"
"path"
"strconv"
"strings"
"github.com/containerd/containerd/errdefs"
@@ -29,6 +29,7 @@ import (
"github.com/containerd/containerd/log"
"github.com/containerd/containerd/reference"
"github.com/containerd/containerd/remotes"
"github.com/containerd/containerd/remotes/docker/schema1"
"github.com/containerd/containerd/version"
digest "github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
@@ -150,6 +151,32 @@ func NewResolver(options ResolverOptions) remotes.Resolver {
}
}
func getManifestMediaType(resp *http.Response) string {
// Strip encoding data (manifests should always be ascii JSON)
contentType := resp.Header.Get("Content-Type")
if sp := strings.IndexByte(contentType, ';'); sp != -1 {
contentType = contentType[0:sp]
}
// As of Apr 30 2019 the registry.access.redhat.com registry does not specify
// the content type of any data but uses schema1 manifests.
if contentType == "text/plain" {
contentType = images.MediaTypeDockerSchema1Manifest
}
return contentType
}
type countingReader struct {
reader io.Reader
bytesRead int64
}
func (r *countingReader) Read(p []byte) (int, error) {
n, err := r.reader.Read(p)
r.bytesRead += int64(n)
return n, err
}
var _ remotes.Resolver = &dockerResolver{}
func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocispec.Descriptor, error) {
@@ -220,40 +247,56 @@ func (r *dockerResolver) Resolve(ctx context.Context, ref string) (string, ocisp
}
return "", ocispec.Descriptor{}, errors.Errorf("unexpected status code %v: %v", u, resp.Status)
}
size := resp.ContentLength
// this is the only point at which we trust the registry. we use the
// content headers to assemble a descriptor for the name. when this becomes
// more robust, we mostly get this information from a secure trust store.
dgstHeader := digest.Digest(resp.Header.Get("Docker-Content-Digest"))
contentType := getManifestMediaType(resp)
if dgstHeader != "" {
if dgstHeader != "" && size != -1 {
if err := dgstHeader.Validate(); err != nil {
return "", ocispec.Descriptor{}, errors.Wrapf(err, "%q in header not a valid digest", dgstHeader)
}
dgst = dgstHeader
}
} else {
log.G(ctx).Debug("no Docker-Content-Digest header, fetching manifest instead")
if dgst == "" {
return "", ocispec.Descriptor{}, errors.Errorf("could not resolve digest for %v", ref)
}
req, err := http.NewRequest(http.MethodGet, u, nil)
if err != nil {
return "", ocispec.Descriptor{}, err
}
req.Header = r.headers
var (
size int64
sizeHeader = resp.Header.Get("Content-Length")
)
resp, err := fetcher.doRequestWithRetries(ctx, req, nil)
if err != nil {
return "", ocispec.Descriptor{}, err
}
defer resp.Body.Close()
size, err = strconv.ParseInt(sizeHeader, 10, 64)
if err != nil {
bodyReader := countingReader{reader: resp.Body}
return "", ocispec.Descriptor{}, errors.Wrapf(err, "invalid size header: %q", sizeHeader)
}
if size < 0 {
return "", ocispec.Descriptor{}, errors.Errorf("%q in header not a valid size", sizeHeader)
contentType = getManifestMediaType(resp)
if contentType == images.MediaTypeDockerSchema1Manifest {
b, err := schema1.ReadStripSignature(&bodyReader)
if err != nil {
return "", ocispec.Descriptor{}, err
}
dgst = digest.FromBytes(b)
} else {
dgst, err = digest.FromReader(&bodyReader)
if err != nil {
return "", ocispec.Descriptor{}, err
}
}
size = bodyReader.bytesRead
}
desc := ocispec.Descriptor{
Digest: dgst,
MediaType: resp.Header.Get("Content-Type"), // need to strip disposition?
MediaType: contentType,
Size: size,
}

View File

@@ -227,6 +227,17 @@ func (c *Converter) Convert(ctx context.Context, opts ...ConvertOpt) (ocispec.De
return desc, nil
}
// ReadStripSignature reads in a schema1 manifest and returns a byte array
// with the "signatures" field stripped
func ReadStripSignature(schema1Blob io.Reader) ([]byte, error) {
b, err := ioutil.ReadAll(io.LimitReader(schema1Blob, manifestSizeLimit)) // limit to 8MB
if err != nil {
return nil, err
}
return stripSignature(b)
}
func (c *Converter) fetchManifest(ctx context.Context, desc ocispec.Descriptor) error {
log.G(ctx).Debug("fetch schema 1")
@@ -235,17 +246,12 @@ func (c *Converter) fetchManifest(ctx context.Context, desc ocispec.Descriptor)
return err
}
b, err := ioutil.ReadAll(io.LimitReader(rc, manifestSizeLimit)) // limit to 8MB
b, err := ReadStripSignature(rc)
rc.Close()
if err != nil {
return err
}
b, err = stripSignature(b)
if err != nil {
return err
}
var m manifest
if err := json.Unmarshal(b, &m); err != nil {
return err

View File

@@ -20,7 +20,7 @@ github.com/gogo/protobuf v1.2.1
github.com/gogo/googleapis v1.2.0
github.com/golang/protobuf v1.2.0
github.com/opencontainers/runtime-spec 29686dbc5559d93fb1ef402eeda3e35c38d75af4 # v1.0.1-59-g29686db
github.com/opencontainers/runc 029124da7af7360afa781a0234d1b083550f797c
github.com/opencontainers/runc v1.0.0-rc8
github.com/konsorten/go-windows-terminal-sequences v1.0.1
github.com/sirupsen/logrus v1.4.1
github.com/urfave/cli 7bc6a0acffa589f415f88aca16cc1de5ffd66f9c
@@ -37,15 +37,15 @@ github.com/Microsoft/go-winio 84b4ab48a50763fe7b3abcef38e5205c12027fac
github.com/Microsoft/hcsshim 8abdbb8205e4192c68b5f84c31197156f31be517
google.golang.org/genproto d80a6e20e776b0b17a324d0ba1ab50a39c8e8944
golang.org/x/text 19e51611da83d6be54ddafce4a4af510cb3e9ea4
github.com/containerd/ttrpc f02858b1457c5ca3aaec3a0803eb0d59f96e41d6
github.com/containerd/ttrpc 699c4e40d1e7416e08bf7019c7ce2e9beced4636
github.com/syndtr/gocapability d98352740cb2c55f81556b63d4a1ec64c5a319c2
gotest.tools v2.3.0
github.com/google/go-cmp v0.2.0
go.etcd.io/bbolt v1.3.2
# cri dependencies
github.com/containerd/cri 6d353571e64417d80c9478ffaea793714dd539d0 # master
github.com/containerd/go-cni 40bcf8ec8acd7372be1d77031d585d5d8e561c90
github.com/containerd/cri 2fc62db8146ce66f27b37306ad5fda34207835f3 # master
github.com/containerd/go-cni 891c2a41e18144b2d7921f971d6c9789a68046b2
github.com/containernetworking/cni v0.6.0
github.com/containernetworking/plugins v0.7.0
github.com/davecgh/go-spew v1.1.0
@@ -59,7 +59,7 @@ github.com/hashicorp/go-multierror ed905158d87462226a13fe39ddf685ea65f1c11f
github.com/json-iterator/go 1.1.5
github.com/modern-go/reflect2 1.0.1
github.com/modern-go/concurrent 1.0.3
github.com/opencontainers/selinux v1.2.1
github.com/opencontainers/selinux v1.2.2
github.com/seccomp/libseccomp-golang 32f571b70023028bd57d9288c20efbcb237f3ce0
github.com/tchap/go-patricia v2.2.6
golang.org/x/crypto 88737f569e3a9c7ab309cdc09a07fe7fc87233c3

View File

@@ -80,7 +80,7 @@ func copyDirectory(dst, src string, inodes map[uint64]string, o *copyDirOpts) er
return errors.Wrapf(err, "failed to stat %s", src)
}
if !stat.IsDir() {
return errors.Errorf("source is not directory")
return errors.Errorf("source %s is not directory", src)
}
if st, err := os.Stat(dst); err != nil {

View File

@@ -427,11 +427,13 @@ func Security(s pb.SecurityMode) RunOption {
}
func Shlex(str string) RunOption {
return Shlexf(str)
return runOptionFunc(func(ei *ExecInfo) {
ei.State = shlexf(str, false)(ei.State)
})
}
func Shlexf(str string, v ...interface{}) RunOption {
return runOptionFunc(func(ei *ExecInfo) {
ei.State = shlexf(str, v...)(ei.State)
ei.State = shlexf(str, true, v...)(ei.State)
})
}
@@ -442,7 +444,9 @@ func Args(a []string) RunOption {
}
func AddEnv(key, value string) RunOption {
return AddEnvf(key, value)
return runOptionFunc(func(ei *ExecInfo) {
ei.State = ei.State.AddEnv(key, value)
})
}
func AddEnvf(key, value string, v ...interface{}) RunOption {
@@ -458,7 +462,9 @@ func User(str string) RunOption {
}
func Dir(str string) RunOption {
return Dirf(str)
return runOptionFunc(func(ei *ExecInfo) {
ei.State = ei.State.Dir(str)
})
}
func Dirf(str string, v ...interface{}) RunOption {
return runOptionFunc(func(ei *ExecInfo) {

View File

@@ -24,19 +24,24 @@ var (
keySecurity = contextKeyT("llb.security")
)
func addEnvf(key, value string, v ...interface{}) StateOption {
func addEnvf(key, value string, replace bool, v ...interface{}) StateOption {
if replace {
value = fmt.Sprintf(value, v...)
}
return func(s State) State {
return s.WithValue(keyEnv, getEnv(s).AddOrReplace(key, fmt.Sprintf(value, v...)))
return s.WithValue(keyEnv, getEnv(s).AddOrReplace(key, value))
}
}
func dir(str string) StateOption {
return dirf(str)
return dirf(str, false)
}
func dirf(str string, v ...interface{}) StateOption {
func dirf(value string, replace bool, v ...interface{}) StateOption {
if replace {
value = fmt.Sprintf(value, v...)
}
return func(s State) State {
value := fmt.Sprintf(str, v...)
if !path.IsAbs(value) {
prev := getDir(s)
if prev == "" {
@@ -100,9 +105,12 @@ func args(args ...string) StateOption {
}
}
func shlexf(str string, v ...interface{}) StateOption {
func shlexf(str string, replace bool, v ...interface{}) StateOption {
if replace {
str = fmt.Sprintf(str, v...)
}
return func(s State) State {
arg, err := shlex.Split(fmt.Sprintf(str, v...))
arg, err := shlex.Split(str)
if err != nil {
// TODO: handle error
}

View File

@@ -240,18 +240,18 @@ func (s State) File(a *FileAction, opts ...ConstraintsOpt) State {
}
func (s State) AddEnv(key, value string) State {
return s.AddEnvf(key, value)
return addEnvf(key, value, false)(s)
}
func (s State) AddEnvf(key, value string, v ...interface{}) State {
return addEnvf(key, value, v...)(s)
return addEnvf(key, value, true, v...)(s)
}
func (s State) Dir(str string) State {
return s.Dirf(str)
return dirf(str, false)(s)
}
func (s State) Dirf(str string, v ...interface{}) State {
return dirf(str, v...)(s)
return dirf(str, true, v...)(s)
}
func (s State) GetEnv(key string) (string, bool) {

View File

@@ -46,8 +46,8 @@ type SolveOpt struct {
type ExportEntry struct {
Type string
Attrs map[string]string
Output io.WriteCloser // for ExporterOCI and ExporterDocker
OutputDir string // for ExporterLocal
Output func(map[string]string) (io.WriteCloser, error) // for ExporterOCI and ExporterDocker
OutputDir string // for ExporterLocal
}
type CacheOptionsEntry struct {

View File

@@ -128,7 +128,7 @@ func (c *grpcClient) Run(ctx context.Context, f client.BuildFunc) (retError erro
}
}
if retError != nil {
st, _ := status.FromError(retError)
st, _ := status.FromError(errors.Cause(retError))
stp := st.Proto()
req.Error = &rpc.Status{
Code: stp.Code,

View File

@@ -4,6 +4,7 @@ import (
"context"
"github.com/moby/buildkit/session"
"github.com/pkg/errors"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
)
@@ -16,10 +17,10 @@ func CredentialsFunc(ctx context.Context, c session.Caller) func(string) (string
Host: host,
})
if err != nil {
if st, ok := status.FromError(err); ok && st.Code() == codes.Unimplemented {
if st, ok := status.FromError(errors.Cause(err)); ok && st.Code() == codes.Unimplemented {
return "", "", nil
}
return "", "", err
return "", "", errors.WithStack(err)
}
return resp.Username, resp.Secret, nil
}

View File

@@ -9,6 +9,7 @@ import (
"github.com/moby/buildkit/session"
digest "github.com/opencontainers/go-digest"
ocispec "github.com/opencontainers/image-spec/specs-go/v1"
"github.com/pkg/errors"
"google.golang.org/grpc/metadata"
)
@@ -31,47 +32,53 @@ func (cs *callerContentStore) choose(ctx context.Context) context.Context {
func (cs *callerContentStore) Info(ctx context.Context, dgst digest.Digest) (content.Info, error) {
ctx = cs.choose(ctx)
return cs.store.Info(ctx, dgst)
info, err := cs.store.Info(ctx, dgst)
return info, errors.WithStack(err)
}
func (cs *callerContentStore) Update(ctx context.Context, info content.Info, fieldpaths ...string) (content.Info, error) {
ctx = cs.choose(ctx)
return cs.store.Update(ctx, info, fieldpaths...)
info, err := cs.store.Update(ctx, info, fieldpaths...)
return info, errors.WithStack(err)
}
func (cs *callerContentStore) Walk(ctx context.Context, fn content.WalkFunc, fs ...string) error {
ctx = cs.choose(ctx)
return cs.store.Walk(ctx, fn, fs...)
return errors.WithStack(cs.store.Walk(ctx, fn, fs...))
}
func (cs *callerContentStore) Delete(ctx context.Context, dgst digest.Digest) error {
ctx = cs.choose(ctx)
return cs.store.Delete(ctx, dgst)
return errors.WithStack(cs.store.Delete(ctx, dgst))
}
func (cs *callerContentStore) ListStatuses(ctx context.Context, fs ...string) ([]content.Status, error) {
ctx = cs.choose(ctx)
return cs.store.ListStatuses(ctx, fs...)
resp, err := cs.store.ListStatuses(ctx, fs...)
return resp, errors.WithStack(err)
}
func (cs *callerContentStore) Status(ctx context.Context, ref string) (content.Status, error) {
ctx = cs.choose(ctx)
return cs.store.Status(ctx, ref)
st, err := cs.store.Status(ctx, ref)
return st, errors.WithStack(err)
}
func (cs *callerContentStore) Abort(ctx context.Context, ref string) error {
ctx = cs.choose(ctx)
return cs.store.Abort(ctx, ref)
return errors.WithStack(cs.store.Abort(ctx, ref))
}
func (cs *callerContentStore) Writer(ctx context.Context, opts ...content.WriterOpt) (content.Writer, error) {
ctx = cs.choose(ctx)
return cs.store.Writer(ctx, opts...)
w, err := cs.store.Writer(ctx, opts...)
return w, errors.WithStack(err)
}
func (cs *callerContentStore) ReaderAt(ctx context.Context, desc ocispec.Descriptor) (content.ReaderAt, error) {
ctx = cs.choose(ctx)
return cs.store.ReaderAt(ctx, desc)
ra, err := cs.store.ReaderAt(ctx, desc)
return ra, errors.WithStack(err)
}
// NewCallerStore creates content.Store from session.Caller with specified storeID

View File

@@ -14,7 +14,7 @@ import (
)
func sendDiffCopy(stream grpc.Stream, fs fsutil.FS, progress progressCb) error {
return fsutil.Send(stream.Context(), stream, fs, progress)
return errors.WithStack(fsutil.Send(stream.Context(), stream, fs, progress))
}
func newStreamWriter(stream grpc.ClientStream) io.WriteCloser {
@@ -29,7 +29,7 @@ type bufferedWriteCloser struct {
func (bwc *bufferedWriteCloser) Close() error {
if err := bwc.Writer.Flush(); err != nil {
return err
return errors.WithStack(err)
}
return bwc.Closer.Close()
}
@@ -40,19 +40,25 @@ type streamWriterCloser struct {
func (wc *streamWriterCloser) Write(dt []byte) (int, error) {
if err := wc.ClientStream.SendMsg(&BytesMessage{Data: dt}); err != nil {
return 0, err
// SendMsg return EOF on remote errors
if errors.Cause(err) == io.EOF {
if err := errors.WithStack(wc.ClientStream.RecvMsg(struct{}{})); err != nil {
return 0, err
}
}
return 0, errors.WithStack(err)
}
return len(dt), nil
}
func (wc *streamWriterCloser) Close() error {
if err := wc.ClientStream.CloseSend(); err != nil {
return err
return errors.WithStack(err)
}
// block until receiver is done
var bm BytesMessage
if err := wc.ClientStream.RecvMsg(&bm); err != io.EOF {
return err
return errors.WithStack(err)
}
return nil
}
@@ -69,19 +75,19 @@ func recvDiffCopy(ds grpc.Stream, dest string, cu CacheUpdater, progress progres
cf = cu.HandleChange
ch = cu.ContentHasher()
}
return fsutil.Receive(ds.Context(), ds, dest, fsutil.ReceiveOpt{
return errors.WithStack(fsutil.Receive(ds.Context(), ds, dest, fsutil.ReceiveOpt{
NotifyHashed: cf,
ContentHasher: ch,
ProgressCb: progress,
Filter: fsutil.FilterFunc(filter),
})
}))
}
func syncTargetDiffCopy(ds grpc.Stream, dest string) error {
if err := os.MkdirAll(dest, 0700); err != nil {
return err
return errors.Wrapf(err, "failed to create synctarget dest dir %s", dest)
}
return fsutil.Receive(ds.Context(), ds, dest, fsutil.ReceiveOpt{
return errors.WithStack(fsutil.Receive(ds.Context(), ds, dest, fsutil.ReceiveOpt{
Merge: true,
Filter: func() func(string, *fstypes.Stat) bool {
uid := os.Getuid()
@@ -92,7 +98,7 @@ func syncTargetDiffCopy(ds grpc.Stream, dest string) error {
return true
}
}(),
})
}))
}
func writeTargetFile(ds grpc.Stream, wc io.WriteCloser) error {
@@ -102,10 +108,10 @@ func writeTargetFile(ds grpc.Stream, wc io.WriteCloser) error {
if errors.Cause(err) == io.EOF {
return nil
}
return err
return errors.WithStack(err)
}
if _, err := wc.Write(bm.Data); err != nil {
return err
return errors.WithStack(err)
}
}
}

View File

@@ -18,11 +18,12 @@ import (
)
const (
keyOverrideExcludes = "override-excludes"
keyIncludePatterns = "include-patterns"
keyExcludePatterns = "exclude-patterns"
keyFollowPaths = "followpaths"
keyDirName = "dir-name"
keyOverrideExcludes = "override-excludes"
keyIncludePatterns = "include-patterns"
keyExcludePatterns = "exclude-patterns"
keyFollowPaths = "followpaths"
keyDirName = "dir-name"
keyExporterMetaPrefix = "exporter-md-"
)
type fsSyncProvider struct {
@@ -238,16 +239,16 @@ func NewFSSyncTargetDir(outdir string) session.Attachable {
}
// NewFSSyncTarget allows writing into an io.WriteCloser
func NewFSSyncTarget(w io.WriteCloser) session.Attachable {
func NewFSSyncTarget(f func(map[string]string) (io.WriteCloser, error)) session.Attachable {
p := &fsSyncTarget{
outfile: w,
f: f,
}
return p
}
type fsSyncTarget struct {
outdir string
outfile io.WriteCloser
outdir string
f func(map[string]string) (io.WriteCloser, error)
}
func (sp *fsSyncTarget) Register(server *grpc.Server) {
@@ -258,11 +259,26 @@ func (sp *fsSyncTarget) DiffCopy(stream FileSend_DiffCopyServer) error {
if sp.outdir != "" {
return syncTargetDiffCopy(stream, sp.outdir)
}
if sp.outfile == nil {
if sp.f == nil {
return errors.New("empty outfile and outdir")
}
defer sp.outfile.Close()
return writeTargetFile(stream, sp.outfile)
opts, _ := metadata.FromIncomingContext(stream.Context()) // if no metadata continue with empty object
md := map[string]string{}
for k, v := range opts {
if strings.HasPrefix(k, keyExporterMetaPrefix) {
md[strings.TrimPrefix(k, keyExporterMetaPrefix)] = strings.Join(v, ",")
}
}
wc, err := sp.f(md)
if err != nil {
return err
}
if wc == nil {
return status.Errorf(codes.AlreadyExists, "target already exists")
}
defer wc.Close()
return writeTargetFile(stream, wc)
}
func CopyToCaller(ctx context.Context, fs fsutil.FS, c session.Caller, progress func(int, bool)) error {
@@ -275,13 +291,13 @@ func CopyToCaller(ctx context.Context, fs fsutil.FS, c session.Caller, progress
cc, err := client.DiffCopy(ctx)
if err != nil {
return err
return errors.WithStack(err)
}
return sendDiffCopy(cc, fs, progress)
}
func CopyFileWriter(ctx context.Context, c session.Caller) (io.WriteCloser, error) {
func CopyFileWriter(ctx context.Context, md map[string]string, c session.Caller) (io.WriteCloser, error) {
method := session.MethodURL(_FileSend_serviceDesc.ServiceName, "diffcopy")
if !c.Supports(method) {
return nil, errors.Errorf("method %s not supported by the client", method)
@@ -289,9 +305,16 @@ func CopyFileWriter(ctx context.Context, c session.Caller) (io.WriteCloser, erro
client := NewFileSendClient(c.Conn())
opts := make(map[string][]string, len(md))
for k, v := range md {
opts[keyExporterMetaPrefix+k] = []string{v}
}
ctx = metadata.NewOutgoingContext(ctx, opts)
cc, err := client.DiffCopy(ctx)
if err != nil {
return nil, err
return nil, errors.WithStack(err)
}
return newStreamWriter(cc), nil

View File

@@ -21,10 +21,10 @@ func GetSecret(ctx context.Context, c session.Caller, id string) ([]byte, error)
ID: id,
})
if err != nil {
if st, ok := status.FromError(err); ok && (st.Code() == codes.Unimplemented || st.Code() == codes.NotFound) {
if st, ok := status.FromError(errors.Cause(err)); ok && (st.Code() == codes.Unimplemented || st.Code() == codes.NotFound) {
return nil, errors.Wrapf(ErrNotFound, "secret %s not found", id)
}
return nil, err
return nil, errors.WithStack(err)
}
return resp.Data, nil
}

View File

@@ -3,23 +3,24 @@ package sshforward
import (
io "io"
"github.com/pkg/errors"
context "golang.org/x/net/context"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc"
)
func Copy(ctx context.Context, conn io.ReadWriteCloser, stream grpc.Stream) error {
func Copy(ctx context.Context, conn io.ReadWriteCloser, stream grpc.Stream, closeStream func() error) error {
g, ctx := errgroup.WithContext(ctx)
g.Go(func() (retErr error) {
p := &BytesMessage{}
for {
if err := stream.RecvMsg(p); err != nil {
conn.Close()
if err == io.EOF {
return nil
}
conn.Close()
return err
return errors.WithStack(err)
}
select {
case <-ctx.Done():
@@ -29,7 +30,7 @@ func Copy(ctx context.Context, conn io.ReadWriteCloser, stream grpc.Stream) erro
}
if _, err := conn.Write(p.Data); err != nil {
conn.Close()
return err
return errors.WithStack(err)
}
p.Data = p.Data[:0]
}
@@ -41,9 +42,12 @@ func Copy(ctx context.Context, conn io.ReadWriteCloser, stream grpc.Stream) erro
n, err := conn.Read(buf)
switch {
case err == io.EOF:
if closeStream != nil {
closeStream()
}
return nil
case err != nil:
return err
return errors.WithStack(err)
}
select {
case <-ctx.Done():
@@ -52,7 +56,7 @@ func Copy(ctx context.Context, conn io.ReadWriteCloser, stream grpc.Stream) erro
}
p := &BytesMessage{Data: buf[:n]}
if err := stream.SendMsg(p); err != nil {
return err
return errors.WithStack(err)
}
}
})

View File

@@ -7,6 +7,7 @@ import (
"path/filepath"
"github.com/moby/buildkit/session"
"github.com/pkg/errors"
context "golang.org/x/net/context"
"golang.org/x/sync/errgroup"
"google.golang.org/grpc/metadata"
@@ -48,7 +49,7 @@ func (s *server) run(ctx context.Context, l net.Listener, id string) error {
return err
}
go Copy(ctx, conn, stream)
go Copy(ctx, conn, stream, stream.CloseSend)
}
})
@@ -65,7 +66,7 @@ type SocketOpt struct {
func MountSSHSocket(ctx context.Context, c session.Caller, opt SocketOpt) (sockPath string, closer func() error, err error) {
dir, err := ioutil.TempDir("", ".buildkit-ssh-sock")
if err != nil {
return "", nil, err
return "", nil, errors.WithStack(err)
}
defer func() {
@@ -78,16 +79,16 @@ func MountSSHSocket(ctx context.Context, c session.Caller, opt SocketOpt) (sockP
l, err := net.Listen("unix", sockPath)
if err != nil {
return "", nil, err
return "", nil, errors.WithStack(err)
}
if err := os.Chown(sockPath, opt.UID, opt.GID); err != nil {
l.Close()
return "", nil, err
return "", nil, errors.WithStack(err)
}
if err := os.Chmod(sockPath, os.FileMode(opt.Mode)); err != nil {
l.Close()
return "", nil, err
return "", nil, errors.WithStack(err)
}
s := &server{caller: c}
@@ -102,12 +103,12 @@ func MountSSHSocket(ctx context.Context, c session.Caller, opt SocketOpt) (sockP
return sockPath, func() error {
err := l.Close()
os.RemoveAll(sockPath)
return err
return errors.WithStack(err)
}, nil
}
func CheckSSHID(ctx context.Context, c session.Caller, id string) error {
client := NewSSHClient(c.Conn())
_, err := client.CheckAgent(ctx, &CheckAgentRequest{ID: id})
return err
return errors.WithStack(err)
}

View File

@@ -114,7 +114,7 @@ func (sp *socketProvider) ForwardAgent(stream sshforward.SSH_ForwardAgentServer)
eg.Go(func() error {
defer s1.Close()
return sshforward.Copy(ctx, s2, stream)
return sshforward.Copy(ctx, s2, stream, nil)
})
return eg.Wait()

View File

@@ -6,6 +6,7 @@ import (
"net/url"
"github.com/moby/buildkit/session"
"github.com/pkg/errors"
"google.golang.org/grpc/metadata"
)
@@ -26,7 +27,7 @@ func New(ctx context.Context, c session.Caller, url *url.URL) (*Upload, error) {
cc, err := client.Pull(ctx)
if err != nil {
return nil, err
return nil, errors.WithStack(err)
}
return &Upload{cc: cc}, nil
@@ -44,12 +45,12 @@ func (u *Upload) WriteTo(w io.Writer) (int, error) {
if err == io.EOF {
return n, nil
}
return n, err
return n, errors.WithStack(err)
}
nn, err := w.Write(bm.Data)
n += nn
if err != nil {
return n, err
return n, errors.WithStack(err)
}
}
}

View File

@@ -30,19 +30,20 @@ const (
CapBuildOpLLBFileName apicaps.CapID = "source.buildop.llbfilename"
CapExecMetaBase apicaps.CapID = "exec.meta.base"
CapExecMetaProxy apicaps.CapID = "exec.meta.proxyenv"
CapExecMetaNetwork apicaps.CapID = "exec.meta.network"
CapExecMetaSecurity apicaps.CapID = "exec.meta.security"
CapExecMetaSetsDefaultPath apicaps.CapID = "exec.meta.setsdefaultpath"
CapExecMountBind apicaps.CapID = "exec.mount.bind"
CapExecMountCache apicaps.CapID = "exec.mount.cache"
CapExecMountCacheSharing apicaps.CapID = "exec.mount.cache.sharing"
CapExecMountSelector apicaps.CapID = "exec.mount.selector"
CapExecMountTmpfs apicaps.CapID = "exec.mount.tmpfs"
CapExecMountSecret apicaps.CapID = "exec.mount.secret"
CapExecMountSSH apicaps.CapID = "exec.mount.ssh"
CapExecCgroupsMounted apicaps.CapID = "exec.cgroup"
CapExecMetaBase apicaps.CapID = "exec.meta.base"
CapExecMetaProxy apicaps.CapID = "exec.meta.proxyenv"
CapExecMetaNetwork apicaps.CapID = "exec.meta.network"
CapExecMetaSecurity apicaps.CapID = "exec.meta.security"
CapExecMetaSetsDefaultPath apicaps.CapID = "exec.meta.setsdefaultpath"
CapExecMountBind apicaps.CapID = "exec.mount.bind"
CapExecMountBindReadWriteNoOuput apicaps.CapID = "exec.mount.bind.readwrite-nooutput"
CapExecMountCache apicaps.CapID = "exec.mount.cache"
CapExecMountCacheSharing apicaps.CapID = "exec.mount.cache.sharing"
CapExecMountSelector apicaps.CapID = "exec.mount.selector"
CapExecMountTmpfs apicaps.CapID = "exec.mount.tmpfs"
CapExecMountSecret apicaps.CapID = "exec.mount.secret"
CapExecMountSSH apicaps.CapID = "exec.mount.ssh"
CapExecCgroupsMounted apicaps.CapID = "exec.cgroup"
CapFileBase apicaps.CapID = "file.base"
@@ -193,6 +194,12 @@ func init() {
Status: apicaps.CapStatusExperimental,
})
Caps.Init(apicaps.Cap{
ID: CapExecMountBindReadWriteNoOuput,
Enabled: true,
Status: apicaps.CapStatusExperimental,
})
Caps.Init(apicaps.Cap{
ID: CapExecMountCache,
Enabled: true,

View File

@@ -49,6 +49,9 @@ func mkstat(path, relpath string, fi os.FileInfo, inodemap map[uint64]string) (*
stat.Mode = noPermPart | permPart
}
// Clear the socket bit since archive/tar.FileInfoHeader does not handle it
stat.Mode &^= uint32(os.ModeSocket)
return stat, nil
}

26
vendor/modules.txt vendored
View File

@@ -27,7 +27,7 @@ github.com/agl/ed25519/edwards25519
github.com/beorn7/perks/quantile
# github.com/containerd/console v0.0.0-20181022165439-0650fd9eeb50
github.com/containerd/console
# github.com/containerd/containerd v1.3.0-0.20190426060238-3a3f0aac8819
# github.com/containerd/containerd v1.3.0-0.20190507210959-7c1e88399ec0
github.com/containerd/containerd/images
github.com/containerd/containerd/platforms
github.com/containerd/containerd/content
@@ -36,20 +36,22 @@ github.com/containerd/containerd/remotes
github.com/containerd/containerd/remotes/docker
github.com/containerd/containerd/log
github.com/containerd/containerd/content/local
github.com/containerd/containerd/containers
github.com/containerd/containerd/oci
github.com/containerd/containerd/labels
github.com/containerd/containerd/reference
github.com/containerd/containerd/remotes/docker/schema1
github.com/containerd/containerd/version
github.com/containerd/containerd/filters
github.com/containerd/containerd/sys
github.com/containerd/containerd/api/services/content/v1
github.com/containerd/containerd/content/proxy
github.com/containerd/containerd/services/content/contentserver
github.com/containerd/containerd/containers
github.com/containerd/containerd/oci
github.com/containerd/containerd
github.com/containerd/containerd/namespaces
github.com/containerd/containerd/mount
github.com/containerd/containerd/namespaces
github.com/containerd/containerd/snapshots
github.com/containerd/containerd
github.com/containerd/containerd/archive/compression
github.com/containerd/containerd/api/services/containers/v1
github.com/containerd/containerd/api/services/diff/v1
github.com/containerd/containerd/api/services/events/v1
@@ -62,7 +64,6 @@ github.com/containerd/containerd/api/services/tasks/v1
github.com/containerd/containerd/api/services/version/v1
github.com/containerd/containerd/api/types
github.com/containerd/containerd/archive
github.com/containerd/containerd/archive/compression
github.com/containerd/containerd/cio
github.com/containerd/containerd/defaults
github.com/containerd/containerd/diff
@@ -73,7 +74,6 @@ github.com/containerd/containerd/leases
github.com/containerd/containerd/leases/proxy
github.com/containerd/containerd/pkg/dialer
github.com/containerd/containerd/plugin
github.com/containerd/containerd/remotes/docker/schema1
github.com/containerd/containerd/rootfs
github.com/containerd/containerd/runtime/linux/runctypes
github.com/containerd/containerd/runtime/v2/runc/options
@@ -81,14 +81,14 @@ github.com/containerd/containerd/snapshots/proxy
github.com/containerd/containerd/api/types/task
github.com/containerd/containerd/events/exchange
github.com/containerd/containerd/identifiers
# github.com/containerd/continuity v0.0.0-20190426062206-aaeac12a7ffc
# github.com/containerd/continuity v0.0.0-20190827140505-75bee3e2ccb6
github.com/containerd/continuity
github.com/containerd/continuity/fs
github.com/containerd/continuity/pathdriver
github.com/containerd/continuity/devices
github.com/containerd/continuity/driver
github.com/containerd/continuity/proto
github.com/containerd/continuity/sysx
github.com/containerd/continuity/fs
github.com/containerd/continuity/syscallx
# github.com/containerd/fifo v0.0.0-20190226154929-a9fb20d87448
github.com/containerd/fifo
@@ -259,13 +259,14 @@ github.com/matttproud/golang_protobuf_extensions/pbutil
github.com/miekg/pkcs11
# github.com/mitchellh/mapstructure v1.1.2
github.com/mitchellh/mapstructure
# github.com/moby/buildkit v0.5.2-0.20190513182223-f238f1efb04f
# github.com/moby/buildkit v0.6.2-0.20190921002054-ae10b292fefb
github.com/moby/buildkit/session/auth/authprovider
github.com/moby/buildkit/client
github.com/moby/buildkit/session
github.com/moby/buildkit/session/secrets/secretsprovider
github.com/moby/buildkit/session/sshforward/sshprovider
github.com/moby/buildkit/session/upload/uploadprovider
github.com/moby/buildkit/util/entitlements
github.com/moby/buildkit/util/appcontext
github.com/moby/buildkit/identity
github.com/moby/buildkit/util/progress/progressui
@@ -285,7 +286,6 @@ github.com/moby/buildkit/session/grpchijack
github.com/moby/buildkit/solver/pb
github.com/moby/buildkit/util/apicaps
github.com/moby/buildkit/util/appdefaults
github.com/moby/buildkit/util/entitlements
github.com/moby/buildkit/session/secrets
github.com/moby/buildkit/session/sshforward
github.com/moby/buildkit/session/upload
@@ -303,7 +303,7 @@ github.com/opencontainers/go-digest
github.com/opencontainers/image-spec/specs-go/v1
github.com/opencontainers/image-spec/specs-go
github.com/opencontainers/image-spec/identity
# github.com/opencontainers/runc v1.0.1-0.20190307181833-2b18fe1d885e
# github.com/opencontainers/runc v1.0.0-rc8
github.com/opencontainers/runc/libcontainer/user
github.com/opencontainers/runc/libcontainer/system
# github.com/opencontainers/runtime-spec v1.0.1
@@ -355,7 +355,7 @@ github.com/theupdateframework/notary/trustmanager/yubikey
github.com/theupdateframework/notary/tuf
github.com/theupdateframework/notary/tuf/utils
github.com/theupdateframework/notary/tuf/validation
# github.com/tonistiigi/fsutil v0.0.0-20190327153851-3bbb99cdbd76
# github.com/tonistiigi/fsutil v0.0.0-20190819224149-3d2716dd0a4d
github.com/tonistiigi/fsutil/types
github.com/tonistiigi/fsutil
# github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea