Commit Graph

162 Commits (bc07b2b81b1c6a62d29981ac564b16a15ce2bfa7)

Author SHA1 Message Date
Tonis Tiigi 07810f0aae llbsolver: validate digest on load
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-04 17:06:34 -08:00
Tonis Tiigi 97eff70c5e inline remote cache support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
Tonis Tiigi ac4037a1d9 solver: keep original cache creation date when copying between stores
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:34 -08:00
Tonis Tiigi 1ec543e384 solver: exclude randomized cache keys from exporter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:25 -08:00
Akihiro Suda 6e7617e889 support local cache exporter and importer
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Tonis Tiigi b521aae3ea dockerfile: allow setting file mode/uid for secrets
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:40 -08:00
Tonis Tiigi 2529762ae7 dockerfile: allow setting file mode/uid for ssh socket
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:34 -08:00
Tibor Vass c9b01072d0 solver: specify SSH key ID in error message when required key was not forwarded
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-10-05 22:50:07 +00:00
Ian Campbell 720d8327e2 Push setting of default PATH down into the executor(s)
Setting the default `PATH` in the `llb.State` on the client side means it
depends on the `GOOS` of the buildkit client, rather than of the environment
where it will actually execute.

Instead defer this to execution time and insert the default PATH at that point
if one is not present. Doing this in solver/llbsolver/ops/exec covers all
executors and also avoids breaking the cache.

Client compatibility is handled via a new capability.

Fixes #604

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-09-28 14:16:47 +01:00
Akihiro Suda aa73951164
Merge pull request #631 from jhowardmsft/boltdb
Revendoring to move boltdb to bbolt
2018-09-19 13:59:08 +09:00
John Howard 2de2c04c8e Revendoring to move boltdb to bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-18 11:18:08 -07:00
Tonis Tiigi 9c6eb52a84 dockerfile: avoid duplicate config resolve lines
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 13:42:21 -07:00
Tonis Tiigi 6888956557 sshforward: implement ssh socket forwarding
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 08:57:06 -07:00
Tõnis Tiigi e3bc522f7c
Merge pull request #582 from ijc/gateway-client-snapshot-leak
clientside-frontend: release non-result refs after completion
2018-08-31 09:22:28 -07:00
Tõnis Tiigi 45aedff363
Merge pull request #571 from ijc/solver-def-and-frontend-mutually-exclusive
solver: Don't solve both the provided Defininiton & frontend
2018-08-30 10:10:52 -07:00
Ian Campbell 59391d1647 clientside-frontend: release non-result refs after completion
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522)
because we weren't releasing the intermediate refs.

Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.

The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.

Fixes #581.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
Ian Campbell cc83b34572 solver: remove redundant conditional
There is now no way to reach this point with the test being false.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:01:27 +01:00
Ian Campbell ab7928edff solver: error if both Definition and Frontend are given.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 13:59:26 +01:00
Ian Campbell fdf13cf96e Allow some Metadata to be return from frontend to the control client
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-17 12:45:21 +01:00
Akihiro Suda af46188e9b
Merge pull request #533 from ijc/client-gateway
access gateway API from client
2018-08-17 05:12:23 +09:00
Ian Campbell e05be2fd2c solver: Don't solve both the provided Defininiton & frontend
This seems wasteful. I think there is no side-effects of the Load+Build of the
definition which would make a difference to the frontend solve.

Note that this changes the presedence of the actual result, previously the
frontend result would overwrite the definition's one, while now the frontend
would not be run.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-15 16:20:02 +01:00
Ian Campbell 22f632f895 allow frontends to be run directly on the control client side
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.

Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.

The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
Tonis Tiigi 95e0348f57 dockerfile: enable global net-mode settings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:40 -07:00
Tonis Tiigi 130f5f5ab0 solver: net host with basic entitlements support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:35 -07:00
Tonis Tiigi 4945fe758c llbsolver: add support for extra host records
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-03 13:56:13 -07:00
Ian Campbell f7937f1989 solver: avoid dereferencing nil res.Metadata
I saw this:
    panic: assignment to entry in nil map

    goroutine 3173 [running]:
    github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export(0xc42094ac40, 0xecac60, 0xc4211ca690, 0xed12c0, 0xc42000fca8, 0x0, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/exporter/oci/export.go:113 +0x156
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2(0xecac60, 0xc4211ca690, 0xc4202207c0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:132 +0x7d
    github.com/moby/buildkit/solver/llbsolver.inVertexContext(0xecac60, 0xc4211ca690, 0xe2ec38, 0x1d, 0xc420d43400, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0x1f6
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc4203f7dc0, 0xecac60, 0xc420deb830, 0xc4203ed200, 0x19, 0xc4202ff840, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:131 +0x761
    github.com/moby/buildkit/control.(*Controller).Solve(0xc42017e870, 0xecac60, 0xc420deb830, 0xc4201edd40, 0xc42017e870, 0x1, 0x1)
    	/go/src/github.com/moby/buildkit/control/control.go:207 +0x4b8
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xecac60, 0xc420deb800, 0xed67a0, 0x1595288)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:810 +0x86
    github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0xebbea0, 0xc420188310)
    	/go/src/github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2ba
    main.unaryInterceptor.func1(0xecaba0, 0xc4206afe40, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/cmd/buildkitd/main.go:330 +0x15f
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd76a00, 0xc42017e870, 0xecac60, 0xc420deb530, 0xc42028d030, 0xc420450760, 0x0, 0x0, 0x34, 0x3)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:812 +0x167
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0xc420497650, 0x152cef8, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1011 +0x4fc
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1249 +0x1318
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42063e0f0, 0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:680 +0x9f
    created by github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:678 +0xa1

Which was due to `res, err := s.Bridge(j).Solve(ctx, req)` having `res.Metadata
== nil`. There are several paths in `llbBridge.Solve()` where this can be the
case, plus a case where this comes from a frontend which should not be allowed
to crash the daemon.

Likely introduced by d70d816dee or 6be1257f5d.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-03 16:08:04 +01:00
Tonis Tiigi 4c44d2c6e4 dockerfile: custom names to vertexes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-31 10:41:30 -07:00
Tonis Tiigi b52c0002db llbsolver: show status of resolving image config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:59:11 -07:00
Tonis Tiigi fe3388e70d solver: change builder.call to builder.context
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:39:14 -07:00
Tonis Tiigi 57b96a0ee5 cache: add record type field to usage record
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi d70d816dee exporter: update export signature
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 14:37:01 -07:00
Akihiro Suda 8fa31102d5
Merge pull request #539 from tonistiigi/fix-cache-metadata-release
Fix cache metadata release
2018-07-24 13:36:22 +09:00
Tonis Tiigi a7fedd9c9f control: clean up cache metadata after prune
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 16:27:01 -07:00
Tonis Tiigi 97ea41ee6a solver: improve multi-key root caching
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 14:27:35 -07:00
Tonis Tiigi 08d324d0dd exec: fix secret mounts for rootless
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:11 -07:00
Tonis Tiigi 2da2aff164 llbsolver: add secret mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:05 -07:00
Tonis Tiigi aa3aef3c25 llbsolver: validate llb caps
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
Tonis Tiigi 303b5da713 llbsolver: use result type with multi ref support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
Tonis Tiigi 01b7a29e86 gateway: return result from solve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Tonis Tiigi 57006ea333 cache: avoid commit on finalize when possible
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-11 16:21:27 -07:00
Tonis Tiigi ddf3f3e59c frontend: fix two data races
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-10 19:31:33 -07:00
Akihiro Suda 80d2f820f9 decopule cache/remotecache from registry
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
Akihiro Suda 1b34bd1e31
Merge pull request #464 from tonistiigi/image-platform
platform support for image and dockerfile
2018-06-27 14:36:02 +09:00
Tonis Tiigi e618878451 dockerfile: platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 11:06:08 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Tonis Tiigi b444b9f323 llbsolver: validate runtime platforms for exec op
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:38:03 -07:00
Tonis Tiigi 242697af59 llb: add marshal roundtrip tests for platform
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 17:55:36 -07:00
Tonis Tiigi ccbf185006 llbsolver: allow sharing modes for cache mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-19 15:57:20 -07:00
Tonis Tiigi 3c973bacfe llbsolver: allow concurrent cache mount access
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 20:02:30 -07:00
Tonis Tiigi 6f569455d0 dockerfile: enable tmpfs mounts on run —mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:42:38 -07:00
Tonis Tiigi 596b03a8a0 llbsolver: allow tmpfs mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:05:14 -07:00
Tonis Tiigi 479419abf2 llbsolver: don’t error on readonly scratch mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 16:23:53 -07:00
Akihiro Suda e2fc1a702d
Merge pull request #414 from tonistiigi/stdio-debug
exec: make stdio debugging optional
2018-05-26 06:59:04 +09:00
Tonis Tiigi 6b23ab01ce exec: make stdio debugging optional
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 11:15:32 -07:00
Tonis Tiigi 97ddeffdb2 exec: remove debug
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 10:45:10 -07:00
Tonis Tiigi eefc49ff72 dockerfile: detect empty images early
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 16:54:04 -07:00
Tonis Tiigi 8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Tonis Tiigi 66488dc6d8 exec: support proxy settings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-21 16:08:11 -07:00
Tonis Tiigi 2bd78c3c6e llbsolver: add support for cached mount type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:24:20 -07:00
Tonis Tiigi 692bdc8aa5 llbsolver: comments for exec op mount handling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:09:59 -07:00
Tonis Tiigi 71f5e6dd21 solver: rename solver-next to solver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00