Commit Graph

21 Commits (8d6eaf6e39b6d77debf7757d76a512a9ef71e087)

Author SHA1 Message Date
Tonis Tiigi 97eff70c5e inline remote cache support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
Akihiro Suda 6e7617e889 support local cache exporter and importer
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Tonis Tiigi 9c6eb52a84 dockerfile: avoid duplicate config resolve lines
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 13:42:21 -07:00
Tõnis Tiigi e3bc522f7c
Merge pull request #582 from ijc/gateway-client-snapshot-leak
clientside-frontend: release non-result refs after completion
2018-08-31 09:22:28 -07:00
Ian Campbell 59391d1647 clientside-frontend: release non-result refs after completion
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522)
because we weren't releasing the intermediate refs.

Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.

The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.

Fixes #581.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
Ian Campbell fdf13cf96e Allow some Metadata to be return from frontend to the control client
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-17 12:45:21 +01:00
Akihiro Suda af46188e9b
Merge pull request #533 from ijc/client-gateway
access gateway API from client
2018-08-17 05:12:23 +09:00
Ian Campbell 22f632f895 allow frontends to be run directly on the control client side
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.

Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.

The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
Tonis Tiigi 95e0348f57 dockerfile: enable global net-mode settings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:40 -07:00
Tonis Tiigi 130f5f5ab0 solver: net host with basic entitlements support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:35 -07:00
Ian Campbell f7937f1989 solver: avoid dereferencing nil res.Metadata
I saw this:
    panic: assignment to entry in nil map

    goroutine 3173 [running]:
    github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export(0xc42094ac40, 0xecac60, 0xc4211ca690, 0xed12c0, 0xc42000fca8, 0x0, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/exporter/oci/export.go:113 +0x156
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2(0xecac60, 0xc4211ca690, 0xc4202207c0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:132 +0x7d
    github.com/moby/buildkit/solver/llbsolver.inVertexContext(0xecac60, 0xc4211ca690, 0xe2ec38, 0x1d, 0xc420d43400, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0x1f6
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc4203f7dc0, 0xecac60, 0xc420deb830, 0xc4203ed200, 0x19, 0xc4202ff840, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:131 +0x761
    github.com/moby/buildkit/control.(*Controller).Solve(0xc42017e870, 0xecac60, 0xc420deb830, 0xc4201edd40, 0xc42017e870, 0x1, 0x1)
    	/go/src/github.com/moby/buildkit/control/control.go:207 +0x4b8
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xecac60, 0xc420deb800, 0xed67a0, 0x1595288)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:810 +0x86
    github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0xebbea0, 0xc420188310)
    	/go/src/github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2ba
    main.unaryInterceptor.func1(0xecaba0, 0xc4206afe40, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/cmd/buildkitd/main.go:330 +0x15f
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd76a00, 0xc42017e870, 0xecac60, 0xc420deb530, 0xc42028d030, 0xc420450760, 0x0, 0x0, 0x34, 0x3)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:812 +0x167
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0xc420497650, 0x152cef8, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1011 +0x4fc
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1249 +0x1318
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42063e0f0, 0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:680 +0x9f
    created by github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:678 +0xa1

Which was due to `res, err := s.Bridge(j).Solve(ctx, req)` having `res.Metadata
== nil`. There are several paths in `llbBridge.Solve()` where this can be the
case, plus a case where this comes from a frontend which should not be allowed
to crash the daemon.

Likely introduced by d70d816dee or 6be1257f5d.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-03 16:08:04 +01:00
Tonis Tiigi fe3388e70d solver: change builder.call to builder.context
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:39:14 -07:00
Tonis Tiigi d70d816dee exporter: update export signature
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 14:37:01 -07:00
Akihiro Suda 8fa31102d5
Merge pull request #539 from tonistiigi/fix-cache-metadata-release
Fix cache metadata release
2018-07-24 13:36:22 +09:00
Tonis Tiigi a7fedd9c9f control: clean up cache metadata after prune
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 16:27:01 -07:00
Tonis Tiigi 97ea41ee6a solver: improve multi-key root caching
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 14:27:35 -07:00
Tonis Tiigi 303b5da713 llbsolver: use result type with multi ref support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
Tonis Tiigi 01b7a29e86 gateway: return result from solve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Akihiro Suda 80d2f820f9 decopule cache/remotecache from registry
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
Tonis Tiigi b444b9f323 llbsolver: validate runtime platforms for exec op
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:38:03 -07:00
Tonis Tiigi 71f5e6dd21 solver: rename solver-next to solver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00