Tonis Tiigi
07810f0aae
llbsolver: validate digest on load
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-04 17:06:34 -08:00
Tonis Tiigi
97eff70c5e
inline remote cache support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
Tonis Tiigi
ac4037a1d9
solver: keep original cache creation date when copying between stores
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:34 -08:00
Tonis Tiigi
1ec543e384
solver: exclude randomized cache keys from exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:25 -08:00
Akihiro Suda
6e7617e889
support local cache exporter and importer
...
Export:
$ buildctl build ... --export-cache type=local,store=/path/to/output-dir
Import:
$ buildctl build ... --import-cache type=local,store=/path/to/input-dir
Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`
Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Tonis Tiigi
b521aae3ea
dockerfile: allow setting file mode/uid for secrets
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:40 -08:00
Tonis Tiigi
2529762ae7
dockerfile: allow setting file mode/uid for ssh socket
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:34 -08:00
Tibor Vass
c9b01072d0
solver: specify SSH key ID in error message when required key was not forwarded
...
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-10-05 22:50:07 +00:00
Ian Campbell
720d8327e2
Push setting of default PATH down into the executor(s)
...
Setting the default `PATH` in the `llb.State` on the client side means it
depends on the `GOOS` of the buildkit client, rather than of the environment
where it will actually execute.
Instead defer this to execution time and insert the default PATH at that point
if one is not present. Doing this in solver/llbsolver/ops/exec covers all
executors and also avoids breaking the cache.
Client compatibility is handled via a new capability.
Fixes #604
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-09-28 14:16:47 +01:00
Akihiro Suda
aa73951164
Merge pull request #631 from jhowardmsft/boltdb
...
Revendoring to move boltdb to bbolt
2018-09-19 13:59:08 +09:00
John Howard
2de2c04c8e
Revendoring to move boltdb to bbolt
...
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-18 11:18:08 -07:00
Tonis Tiigi
9c6eb52a84
dockerfile: avoid duplicate config resolve lines
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 13:42:21 -07:00
Tonis Tiigi
6888956557
sshforward: implement ssh socket forwarding
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 08:57:06 -07:00
Tõnis Tiigi
e3bc522f7c
Merge pull request #582 from ijc/gateway-client-snapshot-leak
...
clientside-frontend: release non-result refs after completion
2018-08-31 09:22:28 -07:00
Tõnis Tiigi
45aedff363
Merge pull request #571 from ijc/solver-def-and-frontend-mutually-exclusive
...
solver: Don't solve both the provided Defininiton & frontend
2018-08-30 10:10:52 -07:00
Ian Campbell
59391d1647
clientside-frontend: release non-result refs after completion
...
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522 )
because we weren't releasing the intermediate refs.
Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.
The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.
Fixes #581 .
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
Ian Campbell
cc83b34572
solver: remove redundant conditional
...
There is now no way to reach this point with the test being false.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:01:27 +01:00
Ian Campbell
ab7928edff
solver: error if both Definition and Frontend are given.
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 13:59:26 +01:00
Ian Campbell
fdf13cf96e
Allow some Metadata to be return from frontend to the control client
...
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-17 12:45:21 +01:00
Akihiro Suda
af46188e9b
Merge pull request #533 from ijc/client-gateway
...
access gateway API from client
2018-08-17 05:12:23 +09:00
Ian Campbell
e05be2fd2c
solver: Don't solve both the provided Defininiton & frontend
...
This seems wasteful. I think there is no side-effects of the Load+Build of the
definition which would make a difference to the frontend solve.
Note that this changes the presedence of the actual result, previously the
frontend result would overwrite the definition's one, while now the frontend
would not be run.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-15 16:20:02 +01:00
Ian Campbell
22f632f895
allow frontends to be run directly on the control client side
...
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.
Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.
The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
Tonis Tiigi
95e0348f57
dockerfile: enable global net-mode settings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:40 -07:00
Tonis Tiigi
130f5f5ab0
solver: net host with basic entitlements support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:35 -07:00
Tonis Tiigi
4945fe758c
llbsolver: add support for extra host records
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-03 13:56:13 -07:00
Ian Campbell
f7937f1989
solver: avoid dereferencing nil res.Metadata
...
I saw this:
panic: assignment to entry in nil map
goroutine 3173 [running]:
github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export(0xc42094ac40, 0xecac60, 0xc4211ca690, 0xed12c0, 0xc42000fca8, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/exporter/oci/export.go:113 +0x156
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2(0xecac60, 0xc4211ca690, 0xc4202207c0, 0x0)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:132 +0x7d
github.com/moby/buildkit/solver/llbsolver.inVertexContext(0xecac60, 0xc4211ca690, 0xe2ec38, 0x1d, 0xc420d43400, 0x0, 0x0)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0x1f6
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc4203f7dc0, 0xecac60, 0xc420deb830, 0xc4203ed200, 0x19, 0xc4202ff840, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:131 +0x761
github.com/moby/buildkit/control.(*Controller).Solve(0xc42017e870, 0xecac60, 0xc420deb830, 0xc4201edd40, 0xc42017e870, 0x1, 0x1)
/go/src/github.com/moby/buildkit/control/control.go:207 +0x4b8
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xecac60, 0xc420deb800, 0xed67a0, 0x1595288)
/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:810 +0x86
github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0xebbea0, 0xc420188310)
/go/src/github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2ba
main.unaryInterceptor.func1(0xecaba0, 0xc4206afe40, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/cmd/buildkitd/main.go:330 +0x15f
github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd76a00, 0xc42017e870, 0xecac60, 0xc420deb530, 0xc42028d030, 0xc420450760, 0x0, 0x0, 0x34, 0x3)
/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:812 +0x167
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0xc420497650, 0x152cef8, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1011 +0x4fc
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0x0)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1249 +0x1318
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42063e0f0, 0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:680 +0x9f
created by github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:678 +0xa1
Which was due to `res, err := s.Bridge(j).Solve(ctx, req)` having `res.Metadata
== nil`. There are several paths in `llbBridge.Solve()` where this can be the
case, plus a case where this comes from a frontend which should not be allowed
to crash the daemon.
Likely introduced by d70d816dee
or 6be1257f5d
.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-03 16:08:04 +01:00
Tonis Tiigi
4c44d2c6e4
dockerfile: custom names to vertexes
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-31 10:41:30 -07:00
Tonis Tiigi
b52c0002db
llbsolver: show status of resolving image config
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:59:11 -07:00
Tonis Tiigi
fe3388e70d
solver: change builder.call to builder.context
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:39:14 -07:00
Tonis Tiigi
57b96a0ee5
cache: add record type field to usage record
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
Tonis Tiigi
9f5e5c7dc7
metaresolver: refactor to option struct and resolvemode
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi
d70d816dee
exporter: update export signature
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 14:37:01 -07:00
Akihiro Suda
8fa31102d5
Merge pull request #539 from tonistiigi/fix-cache-metadata-release
...
Fix cache metadata release
2018-07-24 13:36:22 +09:00
Tonis Tiigi
a7fedd9c9f
control: clean up cache metadata after prune
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 16:27:01 -07:00
Tonis Tiigi
97ea41ee6a
solver: improve multi-key root caching
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 14:27:35 -07:00
Tonis Tiigi
08d324d0dd
exec: fix secret mounts for rootless
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:11 -07:00
Tonis Tiigi
2da2aff164
llbsolver: add secret mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:05 -07:00
Tonis Tiigi
aa3aef3c25
llbsolver: validate llb caps
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
Tonis Tiigi
303b5da713
llbsolver: use result type with multi ref support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
Tonis Tiigi
01b7a29e86
gateway: return result from solve
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Tonis Tiigi
57006ea333
cache: avoid commit on finalize when possible
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-11 16:21:27 -07:00
Tonis Tiigi
ddf3f3e59c
frontend: fix two data races
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-10 19:31:33 -07:00
Akihiro Suda
80d2f820f9
decopule cache/remotecache from registry
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
Akihiro Suda
1b34bd1e31
Merge pull request #464 from tonistiigi/image-platform
...
platform support for image and dockerfile
2018-06-27 14:36:02 +09:00
Tonis Tiigi
e618878451
dockerfile: platform support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 11:06:08 -07:00
Tonis Tiigi
f9383c48a3
source: image source platform support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Tonis Tiigi
b444b9f323
llbsolver: validate runtime platforms for exec op
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:38:03 -07:00
Tonis Tiigi
242697af59
llb: add marshal roundtrip tests for platform
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 17:55:36 -07:00
Tonis Tiigi
ccbf185006
llbsolver: allow sharing modes for cache mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-19 15:57:20 -07:00
Tonis Tiigi
3c973bacfe
llbsolver: allow concurrent cache mount access
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 20:02:30 -07:00
Tonis Tiigi
6f569455d0
dockerfile: enable tmpfs mounts on run —mount
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:42:38 -07:00
Tonis Tiigi
596b03a8a0
llbsolver: allow tmpfs mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:05:14 -07:00
Tonis Tiigi
479419abf2
llbsolver: don’t error on readonly scratch mount
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 16:23:53 -07:00
Akihiro Suda
e2fc1a702d
Merge pull request #414 from tonistiigi/stdio-debug
...
exec: make stdio debugging optional
2018-05-26 06:59:04 +09:00
Tonis Tiigi
6b23ab01ce
exec: make stdio debugging optional
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 11:15:32 -07:00
Tonis Tiigi
97ddeffdb2
exec: remove debug
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 10:45:10 -07:00
Tonis Tiigi
eefc49ff72
dockerfile: detect empty images early
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 16:54:04 -07:00
Tonis Tiigi
8084e8cde9
llbsolver: allow empty returns for opts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Tonis Tiigi
66488dc6d8
exec: support proxy settings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-21 16:08:11 -07:00
Tonis Tiigi
2bd78c3c6e
llbsolver: add support for cached mount type
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:24:20 -07:00
Tonis Tiigi
692bdc8aa5
llbsolver: comments for exec op mount handling
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:09:59 -07:00
Tonis Tiigi
71f5e6dd21
solver: rename solver-next to solver
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00