457482ef67
Merge pull request #858 from tonistiigi/dedupe-fix
...
llbsolver: fix selectors dedupe
2019-03-05 18:31:46 -08:00
6d859fc9b0
llbsolver: fix selectors dedupe
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-05 17:59:13 -08:00
632797cef0
vendor: udpate fsutil to b4281fa
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-05 17:34:01 -08:00
8bcd59a546
contenthash: add nofollow support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-01 23:41:05 -08:00
f472cb193f
exec: fix platform value in cache key
...
Use vertex platform instead of the current system one. This shouldn't break any cache unless qemu was used. Even when qemu was used the issue would not cause any cache collisions because the base images for run were scoped by platform, but better to stop using invalid data for cache key calculation.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-28 11:17:10 -08:00
26010a4864
decouple SessionManager from Worker
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-02-23 22:55:17 +09:00
07810f0aae
llbsolver: validate digest on load
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-04 17:06:34 -08:00
a7df7e2c49
solver: deadlock fix
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-01 17:44:26 -08:00
5be7870549
load parent results on combined cachemanager
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
97eff70c5e
inline remote cache support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
ac4037a1d9
solver: keep original cache creation date when copying between stores
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:34 -08:00
1ec543e384
solver: exclude randomized cache keys from exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:25 -08:00
6e7617e889
support local cache exporter and importer
...
Export:
$ buildctl build ... --export-cache type=local,store=/path/to/output-dir
Import:
$ buildctl build ... --import-cache type=local,store=/path/to/input-dir
Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`
Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
b521aae3ea
dockerfile: allow setting file mode/uid for secrets
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:40 -08:00
2529762ae7
dockerfile: allow setting file mode/uid for ssh socket
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:34 -08:00
2a7287a189
dockerfile2llb/pipe: remove the useless assignment
...
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-11-28 19:44:53 +08:00
f6a896157b
llbsolver: keep session for context calls
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-19 15:24:35 -08:00
14b1afa3d0
solver: fix possible nil dereference
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-08 23:14:04 -07:00
c9b01072d0
solver: specify SSH key ID in error message when required key was not forwarded
...
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-10-05 22:50:07 +00:00
a0c740485c
Merge pull request #663 from tonistiigi/opentracing-fix
...
solver: fix opentracing providers
2018-10-03 11:39:29 -07:00
ec0e352aae
executor: mount cgroups in default spec
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-02 23:22:07 -07:00
b26e666db9
solver: fix opentracing providers
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-02 23:11:17 -07:00
758a267235
s/CapMount/CapExecMount/g
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-03 15:03:36 +09:00
900a1b96c4
define CapMountSSH
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-03 15:00:10 +09:00
f14886ac1d
Merge pull request #650 from tonistiigi/solver-test
...
solver: add TestSlowCacheAvoidLoadOnCache test
2018-10-01 10:57:11 -07:00
3bc8b24fe0
Merge pull request #637 from ijc/llb-image-default-path
...
Push setting of default PATH down into the executor(s)
2018-09-28 08:37:19 -07:00
720d8327e2
Push setting of default PATH down into the executor(s)
...
Setting the default `PATH` in the `llb.State` on the client side means it
depends on the `GOOS` of the buildkit client, rather than of the environment
where it will actually execute.
Instead defer this to execution time and insert the default PATH at that point
if one is not present. Doing this in solver/llbsolver/ops/exec covers all
executors and also avoids breaking the cache.
Client compatibility is handled via a new capability.
Fixes #604
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-09-28 14:16:47 +01:00
3a1f59f85f
solver: add TestSlowCacheAvoidLoadOnCache test
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-27 15:03:56 -07:00
8f4dff0d16
Merge pull request #648 from tonistiigi/solver-early
...
solver: fix early resolution of cache-fast deps
2018-09-27 14:07:21 -07:00
a711fbb8f3
solver: fix early resolution of cache-fast deps
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-26 18:25:33 -07:00
f99352fee1
solver: make sure to return proper canceled errors
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-26 17:38:16 -07:00
cb0d51cca3
Merge pull request #630 from tonistiigi/cleanup
...
golint and misspell cleanup
2018-09-19 12:01:45 -07:00
0940cdc6fe
update golint comments
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-18 22:06:47 -07:00
301da72ce3
misspell fixes
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-18 22:06:38 -07:00
aa73951164
Merge pull request #631 from jhowardmsft/boltdb
...
Revendoring to move boltdb to bbolt
2018-09-19 13:59:08 +09:00
2de2c04c8e
Revendoring to move boltdb to bbolt
...
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-18 11:18:08 -07:00
3938aebdfc
llb: make sure to use stable marshaler
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 15:19:04 -07:00
9c6eb52a84
dockerfile: avoid duplicate config resolve lines
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 13:42:21 -07:00
bf571a519e
update Go to 1.11
...
For consistency with Moby (moby/moby#37358 )
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-09-15 14:25:11 +09:00
1508ae0fc7
Merge pull request #608 from tonistiigi/ssh-forwarding
...
sshforward: implement ssh socket forwarding
2018-09-11 08:01:58 -07:00
a2db26227f
solver: fix data race
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 17:17:17 -07:00
6888956557
sshforward: implement ssh socket forwarding
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 08:57:06 -07:00
e3bc522f7c
Merge pull request #582 from ijc/gateway-client-snapshot-leak
...
clientside-frontend: release non-result refs after completion
2018-08-31 09:22:28 -07:00
9d1b0e6b31
Merge pull request #597 from tonistiigi/graceful-scheduler-errors
...
solver: gracefully handle scheduler bugs
2018-08-30 10:56:39 -07:00
7e971435a4
Merge pull request #590 from tonistiigi/fix-panic
...
solver: fix panic on creating input requests
2018-08-30 10:54:41 -07:00
45aedff363
Merge pull request #571 from ijc/solver-def-and-frontend-mutually-exclusive
...
solver: Don't solve both the provided Defininiton & frontend
2018-08-30 10:10:52 -07:00
59391d1647
clientside-frontend: release non-result refs after completion
...
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522 )
because we weren't releasing the intermediate refs.
Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.
The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.
Fixes #581 .
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
cc83b34572
solver: remove redundant conditional
...
There is now no way to reach this point with the test being false.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:01:27 +01:00
ab7928edff
solver: error if both Definition and Frontend are given.
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 13:59:26 +01:00
0d68543b1e
solver: mark build failed instead of panicking on scheduler error
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-29 11:10:05 -07:00
c77b6d4759
solver: gracefully handle bugs leaving incoming requests open
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-28 17:07:50 -07:00
a384bccbbf
solver: add test for input request deadlock
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-28 16:47:31 -07:00
ee82ffe538
solver: correct dep key conditions
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:18:23 -07:00
1127380f91
solver: allow enabling debugger on runtime
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:16:35 -07:00
23904e6ffa
solver: add test for input request deadlock
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:12:50 -07:00
fdf13cf96e
Allow some Metadata to be return from frontend to the control client
...
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-17 12:45:21 +01:00
af46188e9b
Merge pull request #533 from ijc/client-gateway
...
access gateway API from client
2018-08-17 05:12:23 +09:00
e05be2fd2c
solver: Don't solve both the provided Defininiton & frontend
...
This seems wasteful. I think there is no side-effects of the Load+Build of the
definition which would make a difference to the frontend solve.
Note that this changes the presedence of the actual result, previously the
frontend result would overwrite the definition's one, while now the frontend
would not be run.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-15 16:20:02 +01:00
22f632f895
allow frontends to be run directly on the control client side
...
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.
Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.
The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
95e0348f57
dockerfile: enable global net-mode settings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:40 -07:00
130f5f5ab0
solver: net host with basic entitlements support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:35 -07:00
4945fe758c
llbsolver: add support for extra host records
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-03 13:56:13 -07:00
f7937f1989
solver: avoid dereferencing nil res.Metadata
...
I saw this:
panic: assignment to entry in nil map
goroutine 3173 [running]:
github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export(0xc42094ac40, 0xecac60, 0xc4211ca690, 0xed12c0, 0xc42000fca8, 0x0, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/exporter/oci/export.go:113 +0x156
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2(0xecac60, 0xc4211ca690, 0xc4202207c0, 0x0)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:132 +0x7d
github.com/moby/buildkit/solver/llbsolver.inVertexContext(0xecac60, 0xc4211ca690, 0xe2ec38, 0x1d, 0xc420d43400, 0x0, 0x0)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0x1f6
github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc4203f7dc0, 0xecac60, 0xc420deb830, 0xc4203ed200, 0x19, 0xc4202ff840, 0x0, 0x0, 0x0, 0x0, ...)
/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:131 +0x761
github.com/moby/buildkit/control.(*Controller).Solve(0xc42017e870, 0xecac60, 0xc420deb830, 0xc4201edd40, 0xc42017e870, 0x1, 0x1)
/go/src/github.com/moby/buildkit/control/control.go:207 +0x4b8
github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xecac60, 0xc420deb800, 0xed67a0, 0x1595288)
/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:810 +0x86
github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0xebbea0, 0xc420188310)
/go/src/github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2ba
main.unaryInterceptor.func1(0xecaba0, 0xc4206afe40, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/cmd/buildkitd/main.go:330 +0x15f
github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd76a00, 0xc42017e870, 0xecac60, 0xc420deb530, 0xc42028d030, 0xc420450760, 0x0, 0x0, 0x34, 0x3)
/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:812 +0x167
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0xc420497650, 0x152cef8, 0x0, 0x0, 0x0)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1011 +0x4fc
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0x0)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1249 +0x1318
github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42063e0f0, 0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680)
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:680 +0x9f
created by github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:678 +0xa1
Which was due to `res, err := s.Bridge(j).Solve(ctx, req)` having `res.Metadata
== nil`. There are several paths in `llbBridge.Solve()` where this can be the
case, plus a case where this comes from a frontend which should not be allowed
to crash the daemon.
Likely introduced by d70d816dee6be1257f5d
2018-08-03 16:08:04 +01:00
4c44d2c6e4
dockerfile: custom names to vertexes
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-31 10:41:30 -07:00
b52c0002db
llbsolver: show status of resolving image config
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:59:11 -07:00
fe3388e70d
solver: change builder.call to builder.context
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:39:14 -07:00
489246dd28
cache: support for internal/frontend record type
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
57b96a0ee5
cache: add record type field to usage record
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
9f5e5c7dc7
metaresolver: refactor to option struct and resolvemode
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
a3b5d22419
llb: add resolvemode to image source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:04 -07:00
d70d816dee
exporter: update export signature
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 14:37:01 -07:00
8fa31102d5
Merge pull request #539 from tonistiigi/fix-cache-metadata-release
...
Fix cache metadata release
2018-07-24 13:36:22 +09:00
031c065a06
Merge pull request #536 from tonistiigi/better-errors
...
Improve errors on missing sources and hash calculation
2018-07-24 11:25:58 +09:00
89eee2126e
Merge pull request #535 from tonistiigi/merge-fix
...
solver: fix edge merge skipping issue
2018-07-24 11:25:09 +09:00
73eaeba88c
boltdbcache: fix releasing results
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 18:43:32 -07:00
a7fedd9c9f
control: clean up cache metadata after prune
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 16:27:01 -07:00
97ea41ee6a
solver: improve multi-key root caching
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 14:27:35 -07:00
c575077cd6
solver: fix edge merge skipping issue
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 10:52:31 -07:00
354a967470
solver: join slow-cache error with vertex
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-21 13:03:09 -07:00
08d324d0dd
exec: fix secret mounts for rootless
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:11 -07:00
2da2aff164
llbsolver: add secret mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:05 -07:00
aa3aef3c25
llbsolver: validate llb caps
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
d4c57aec5d
llb: add caps support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
303b5da713
llbsolver: use result type with multi ref support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
01b7a29e86
gateway: return result from solve
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
57006ea333
cache: avoid commit on finalize when possible
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-11 16:21:27 -07:00
ddf3f3e59c
frontend: fix two data races
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-10 19:31:33 -07:00
dca7bdcd4c
Merge pull request #495 from tonistiigi/unique-local
...
llb: set unique ID to local sources without session
2018-07-10 20:34:04 +09:00
946a12eb42
gateway: expose worker information
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:39 -07:00
40e270a2e6
llb: set unique ID to local sources without session
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:27:12 -07:00
80d2f820f9
decopule cache/remotecache from registry
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
1b34bd1e31
Merge pull request #464 from tonistiigi/image-platform
...
platform support for image and dockerfile
2018-06-27 14:36:02 +09:00
e618878451
dockerfile: platform support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 11:06:08 -07:00
f9383c48a3
source: image source platform support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
b444b9f323
llbsolver: validate runtime platforms for exec op
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:38:03 -07:00
242697af59
llb: add marshal roundtrip tests for platform
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 17:55:36 -07:00
25d975ce6e
llb: define platforms types
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 16:49:03 -07:00
ccbf185006
llbsolver: allow sharing modes for cache mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-19 15:57:20 -07:00
3c973bacfe
llbsolver: allow concurrent cache mount access
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 20:02:30 -07:00
6f569455d0
dockerfile: enable tmpfs mounts on run —mount
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:42:38 -07:00
596b03a8a0
llbsolver: allow tmpfs mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:05:14 -07:00
60344aa592
Merge pull request #442 from tonistiigi/runmount
...
dockerfile: add run --mount support
2018-06-09 02:35:34 +09:00
055dcb6c09
local: allow followpaths for local source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 23:23:13 -07:00
479419abf2
llbsolver: don’t error on readonly scratch mount
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 16:23:53 -07:00
1067c24df2
git: support all git transports
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-29 10:56:08 -07:00
e2fc1a702d
Merge pull request #414 from tonistiigi/stdio-debug
...
exec: make stdio debugging optional
2018-05-26 06:59:04 +09:00
6b23ab01ce
exec: make stdio debugging optional
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 11:15:32 -07:00
97ddeffdb2
exec: remove debug
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 10:45:10 -07:00
eefc49ff72
dockerfile: detect empty images early
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 16:54:04 -07:00
8084e8cde9
llbsolver: allow empty returns for opts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
66488dc6d8
exec: support proxy settings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-21 16:08:11 -07:00
2bd78c3c6e
llbsolver: add support for cached mount type
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:24:20 -07:00
692bdc8aa5
llbsolver: comments for exec op mount handling
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:09:59 -07:00
fc888e6413
solver: move cachestorage testsuite
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
71f5e6dd21
solver: rename solver-next to solver
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
335b1c3660
llbsolver: update for new cache variables
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
75bc38c15b
client: add basic cache import test
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
77c2793ebb
remove unused packages
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
b80f45955a
solver: allow exporters to set data to solve results
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-07 20:52:11 -07:00
aabfa3b84a
Merge pull request #324 from ijc/readonly-rootfs-execop
...
Make llb.ReadonlyRootFS usable with common container images
2018-04-04 11:43:44 +09:00
d9b64588cb
Regenerate protobuf go file with gogoproto v0.5
...
This matches the version in vendor.conf, the previous version was `v0.5~25`
according to `git describe`.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 13:45:50 +01:00
2e6adc027d
Reduce code duplication with a closure.
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 11:13:15 +01:00
ddae5a6ea4
Make llb.ReadonlyRootFS usable with common container images
...
e.g. with busybox image:
OCI runtime create failed: container_linux.go:348:
starting container process caused "process_linux.go:402:
container init caused \"rootfs_linux.go:58:
mounting \\\"proc\\\" to rootfs \\\"/.../rootfs\\\" at \\\"/proc\\\"
caused \\\"mkdir /.../rootfs/proc: read-only file system\\\"\"": unknown
This is because we were setting the underlying snapshot readonly so the various
mountpoints (here /proc) cannot be created. This would not be necessary if
those mountpoints were present in images but they typically are not.
The right way to get around this (used e.g. by `ctr`) is to use a writeable
snapshot but to set root readonly in the OCI spec. In this configuration the
rootfs is writeable when mounts are processed but is then made readonly by the
runtime (runc) just before entering the user specified binary within the
container.
This involved a surprising amount of plumbing.
Use this new found ability in the dockerfile converter's `dispatchCopy`
function.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 11:04:07 +01:00
ac6598d255
Check that mounts always have a target.
...
Otherwise the daemon panics when generating the OCI spec.
For belt and braces check in the ExecOp Run function but also when generating the spec.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-22 12:05:12 +00:00
ecf2e8dbc9
llb: update docs
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-02-27 16:06:01 +09:00
7906fb3085
vendor: update containerd to v1.0.2
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
79e346f59a
Add deadcode to the linter
...
it tends to help keeping uneeded code out O:)
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
6e40e83d35
Remove net/context dependencies
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
29b72a3912
solver: add some trace points
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
b94704e55c
session: fix session closing and tracing
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
d804819dad
progress: fix skipping cached vertex progress
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-20 21:33:36 -08:00
dd49b92fd1
solver: add simple locking to catch content duplicated
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-20 16:42:09 -08:00
84c0dd0b07
worker: add labels
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-20 16:28:47 +09:00
dca082582e
worker, solver: s/ResolveVertex/ResolveOp/g
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-19 13:20:30 +09:00
21bebaea80
solver, worker: minor cleanup
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-18 16:20:59 +09:00
efde4f2340
worker, solver: update interfaces
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-18 16:20:59 +09:00
a4316d16c7
solver: split llbop pkg
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-18 16:20:59 +09:00
e58e6d1175
replace FollowSymlink with fs.RootPath
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-17 20:54:49 -08:00
2d3f36d359
source: add sharedkeyhint to local source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:13:15 -08:00
08e1c2990c
dockerfile: add dockerignore support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:03:35 -08:00
ad7c6a905c
source: add function for parsing llb to identifier
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:43:49 -08:00
ead75c0d95
solver.mergeRemoteCache -> cache/instructioncache.Union
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 15:16:46 +09:00
89450211fa
*: deduplicate instructioncache definitions
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 15:03:14 +09:00
29e5bef684
solver: split llbbridge to a separate file (TODO: separate pkg)
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 14:47:09 +09:00
7a5390f355
worker: add support for custom user
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 10:26:30 -08:00
c3aa849014
multi-worker daemon
...
- [X] put multiples workers in a single binary ("-tags containerd standalone")
- [X] add worker selector to LLB vertex metadata
- [X] s/worker/executor/g
- [X] introduce the new "worker" concept https://github.com/moby/buildkit/pull/176#discussion_r153693928
- [X] fix up CLI
- [X] fix up tests
- allow using multiples workers (requires inter-vertex cache copier, HUGE!) --> will be separate PR
Implementation notes:
- "Workers" are renamed to "executors" now
- The new "worker" instance holds an "executor" instance and its
related stuffs such as the snapshotter
- The default worker is "runc-overlay"
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-12 15:17:58 +09:00
438c850a16
dockerfile: add original command metadata to llb
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-07 16:13:50 -08:00
68165c07f4
exporter: allow exporting config with no snapshots
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 23:18:09 -08:00
e0af9d1218
source: add more properties to http source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 12:00:46 -08:00
59628fb20f
add http source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-02 19:52:15 -08:00
6c076f5d6a
cleanup linter warnings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-16 18:09:35 -08:00
b888b38e5a
solver: implement IgnoreCache LLB Metadata
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-07 03:40:07 +00:00
dfb786a292
Merge pull request #143 from AkihiroSuda/refactor-ref
...
cache, solver: refactor Reference interface
2017-11-06 14:39:15 -08:00
565d08914f
cache, solver: refactor Reference interface
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-06 06:30:16 +00:00
65dc07eb28
report imported cache lookup progress
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 20:41:47 +02:00
914ea2110b
cache: include os/arch in cache checksum
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 19:04:53 +02:00
7171f5a27d
cache: fix root snapshot cache lookup
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 19:00:23 +02:00
014a2dd8a1
cache: cache importing from a ref
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:38:08 -07:00
56e1f71e6b
cache: allow pushing exported cache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:37:55 -07:00
1b3f191b37
cache: base code for cache importing
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 17:53:28 -07:00
8e611643f7
cache: export cache to content store
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 15:04:33 -07:00
0fd670fb73
solver: allow access to cached references for exporting
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 11:23:42 -07:00
0ee59e2bdf
Merge pull request #137 from tonistiigi/contentmask
...
solver: replace contentkey with contentmask
2017-10-07 09:47:38 +09:00
2716524dee
exclude selector from content based input hash
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-06 15:43:48 -07:00
53018a8113
solver: replace contentkey with contentmask
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 19:58:05 -07:00
59910481ca
solver: make nested build use solverequest
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 17:51:04 -07:00
23a114a977
solver: simplify solver public api
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 10:05:38 -07:00
80bc5f6097
fix exporter attributes interface returns
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:13 -07:00
2809d01cf6
frontend: add gateway frontend / image frontend support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:03 -07:00
f89ffc2ab4
llb: per-vertex metadata (e.g. IgnoreCache)
...
cache invalidation itself it not implemented yet
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-10-03 05:36:24 +00:00
4a9142223d
solver: fix possible data races
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 22:38:54 -07:00
f41283caed
solver: fix shared request progress and cancellation
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 12:45:47 -07:00
05c39612d8
dockerfile: resolve to digest ref in frontend
...
Less hacky than keeping the cached resolver.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 15:49:37 -07:00
d258bcab7b
dockerfile: clean up converter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 10:49:19 -07:00
cc1b4140af
bgfunc: safer locking
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-19 12:29:29 -07:00
01aaf130e5
solver: refactor to a shared vertex solver model
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-18 13:58:33 -07:00
fd0f7ee2a8
frontend: fix dockerfile frontend resolver
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 18:09:28 -07:00
4c22f6c25a
frontend: share attributes to exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 18:09:28 -07:00
a26eba7cda
fix for read-only mount
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-09-14 06:42:12 +00:00
6d335c3c86
imagesource: share config resolver with frontend
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-05 13:11:27 -07:00
4040029810
dockerfile: improve copy implementation
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-01 19:27:54 -07:00
2d290db465
frontend: base of dockerfile frontend
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-31 23:27:25 -07:00
eb7b5bb76a
contenthash: use selector for content path
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-21 17:07:24 -07:00
8162e15787
llb: add support for mounting subpath or reference
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-07 18:45:50 -07:00
9f6d9a9e78
exporter: add local exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-04 17:12:19 -07:00
8738929b8c
solver: implement content based cache support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-04 10:03:26 -07:00
39b9dec52f
Merge pull request #87 from tonistiigi/snapshot-metadata
...
cache: add more metadata to snapshots
2017-07-27 09:36:10 +09:00
5ecab5e142
Merge pull request #11 from dmcgowan/containerd-worker
...
Containerd worker
2017-07-26 14:36:51 -07:00
38ac00090c
cache: add more metadata values to snapshots
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-26 11:28:20 -07:00
9ce5053a8b
Update containerd version
...
Fix for logrus rename, use fork until fixed in moby.
Removed unused tar stream.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-26 11:07:45 -07:00
93757aef7f
nested llb builds support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-24 18:09:16 -07:00
2e89d2dc72
llb: add custom types for input indexes
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-21 16:39:25 -07:00
8b09e89dbb
solver: avoid removing references required for future cache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-20 17:05:37 -07:00
97b8395272
solver: always cache the original reference instance
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-19 11:08:49 -07:00
ff951eecd9
llb: add readonly mounts support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-18 16:07:37 -07:00
eac79f7c7e
cache: refactor reference reuse and caching
...
Replaces previous mutable.Freeze logic with
commits that can live together with mutable data.
Finalize method is added if the implementation
needs to make sure that the immutable ref is
flushed to the driver. Refs are automaitcally
finalized when writable layers are created on
top of them.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-18 16:01:01 -07:00
5c2e675d18
source: add local file source fundamentals
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-12 23:23:20 -07:00
29d912fddf
exporter: fix exporter progress reporting
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 23:29:21 -07:00
45bfdd5e42
exporter: add image exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 23:29:14 -07:00
2be8499bdc
llb: add scratch support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 22:19:05 -07:00
Tibor Vass
Tonis Tiigi
Akihiro Suda
Wei Fu
Tibor Vass
Ian Campbell
Akihiro Suda
John Howard
Vincent Demeester
Derek McGowan