Commit Graph

46 Commits (21034f4234c5fdbdd7f25d3373e12bd0b950e4db)

Author SHA1 Message Date
Tonis Tiigi 649f68aae2 gateway: handle grpc errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-11 22:42:19 -07:00
Tonis Tiigi 03a2f79561 gateway: allow frontends without network
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:38:07 -08:00
Akihiro Suda 6e7617e889 support local cache exporter and importer
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Ian Campbell b4454a169f implement `gateway.Reference.StatFile` method
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:45 +01:00
Ian Campbell df76f6d380 Move cache/fsutil into a separate cache/util subpackage
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
Ian Campbell b30ee91a53 implement `gateway.Reference.ReadDir` method
The alternative is a helper container running e.g. find, capturing the output
to a scratch to be read back (with ref.ReadFile) and parsed.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
Ian Campbell 53921cf110 llbBridfgeForwarder: Drop refs as they are released in `Discard`.
This makes it safe(r) for `Discard` to be called more than once.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 15:23:49 +01:00
Ian Campbell b558d75d3a Consistently protect `llbBridgeForwarder.refs` with mutex
Most accesses to this map are protected. Add the missing locks (some of which
are in code paths where we wouldn't normally expect concurrent accesses, but
better to be safe).

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 15:03:29 +01:00
Ian Campbell 59391d1647 clientside-frontend: release non-result refs after completion
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522)
because we weren't releasing the intermediate refs.

Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.

The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.

Fixes #581.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
Ian Campbell 22f632f895 allow frontends to be run directly on the control client side
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.

Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.

The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
Ian Campbell 3f5276b3c4 gateway: make `Return` a one shot operation and check for various edge cases.
Refactor the setting/getting of the result for a `llbBridgeForwarder` in order
to check that `Return` is only called once and only with the correct options.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-09 11:19:42 +01:00
Tonis Tiigi 4c44d2c6e4 dockerfile: custom names to vertexes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-31 10:41:30 -07:00
Tonis Tiigi 489246dd28 cache: support for internal/frontend record type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi 8c065401f2 exporter: reuse exptypes constants
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 16:01:12 -07:00
Tonis Tiigi d4c57aec5d llb: add caps support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
Tonis Tiigi 303b5da713 llbsolver: use result type with multi ref support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
Tonis Tiigi c3a29cce8b gateway: implementation for return
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Tonis Tiigi 01b7a29e86 gateway: return result from solve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Tonis Tiigi d9f1a1d99f gateway: expose buildopts and result
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:50:19 -07:00
Tonis Tiigi 30601fb7e5 gateway: fix product env passing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 16:04:58 -07:00
Tonis Tiigi d5c9d39128 apicaps: set exportedproduct from binary
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:39 -07:00
Tonis Tiigi c8fa75ffc1 dockerfile: read default build platform from workers
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:39 -07:00
Tonis Tiigi 946a12eb42 gateway: expose worker information
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:39 -07:00
Tonis Tiigi 90d164ced5 gateway: inital caps detection support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:38 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Akihiro Suda b36d2dc272
Merge pull request #459 from tonistiigi/optional-digest
gateway: allow digest to be optional
2018-06-21 08:42:42 +09:00
Tonis Tiigi bdb77f073a gateway: protect gateway forwarder map access
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-20 11:17:05 -07:00
Tonis Tiigi e5449521f5 gateway: allow digest to be optional
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-20 11:09:32 -07:00
Tonis Tiigi 6b25a56906 dockerfile: support building from a Dockerfile URL
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-30 13:20:27 -07:00
Tonis Tiigi 71f5e6dd21 solver: rename solver-next to solver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 644540e0f1 gateway: update gateway to cache import refs
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi b6c782d53e control: switch to use solver-next
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi e24e194078 remove net/context references
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-04-30 18:10:54 -07:00
Ian Campbell ddae5a6ea4 Make llb.ReadonlyRootFS usable with common container images
e.g. with busybox image:

    OCI runtime create failed: container_linux.go:348:
    starting container process caused "process_linux.go:402:
    container init caused \"rootfs_linux.go:58:
    mounting \\\"proc\\\" to rootfs \\\"/.../rootfs\\\" at \\\"/proc\\\"
    caused \\\"mkdir /.../rootfs/proc: read-only file system\\\"\"": unknown

This is because we were setting the underlying snapshot readonly so the various
mountpoints (here /proc) cannot be created. This would not be necessary if
those mountpoints were present in images but they typically are not.

The right way to get around this (used e.g. by `ctr`) is to use a writeable
snapshot but to set root readonly in the OCI spec. In this configuration the
rootfs is writeable when mounts are processed but is then made readonly by the
runtime (runc) just before entering the user specified binary within the
container.

This involved a surprising amount of plumbing.

Use this new found ability in the dockerfile converter's `dispatchCopy`
function.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 11:04:07 +01:00
Tonis Tiigi 12fd7edc24 gateway: typo fix
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-03-08 10:13:26 -08:00
Tonis Tiigi 6e40e83d35 Remove net/context dependencies
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
Tonis Tiigi 29b72a3912 solver: add some trace points
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
Tonis Tiigi 3407de2e04 gateway: allow skipping tag in source parameter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-20 22:51:33 -08:00
Tonis Tiigi 46e816965b gateway: fix returning nil references
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 17:29:03 -08:00
Tonis Tiigi fc5d833cc1 gateway: fix unreleased references
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:51:31 -08:00
Tonis Tiigi 98e91f8b70 frontend: make dockerfile.v0 and external frontend use same code
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:51:25 -08:00
Akihiro Suda c3aa849014 multi-worker daemon
- [X] put multiples workers in a single binary ("-tags containerd standalone")
- [X] add worker selector to LLB vertex metadata
- [X] s/worker/executor/g
- [X] introduce the new "worker" concept https://github.com/moby/buildkit/pull/176#discussion_r153693928
- [X] fix up CLI
- [X] fix up tests
- allow using multiples workers (requires inter-vertex cache copier, HUGE!) --> will be separate PR

Implementation notes:
- "Workers" are renamed to "executors" now
- The new "worker" instance holds an "executor" instance and its
related stuffs such as the snapshotter
- The default worker is "runc-overlay"

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-12 15:17:58 +09:00
Tonis Tiigi 23a114a977 solver: simplify solver public api
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 10:05:38 -07:00
Tonis Tiigi 80bc5f6097 fix exporter attributes interface returns
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:13 -07:00
Tonis Tiigi 2809d01cf6 frontend: add gateway frontend / image frontend support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:03 -07:00