Commit Graph

508 Commits (master)

Author SHA1 Message Date
Tonis Tiigi cce301badd solver: attach causing vertex in an error
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-21 23:46:17 -07:00
Tonis Tiigi ae3b75d56d errdefs: report component version in stack
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-21 23:46:17 -07:00
Tonis Tiigi e4cc0866f5 gateway: fix error details passing through gateway
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-21 23:46:17 -07:00
Tonis Tiigi 3f77f0495b grpc: add error handling wrappers to client/server
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-21 23:46:11 -07:00
Tonis Tiigi 02fff48cbd errdefs: add support for typed errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-21 22:57:23 -07:00
Tonis Tiigi 37b8832d00 upgrade errors checks to Is()
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-04-18 22:53:38 -07:00
Tonis Tiigi cfabf3009b solver: avoid looping over same keys in loadwithparents
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-03-18 17:07:57 -07:00
Tonis Tiigi 0010996598 solver: avoid recursive loop on cache-export
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-03-02 22:35:59 -08:00
Akihiro Suda 7b579cdb98 vendor: update containerd (and various packages)
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-02-28 10:24:10 +09:00
Tonis Tiigi d1458a6587 update supported platforms without restart
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-26 08:15:42 -08:00
Edgar Lee 7e0f923181 Implement CapFrontendInput to pass llb.States to frontends
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-02-24 14:38:01 -08:00
Tõnis Tiigi 332cdb5e25
Merge pull request #1355 from tonistiigi/mounts-deadlock
ops: fix deadlock on releasing shared mounts
2020-02-21 10:34:57 -08:00
Akihiro Suda 2f5ad30ae5
Merge pull request #1351 from tonistiigi/security-devices
mount whitelist of devices on insecure security mode
2020-02-14 16:52:26 +09:00
Tonis Tiigi b0e76973ee solver: use correct context for getting cache managers lazily
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-06 15:11:49 -08:00
Tonis Tiigi 8cfe2de889 solver: evaluate solve results lazily
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-05 15:29:43 -08:00
Tonis Tiigi b2fffc20d3 ops: add tests for shared and locked cache mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-03 18:48:02 -08:00
Tonis Tiigi 6d907b6893 ops: fix deadlock on releasing shared mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-03 18:47:42 -08:00
Tonis Tiigi bf2dc85f80 ops: refactor cache mounts to have unit tests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-02-03 15:00:22 -08:00
Tonis Tiigi e0e29722e2 file: fix compilation on windows
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-01-30 20:21:40 -08:00
Tonis Tiigi fc186a8b89 oci: mount whitelist of devices on insecure security mode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-01-30 13:07:29 -08:00
Tonis Tiigi 469e7552d0 fileop: keep correct user for parent dir on userns
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-01-28 11:36:42 -08:00
Edgar Lee 2edb431a0f Allow previous solve results to be used in new solves
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-01-23 11:02:40 -08:00
Akihiro Suda aa4311b7fa
Merge pull request #1328 from tonistiigi/combinedcache-panic
solver: avoid panic on combined cache load
2020-01-15 12:30:45 +09:00
Edgar Lee d81832cf6c Detect support for protobuf array ref on server on frontend return call
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-01-14 17:16:21 -08:00
Tonis Tiigi 7fc7f6dbf6 solver: avoid panic on combined cache load
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-01-14 11:19:18 -08:00
Edgar Lee e8326b213b Fixup doc strings for solver types
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2019-11-07 10:00:20 -08:00
Edgar Lee 7846d924ff Improve solver type godocs
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2019-11-06 14:45:26 -08:00
Akihiro Suda e759bd1294
Merge pull request #1231 from tonistiigi/userns-secret-mount
exec: fix mount options on userns remapping
2019-11-01 12:33:12 +09:00
Edgar Lee c122adacb5 Implement file action remove for wildcards
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2019-10-30 14:25:00 -07:00
Tonis Tiigi 307bdc3c3d exec: fix mount options on userns remapping
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-29 14:20:40 -07:00
Tonis Tiigi eede0facb7 solver: fix pipe signaling on incoming updates
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-09-10 16:59:56 -07:00
Tonis Tiigi 1d7e7a6b51 llbsolver: fix error on multiple cache importers
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-29 15:21:17 -07:00
Tonis Tiigi a0dead0809 fix possible double release on mountable
Refactor the interface to avoid such issues in the future.

BuildKit own mounts are stateless and not affected but
a different mountable implementation could get confused.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-16 08:03:16 -07:00
Tonis Tiigi c114e438f5 remotecache: fix inline cache in manifest lists
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-13 16:25:08 -07:00
Tonis Tiigi 872bf0b8c8 dockerfile: add cap for detecting correct nooutput support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-08 22:07:34 -07:00
Tonis Tiigi 360a856527 ops: fix readwrite mounts it no output
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-08 22:07:30 -07:00
Tonis Tiigi 8ad218804e correct int64 alignment
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-05 17:20:23 -07:00
Tonis Tiigi 92c3fd477b worker: remove unreferenced cache mount after release
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-29 15:41:49 -07:00
Tonis Tiigi 7b1bae7a42 solver: support no-cache for exec cache mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-26 14:42:32 -07:00
Tibor Vass fe8289063b
Merge pull request #1049 from tonistiigi/llb-validation
llbsolver: add more llb validation
2019-07-02 10:13:25 -07:00
Tonis Tiigi 5fc18fda78 llbsolver: add more llb validation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-01 23:52:20 -07:00
Tibor Vass b7ea8f13b8
Merge pull request #1036 from tonistiigi/error-tracing
add more error tracing
2019-06-17 10:25:23 -07:00
Tonis Tiigi 17300c1046 file: avoid setting implicit root on idmap
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-10 16:34:43 -07:00
Tonis Tiigi e2dcafa5ca Removing wrapf for review
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-07 12:04:47 -07:00
Tonis Tiigi 0f1c7d0412 session: use errors cause
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-06 21:15:21 -07:00
Tonis Tiigi 61f1bc138b solver: add error tracing to edge connections
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-06 21:15:21 -07:00
msg 78a8e643f6 Add missing locks around broadcasts
Signed-off-by: Mark Gordon <msg555@gmail.com>
2019-05-10 18:27:37 -04:00
Tonis Tiigi 84bbb344fa llbsolver: make remote cache errors not fatal
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-22 16:50:18 -07:00
Tonis Tiigi 4ffda7fc86 llbsolver: use deterministic digest for one-off progress
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-09 18:14:41 -07:00
Akihiro Suda a2dcdf4277
Merge pull request #897 from tonistiigi/userns
userns remapping support base
2019-04-03 13:33:47 +09:00
Tonis Tiigi 15ef4e2b8a dockerfile: allow content cache for rw mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 19:40:30 -07:00
Tonis Tiigi 9b9ff6e4b5 revert userns mapping functions
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 18:26:06 -07:00
Tonis Tiigi 4a12fe526a fileop: userns support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 18:26:06 -07:00
Tonis Tiigi 6921dbe0f6 snapshot: base for userns remapping
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 18:23:46 -07:00
Kunal Kushwaha a2bbb5ff39 security entitlement support
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2019-03-27 13:57:03 +09:00
Kunal Kushwaha 86c7933ac3 proto defination for security entitlements support
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
2019-03-27 12:24:31 +09:00
Tibor Vass dc387eda7f solver: separate linux-only files
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-03-21 03:53:50 +00:00
Akihiro Suda 92f1fb55d5
Merge pull request #888 from tonistiigi/fileop-cap
client: set cap if fileop used
2019-03-20 13:47:45 +09:00
Tonis Tiigi 12a55d6f19 ops: mark tests parallel
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-19 17:07:24 -07:00
Tonis Tiigi 17d78abc12 ops: fix concurrent map writes in test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-19 17:07:15 -07:00
Tonis Tiigi 210679d35f client: set cap if fileop used
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-18 16:35:30 -07:00
Tonis Tiigi c6149da2eb fileop: review fixes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 17:49:58 -07:00
Tonis Tiigi 0d17ac323e fileop: updates with new fsutil copy pkg
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 17:49:58 -07:00
Tonis Tiigi 8a4674bab4 fileop: add dockerfile support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 17:49:50 -07:00
Tonis Tiigi 7210bf6806 fileop: add chown support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 17:49:50 -07:00
Tonis Tiigi 4ffd79735b fileop: connect with contenthash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:31:00 -07:00
Tonis Tiigi 81a5fa5a2e llbsolver: fileop implementation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi 2be999ba52 fileop: llbsolver implementation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi b2b0e3dbfa fileop: add release support and tests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi 9fb1f09a1e llbsolver: fileop base
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi a443cfff05 fileop: resolve review comments
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi 89e6614b38 solver: change uid to uint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi 431d11dda3 llb: add timestamp override to fileop
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
Tonis Tiigi 5b4841f308 llb: initial fileop implementation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:01 -07:00
Tibor Vass 0515c76243
Merge pull request #868 from tonistiigi/solver-cache-fix
solver: change early input resolve strategy
2019-03-13 19:09:05 -07:00
Tonis Tiigi fbb71123a7 solver: optimize early input resolve
Detect the cases where open input keys are not possible even
when computed keys are calculated to skip to cache lookup early.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-13 15:34:18 -07:00
Dave Chen 550fd00160 Don't miss the Variant for arm2 platform
The `defaultPlatform` got on the build system include the variant for arm v6, v7 etc.
This is an important information for us to find the matched platform, but this information
is missed since the code doesn't include variant when assemble the platform information.

And since "V8" variant for arm64 is normalized to an empty string, so we need to take care
of this as well.

Signed-off-by: Dave Chen <dave.chen@arm.com>
2019-03-12 22:32:45 -07:00
Tonis Tiigi b988403262 solver: fix early input resolve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-12 17:49:54 -07:00
Tibor Vass 457482ef67
Merge pull request #858 from tonistiigi/dedupe-fix
llbsolver: fix selectors dedupe
2019-03-05 18:31:46 -08:00
Tonis Tiigi 6d859fc9b0 llbsolver: fix selectors dedupe
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-05 17:59:13 -08:00
Tonis Tiigi 632797cef0 vendor: udpate fsutil to b4281fa
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-05 17:34:01 -08:00
Tonis Tiigi 8bcd59a546 contenthash: add nofollow support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-01 23:41:05 -08:00
Tonis Tiigi f472cb193f exec: fix platform value in cache key
Use vertex platform instead of the current system one. This shouldn't break any cache unless qemu was used. Even when qemu was used the issue would not cause any cache collisions because the base images for run were scoped by platform, but better to stop using invalid data for cache key calculation.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-28 11:17:10 -08:00
Akihiro Suda 26010a4864 decouple SessionManager from Worker
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-02-23 22:55:17 +09:00
Tonis Tiigi 07810f0aae llbsolver: validate digest on load
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-04 17:06:34 -08:00
Tonis Tiigi a7df7e2c49 solver: deadlock fix
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-01 17:44:26 -08:00
Tonis Tiigi 5be7870549 load parent results on combined cachemanager
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
Tonis Tiigi 97eff70c5e inline remote cache support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
Tonis Tiigi ac4037a1d9 solver: keep original cache creation date when copying between stores
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:34 -08:00
Tonis Tiigi 1ec543e384 solver: exclude randomized cache keys from exporter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:25 -08:00
Akihiro Suda 6e7617e889 support local cache exporter and importer
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
Tonis Tiigi b521aae3ea dockerfile: allow setting file mode/uid for secrets
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:40 -08:00
Tonis Tiigi 2529762ae7 dockerfile: allow setting file mode/uid for ssh socket
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-03 11:13:34 -08:00
Wei Fu 2a7287a189 dockerfile2llb/pipe: remove the useless assignment
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2018-11-28 19:44:53 +08:00
Tonis Tiigi f6a896157b llbsolver: keep session for context calls
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-11-19 15:24:35 -08:00
Tonis Tiigi 14b1afa3d0 solver: fix possible nil dereference
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-08 23:14:04 -07:00
Tibor Vass c9b01072d0 solver: specify SSH key ID in error message when required key was not forwarded
Signed-off-by: Tibor Vass <tibor@docker.com>
2018-10-05 22:50:07 +00:00
Tibor Vass a0c740485c
Merge pull request #663 from tonistiigi/opentracing-fix
solver: fix opentracing providers
2018-10-03 11:39:29 -07:00
Tonis Tiigi ec0e352aae executor: mount cgroups in default spec
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-02 23:22:07 -07:00
Tonis Tiigi b26e666db9 solver: fix opentracing providers
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-02 23:11:17 -07:00
Akihiro Suda 758a267235 s/CapMount/CapExecMount/g
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-03 15:03:36 +09:00
Akihiro Suda 900a1b96c4 define CapMountSSH
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-10-03 15:00:10 +09:00
Tõnis Tiigi f14886ac1d
Merge pull request #650 from tonistiigi/solver-test
solver: add TestSlowCacheAvoidLoadOnCache test
2018-10-01 10:57:11 -07:00
Tõnis Tiigi 3bc8b24fe0
Merge pull request #637 from ijc/llb-image-default-path
Push setting of default PATH down into the executor(s)
2018-09-28 08:37:19 -07:00
Ian Campbell 720d8327e2 Push setting of default PATH down into the executor(s)
Setting the default `PATH` in the `llb.State` on the client side means it
depends on the `GOOS` of the buildkit client, rather than of the environment
where it will actually execute.

Instead defer this to execution time and insert the default PATH at that point
if one is not present. Doing this in solver/llbsolver/ops/exec covers all
executors and also avoids breaking the cache.

Client compatibility is handled via a new capability.

Fixes #604

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-09-28 14:16:47 +01:00
Tonis Tiigi 3a1f59f85f solver: add TestSlowCacheAvoidLoadOnCache test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-27 15:03:56 -07:00
Tibor Vass 8f4dff0d16
Merge pull request #648 from tonistiigi/solver-early
solver: fix early resolution of cache-fast deps
2018-09-27 14:07:21 -07:00
Tonis Tiigi a711fbb8f3 solver: fix early resolution of cache-fast deps
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-26 18:25:33 -07:00
Tonis Tiigi f99352fee1 solver: make sure to return proper canceled errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-26 17:38:16 -07:00
Tibor Vass cb0d51cca3
Merge pull request #630 from tonistiigi/cleanup
golint and misspell cleanup
2018-09-19 12:01:45 -07:00
Tonis Tiigi 0940cdc6fe update golint comments
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-18 22:06:47 -07:00
Tonis Tiigi 301da72ce3 misspell fixes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-18 22:06:38 -07:00
Akihiro Suda aa73951164
Merge pull request #631 from jhowardmsft/boltdb
Revendoring to move boltdb to bbolt
2018-09-19 13:59:08 +09:00
John Howard 2de2c04c8e Revendoring to move boltdb to bbolt
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-18 11:18:08 -07:00
Tonis Tiigi 3938aebdfc llb: make sure to use stable marshaler
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 15:19:04 -07:00
Tonis Tiigi 9c6eb52a84 dockerfile: avoid duplicate config resolve lines
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-17 13:42:21 -07:00
Akihiro Suda bf571a519e update Go to 1.11
For consistency with Moby (moby/moby#37358)

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-09-15 14:25:11 +09:00
Tõnis Tiigi 1508ae0fc7
Merge pull request #608 from tonistiigi/ssh-forwarding
sshforward: implement ssh socket forwarding
2018-09-11 08:01:58 -07:00
Tonis Tiigi a2db26227f solver: fix data race
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 17:17:17 -07:00
Tonis Tiigi 6888956557 sshforward: implement ssh socket forwarding
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 08:57:06 -07:00
Tõnis Tiigi e3bc522f7c
Merge pull request #582 from ijc/gateway-client-snapshot-leak
clientside-frontend: release non-result refs after completion
2018-08-31 09:22:28 -07:00
Tibor Vass 9d1b0e6b31
Merge pull request #597 from tonistiigi/graceful-scheduler-errors
solver: gracefully handle scheduler bugs
2018-08-30 10:56:39 -07:00
Tibor Vass 7e971435a4
Merge pull request #590 from tonistiigi/fix-panic
solver: fix panic on creating input requests
2018-08-30 10:54:41 -07:00
Tõnis Tiigi 45aedff363
Merge pull request #571 from ijc/solver-def-and-frontend-mutually-exclusive
solver: Don't solve both the provided Defininiton & frontend
2018-08-30 10:10:52 -07:00
Ian Campbell 59391d1647 clientside-frontend: release non-result refs after completion
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522)
because we weren't releasing the intermediate refs.

Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.

The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.

Fixes #581.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:54:50 +01:00
Ian Campbell cc83b34572 solver: remove redundant conditional
There is now no way to reach this point with the test being false.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 14:01:27 +01:00
Ian Campbell ab7928edff solver: error if both Definition and Frontend are given.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-30 13:59:26 +01:00
Tonis Tiigi 0d68543b1e solver: mark build failed instead of panicking on scheduler error
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-29 11:10:05 -07:00
Tonis Tiigi c77b6d4759 solver: gracefully handle bugs leaving incoming requests open
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-28 17:07:50 -07:00
Tonis Tiigi a384bccbbf solver: add test for input request deadlock
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-28 16:47:31 -07:00
Tonis Tiigi ee82ffe538 solver: correct dep key conditions
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:18:23 -07:00
Tonis Tiigi 1127380f91 solver: allow enabling debugger on runtime
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:16:35 -07:00
Tonis Tiigi 23904e6ffa solver: add test for input request deadlock
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-27 16:12:50 -07:00
Ian Campbell fdf13cf96e Allow some Metadata to be return from frontend to the control client
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-17 12:45:21 +01:00
Akihiro Suda af46188e9b
Merge pull request #533 from ijc/client-gateway
access gateway API from client
2018-08-17 05:12:23 +09:00
Ian Campbell e05be2fd2c solver: Don't solve both the provided Defininiton & frontend
This seems wasteful. I think there is no side-effects of the Load+Build of the
definition which would make a difference to the frontend solve.

Note that this changes the presedence of the actual result, previously the
frontend result would overwrite the definition's one, while now the frontend
would not be run.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-15 16:20:02 +01:00
Ian Campbell 22f632f895 allow frontends to be run directly on the control client side
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.

Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.

The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-14 11:50:08 +01:00
Tonis Tiigi 95e0348f57 dockerfile: enable global net-mode settings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:40 -07:00
Tonis Tiigi 130f5f5ab0 solver: net host with basic entitlements support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-09 14:03:35 -07:00
Tonis Tiigi 4945fe758c llbsolver: add support for extra host records
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-03 13:56:13 -07:00
Ian Campbell f7937f1989 solver: avoid dereferencing nil res.Metadata
I saw this:
    panic: assignment to entry in nil map

    goroutine 3173 [running]:
    github.com/moby/buildkit/exporter/oci.(*imageExporterInstance).Export(0xc42094ac40, 0xecac60, 0xc4211ca690, 0xed12c0, 0xc42000fca8, 0x0, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/exporter/oci/export.go:113 +0x156
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve.func2(0xecac60, 0xc4211ca690, 0xc4202207c0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:132 +0x7d
    github.com/moby/buildkit/solver/llbsolver.inVertexContext(0xecac60, 0xc4211ca690, 0xe2ec38, 0x1d, 0xc420d43400, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:203 +0x1f6
    github.com/moby/buildkit/solver/llbsolver.(*Solver).Solve(0xc4203f7dc0, 0xecac60, 0xc420deb830, 0xc4203ed200, 0x19, 0xc4202ff840, 0x0, 0x0, 0x0, 0x0, ...)
    	/go/src/github.com/moby/buildkit/solver/llbsolver/solver.go:131 +0x761
    github.com/moby/buildkit/control.(*Controller).Solve(0xc42017e870, 0xecac60, 0xc420deb830, 0xc4201edd40, 0xc42017e870, 0x1, 0x1)
    	/go/src/github.com/moby/buildkit/control/control.go:207 +0x4b8
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xecac60, 0xc420deb800, 0xed67a0, 0x1595288)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:810 +0x86
    github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0xecac60, 0xc420deb800, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0xebbea0, 0xc420188310)
    	/go/src/github.com/moby/buildkit/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:57 +0x2ba
    main.unaryInterceptor.func1(0xecaba0, 0xc4206afe40, 0xde2180, 0xc4201edd40, 0xc4202ff9e0, 0xc4202ffa00, 0x0, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/cmd/buildkitd/main.go:330 +0x15f
    github.com/moby/buildkit/api/services/control._Control_Solve_Handler(0xd76a00, 0xc42017e870, 0xecac60, 0xc420deb530, 0xc42028d030, 0xc420450760, 0x0, 0x0, 0x34, 0x3)
    	/go/src/github.com/moby/buildkit/api/services/control/control.pb.go:812 +0x167
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0xc420497650, 0x152cef8, 0x0, 0x0, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1011 +0x4fc
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).handleStream(0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680, 0x0)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:1249 +0x1318
    github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc42063e0f0, 0xc420244700, 0xed39c0, 0xc420694000, 0xc420671680)
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:680 +0x9f
    created by github.com/moby/buildkit/vendor/google.golang.org/grpc.(*Server).serveStreams.func1
    	/go/src/github.com/moby/buildkit/vendor/google.golang.org/grpc/server.go:678 +0xa1

Which was due to `res, err := s.Bridge(j).Solve(ctx, req)` having `res.Metadata
== nil`. There are several paths in `llbBridge.Solve()` where this can be the
case, plus a case where this comes from a frontend which should not be allowed
to crash the daemon.

Likely introduced by d70d816dee or 6be1257f5d.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-08-03 16:08:04 +01:00
Tonis Tiigi 4c44d2c6e4 dockerfile: custom names to vertexes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-31 10:41:30 -07:00
Tonis Tiigi b52c0002db llbsolver: show status of resolving image config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:59:11 -07:00
Tonis Tiigi fe3388e70d solver: change builder.call to builder.context
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 15:39:14 -07:00
Tonis Tiigi 489246dd28 cache: support for internal/frontend record type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
Tonis Tiigi 57b96a0ee5 cache: add record type field to usage record
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi a3b5d22419 llb: add resolvemode to image source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:04 -07:00
Tonis Tiigi d70d816dee exporter: update export signature
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 14:37:01 -07:00
Akihiro Suda 8fa31102d5
Merge pull request #539 from tonistiigi/fix-cache-metadata-release
Fix cache metadata release
2018-07-24 13:36:22 +09:00
Akihiro Suda 031c065a06
Merge pull request #536 from tonistiigi/better-errors
Improve errors on missing sources and hash calculation
2018-07-24 11:25:58 +09:00
Akihiro Suda 89eee2126e
Merge pull request #535 from tonistiigi/merge-fix
solver: fix edge merge skipping issue
2018-07-24 11:25:09 +09:00
Tonis Tiigi 73eaeba88c boltdbcache: fix releasing results
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 18:43:32 -07:00
Tonis Tiigi a7fedd9c9f control: clean up cache metadata after prune
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 16:27:01 -07:00
Tonis Tiigi 97ea41ee6a solver: improve multi-key root caching
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 14:27:35 -07:00
Tonis Tiigi c575077cd6 solver: fix edge merge skipping issue
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-23 10:52:31 -07:00
Tonis Tiigi 354a967470 solver: join slow-cache error with vertex
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-21 13:03:09 -07:00
Tonis Tiigi 08d324d0dd exec: fix secret mounts for rootless
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:11 -07:00
Tonis Tiigi 2da2aff164 llbsolver: add secret mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-19 22:02:05 -07:00
Tonis Tiigi aa3aef3c25 llbsolver: validate llb caps
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
Tonis Tiigi d4c57aec5d llb: add caps support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-18 17:22:30 -07:00
Tonis Tiigi 303b5da713 llbsolver: use result type with multi ref support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:52:23 -07:00
Tonis Tiigi 01b7a29e86 gateway: return result from solve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-12 17:51:44 -07:00
Tonis Tiigi 57006ea333 cache: avoid commit on finalize when possible
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-11 16:21:27 -07:00
Tonis Tiigi ddf3f3e59c frontend: fix two data races
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-10 19:31:33 -07:00
Akihiro Suda dca7bdcd4c
Merge pull request #495 from tonistiigi/unique-local
llb: set unique ID to local sources without session
2018-07-10 20:34:04 +09:00
Tonis Tiigi 946a12eb42 gateway: expose worker information
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:32:39 -07:00
Tonis Tiigi 40e270a2e6 llb: set unique ID to local sources without session
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-09 16:27:12 -07:00
Akihiro Suda 80d2f820f9 decopule cache/remotecache from registry
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
Akihiro Suda 1b34bd1e31
Merge pull request #464 from tonistiigi/image-platform
platform support for image and dockerfile
2018-06-27 14:36:02 +09:00
Tonis Tiigi e618878451 dockerfile: platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 11:06:08 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Tonis Tiigi b444b9f323 llbsolver: validate runtime platforms for exec op
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:38:03 -07:00
Tonis Tiigi 242697af59 llb: add marshal roundtrip tests for platform
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 17:55:36 -07:00
Tonis Tiigi 25d975ce6e llb: define platforms types
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-22 16:49:03 -07:00
Tonis Tiigi ccbf185006 llbsolver: allow sharing modes for cache mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-19 15:57:20 -07:00
Tonis Tiigi 3c973bacfe llbsolver: allow concurrent cache mount access
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 20:02:30 -07:00
Tonis Tiigi 6f569455d0 dockerfile: enable tmpfs mounts on run —mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:42:38 -07:00
Tonis Tiigi 596b03a8a0 llbsolver: allow tmpfs mounts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:05:14 -07:00
Akihiro Suda 60344aa592
Merge pull request #442 from tonistiigi/runmount
dockerfile: add run --mount support
2018-06-09 02:35:34 +09:00
Tonis Tiigi 055dcb6c09 local: allow followpaths for local source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 23:23:13 -07:00
Tonis Tiigi 479419abf2 llbsolver: don’t error on readonly scratch mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 16:23:53 -07:00
Tonis Tiigi 1067c24df2 git: support all git transports
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-29 10:56:08 -07:00
Akihiro Suda e2fc1a702d
Merge pull request #414 from tonistiigi/stdio-debug
exec: make stdio debugging optional
2018-05-26 06:59:04 +09:00
Tonis Tiigi 6b23ab01ce exec: make stdio debugging optional
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 11:15:32 -07:00
Tonis Tiigi 97ddeffdb2 exec: remove debug
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 10:45:10 -07:00
Tonis Tiigi eefc49ff72 dockerfile: detect empty images early
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 16:54:04 -07:00
Tonis Tiigi 8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Tonis Tiigi 66488dc6d8 exec: support proxy settings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-21 16:08:11 -07:00
Tonis Tiigi 2bd78c3c6e llbsolver: add support for cached mount type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:24:20 -07:00
Tonis Tiigi 692bdc8aa5 llbsolver: comments for exec op mount handling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-15 12:09:59 -07:00
Tonis Tiigi fc888e6413 solver: move cachestorage testsuite
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 71f5e6dd21 solver: rename solver-next to solver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 335b1c3660 llbsolver: update for new cache variables
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 75bc38c15b client: add basic cache import test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 77c2793ebb remove unused packages
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi b80f45955a solver: allow exporters to set data to solve results
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-07 20:52:11 -07:00
Akihiro Suda aabfa3b84a
Merge pull request #324 from ijc/readonly-rootfs-execop
Make llb.ReadonlyRootFS usable with common container images
2018-04-04 11:43:44 +09:00
Ian Campbell d9b64588cb Regenerate protobuf go file with gogoproto v0.5
This matches the version in vendor.conf, the previous version was `v0.5~25`
according to `git describe`.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 13:45:50 +01:00
Ian Campbell 2e6adc027d Reduce code duplication with a closure.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 11:13:15 +01:00
Ian Campbell ddae5a6ea4 Make llb.ReadonlyRootFS usable with common container images
e.g. with busybox image:

    OCI runtime create failed: container_linux.go:348:
    starting container process caused "process_linux.go:402:
    container init caused \"rootfs_linux.go:58:
    mounting \\\"proc\\\" to rootfs \\\"/.../rootfs\\\" at \\\"/proc\\\"
    caused \\\"mkdir /.../rootfs/proc: read-only file system\\\"\"": unknown

This is because we were setting the underlying snapshot readonly so the various
mountpoints (here /proc) cannot be created. This would not be necessary if
those mountpoints were present in images but they typically are not.

The right way to get around this (used e.g. by `ctr`) is to use a writeable
snapshot but to set root readonly in the OCI spec. In this configuration the
rootfs is writeable when mounts are processed but is then made readonly by the
runtime (runc) just before entering the user specified binary within the
container.

This involved a surprising amount of plumbing.

Use this new found ability in the dockerfile converter's `dispatchCopy`
function.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-04-03 11:04:07 +01:00
Ian Campbell ac6598d255 Check that mounts always have a target.
Otherwise the daemon panics when generating the OCI spec.

For belt and braces check in the ExecOp Run function but also when generating the spec.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-22 12:05:12 +00:00
Akihiro Suda ecf2e8dbc9 llb: update docs
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-02-27 16:06:01 +09:00
Tonis Tiigi 7906fb3085 vendor: update containerd to v1.0.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
Vincent Demeester 79e346f59a
Add deadcode to the linter
it tends to help keeping uneeded code out O:)

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
Tonis Tiigi 6e40e83d35 Remove net/context dependencies
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
Tonis Tiigi 29b72a3912 solver: add some trace points
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
Tonis Tiigi b94704e55c session: fix session closing and tracing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
Tonis Tiigi d804819dad progress: fix skipping cached vertex progress
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-20 21:33:36 -08:00
Tonis Tiigi dd49b92fd1 solver: add simple locking to catch content duplicated
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-20 16:42:09 -08:00
Akihiro Suda 84c0dd0b07 worker: add labels
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-20 16:28:47 +09:00
Akihiro Suda dca082582e worker, solver: s/ResolveVertex/ResolveOp/g
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-19 13:20:30 +09:00
Akihiro Suda 21bebaea80 solver, worker: minor cleanup
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-18 16:20:59 +09:00
Tonis Tiigi efde4f2340 worker, solver: update interfaces
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-18 16:20:59 +09:00
Akihiro Suda a4316d16c7 solver: split llbop pkg
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-18 16:20:59 +09:00
Tonis Tiigi e58e6d1175 replace FollowSymlink with fs.RootPath
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-17 20:54:49 -08:00
Tonis Tiigi 2d3f36d359 source: add sharedkeyhint to local source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:13:15 -08:00
Tonis Tiigi 08e1c2990c dockerfile: add dockerignore support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:03:35 -08:00
Tonis Tiigi ad7c6a905c source: add function for parsing llb to identifier
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:43:49 -08:00
Akihiro Suda ead75c0d95 solver.mergeRemoteCache -> cache/instructioncache.Union
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 15:16:46 +09:00
Akihiro Suda 89450211fa *: deduplicate instructioncache definitions
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 15:03:14 +09:00
Akihiro Suda 29e5bef684 solver: split llbbridge to a separate file (TODO: separate pkg)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-13 14:47:09 +09:00
Tonis Tiigi 7a5390f355 worker: add support for custom user
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 10:26:30 -08:00
Akihiro Suda c3aa849014 multi-worker daemon
- [X] put multiples workers in a single binary ("-tags containerd standalone")
- [X] add worker selector to LLB vertex metadata
- [X] s/worker/executor/g
- [X] introduce the new "worker" concept https://github.com/moby/buildkit/pull/176#discussion_r153693928
- [X] fix up CLI
- [X] fix up tests
- allow using multiples workers (requires inter-vertex cache copier, HUGE!) --> will be separate PR

Implementation notes:
- "Workers" are renamed to "executors" now
- The new "worker" instance holds an "executor" instance and its
related stuffs such as the snapshotter
- The default worker is "runc-overlay"

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-12-12 15:17:58 +09:00
Tonis Tiigi 438c850a16 dockerfile: add original command metadata to llb
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-07 16:13:50 -08:00
Tonis Tiigi 68165c07f4 exporter: allow exporting config with no snapshots
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 23:18:09 -08:00
Tonis Tiigi e0af9d1218 source: add more properties to http source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 12:00:46 -08:00
Tonis Tiigi 59628fb20f add http source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-02 19:52:15 -08:00
Tonis Tiigi 6c076f5d6a cleanup linter warnings
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-16 18:09:35 -08:00
Akihiro Suda b888b38e5a solver: implement IgnoreCache LLB Metadata
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-07 03:40:07 +00:00
Tõnis Tiigi dfb786a292
Merge pull request #143 from AkihiroSuda/refactor-ref
cache, solver: refactor Reference interface
2017-11-06 14:39:15 -08:00
Akihiro Suda 565d08914f cache, solver: refactor Reference interface
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-06 06:30:16 +00:00
Tonis Tiigi 65dc07eb28 report imported cache lookup progress
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 20:41:47 +02:00
Tonis Tiigi 914ea2110b cache: include os/arch in cache checksum
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 19:04:53 +02:00
Tonis Tiigi 7171f5a27d cache: fix root snapshot cache lookup
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-15 19:00:23 +02:00
Tonis Tiigi 014a2dd8a1 cache: cache importing from a ref
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:38:08 -07:00
Tonis Tiigi 56e1f71e6b cache: allow pushing exported cache
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:37:55 -07:00
Tonis Tiigi 1b3f191b37 cache: base code for cache importing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 17:53:28 -07:00
Tonis Tiigi 8e611643f7 cache: export cache to content store
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 15:04:33 -07:00
Tonis Tiigi 0fd670fb73 solver: allow access to cached references for exporting
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-12 11:23:42 -07:00
Akihiro Suda 0ee59e2bdf Merge pull request #137 from tonistiigi/contentmask
solver: replace contentkey with contentmask
2017-10-07 09:47:38 +09:00
Tonis Tiigi 2716524dee exclude selector from content based input hash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-06 15:43:48 -07:00
Tonis Tiigi 53018a8113 solver: replace contentkey with contentmask
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 19:58:05 -07:00
Tonis Tiigi 59910481ca solver: make nested build use solverequest
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 17:51:04 -07:00
Tonis Tiigi 23a114a977 solver: simplify solver public api
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-05 10:05:38 -07:00
Tonis Tiigi 80bc5f6097 fix exporter attributes interface returns
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:13 -07:00
Tonis Tiigi 2809d01cf6 frontend: add gateway frontend / image frontend support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-03 21:14:03 -07:00
Akihiro Suda f89ffc2ab4 llb: per-vertex metadata (e.g. IgnoreCache)
cache invalidation itself it not implemented yet

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-10-03 05:36:24 +00:00
Tonis Tiigi 4a9142223d solver: fix possible data races
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 22:38:54 -07:00
Tonis Tiigi f41283caed solver: fix shared request progress and cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 12:45:47 -07:00