JohnHammond
/
buildkit
mirror of https://github.com/JohnHammond/buildkit.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,647 Commits
7 Branches
72 Tags
26 MiB
Commit Graph

482 Commits (e99111112725e9d74232c044bedff0ea98ca229d)

Author SHA1 Message Date
Erik Sipsma 55cbd19dec Add support for lazily-pulled blobs in cache manager. 
This allows the layers of images to only be pulled if/once they are actually
required.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
 2020-08-05 17:18:43 -07:00
Shingo Omura 804aa5d21a
setting host part of address as :authority pseudo header. 
grpc-go uses a slightly different naming scheme(https://github.com/grpc/grpc/blob/master/doc/naming.md)
This will end up setting rfc non-complient :authority header to address string (e.g. tcp://127.0.0.1:1234).
So, this commit changes to sets right authority header via WithAuthority DialOption.

Signed-off-by: Shingo Omura <everpeace@gmail.com>
 2020-07-31 21:10:45 +09:00
Paul "TBBle" Hampson b126501d78 Provide the known default PATH to the llbtest test 
The existing test fails because the Windows default PATH (as far as LLB
knows) is empty. See the system.DefaultPathEnv comments.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
 2020-07-28 18:07:20 +10:00
Paul "TBBle" Hampson 5e3e18a72d Use OS-agnostic os.SameFile instead of inode comparision 
This fixes the build of this test under Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
 2020-07-28 13:50:48 +10:00
Tõnis Tiigi a67af5b10c
Merge pull request #1581 from alexcb/tar-exporter-with-socket-copy-reproduce-error 
fix socket handling during copy
 2020-07-23 12:07:15 -07:00
Alex Couture-Beil 5382a2056e Treat unix sockets as regular files 
This fix is similar to the fix in #1144; but was hit in a different code
path.

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
 2020-07-23 11:25:48 -07:00
Tonis Tiigi 5da4a40ae8 lint: enable more linters 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-07-19 09:28:36 -07:00
Tonis Tiigi dc40e5708c client: update dialer to contextDialer 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-07-19 09:28:36 -07:00
Tonis Tiigi c8190b1518 lint: enable golint 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-07-19 09:28:24 -07:00
Tonis Tiigi 57a88b0afb lint: add misspell and unused 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-07-18 08:13:57 -07:00
Tonis Tiigi 04233f90b3 replace gometalinter with golangci linter 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-07-18 08:13:48 -07:00
Paul "TBBle" Hampson 156d66da02 Revert "Pin containerd runtime to v1" 
This reverts commit 9290c15ffc.

This was pinned during the upgrade to containerd 1.3 series, which
changed the default runtime on Linux to io.containerd.runc.v2.

No specific rationale was listed for this pinning, and clearly it's the
wrong thing to do in the presence of Windows, which does not have this
runtime.

Instead, we rely on the containerd-internal defaults, which distinguish
the runtimes for Linux and Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
 2020-07-16 15:06:20 +10:00
Erik Sipsma 43b58b2016 client test: log skip in checkAllReleasable. 
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
 2020-06-24 12:07:00 -07:00
Erik Sipsma 9e0870415d Test media type after first push in TestBuildExportWithUncompressed 
There is a bug in the way images are pushed that results in oci types being used
for layers even when docker types should be used, but only in single-layer
images.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
 2020-06-24 09:53:34 -07:00
Erik Sipsma 83af7c3c94 client test: Fix TestBuildExportWithUncompressed 
This test had previously been accidentally not executing past the check for
whether containerd was running. A previous commit fixes that check, this commit
fixes a few bugs that had been going unnoticed in the containerd-specific part
of the test's code.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
 2020-06-21 20:42:53 -07:00
Erik Sipsma 463ec47ba0 client test: Fix check for whether sandbox has containerd 
Before this, the check was always returning that containerd wasn't running and
thus skipping the rest of several test cases.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
 2020-06-21 20:42:53 -07:00
Tonis Tiigi 49de675234 git: support for token authentication 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-06-16 17:00:06 -07:00
Cory Bennett e351bfdd01 add unit test for getmount output index 
Signed-off-by: Cory Bennett <cbennett@netflix.com>
 2020-06-04 18:25:03 -07:00
Cory Bennett 2b462fc6ca skip tmpfs mounts when calculating indexes for llb.GetMount 
Signed-off-by: Cory Bennett <cbennett@netflix.com>
 2020-06-04 10:54:32 -07:00
Edgar Lee fbee6cccbd Fix source map test in client_test 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-05-20 19:04:25 -07:00
Edgar Lee 7c81e16b8a Fix duplicate source maps and fix issue preventing multiple locations per source map 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-05-20 18:48:20 -07:00
Edgar Lee 59fa9fc9a0 Allow multiple source locations to be added as a constraint 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-05-20 16:48:44 -07:00
Edgar Lee 7a90a36b46 Support multiple source maps and locations for source-level stacktraces 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-05-20 16:17:35 -07:00
Tonis Tiigi 90c5e67496 client: add source mapping tests 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-05-19 17:22:14 -07:00
Tonis Tiigi 6073e6cff3 llb: enable source tracking 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-05-18 15:50:03 -07:00
Tonis Tiigi 1f9599aba3 llb: move source mapping to llb metadata 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-05-18 15:49:41 -07:00
Tonis Tiigi 725f5e1207 grpc interceptors for errors 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-22 18:31:32 -07:00
Tonis Tiigi 2fc9aee26f Revert "grpc: add error handling wrappers to client/server" 
This reverts commit 3f77f0495b.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-22 18:31:27 -07:00
Tonis Tiigi 3f77f0495b grpc: add error handling wrappers to client/server 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-21 23:46:11 -07:00
Tonis Tiigi 37b8832d00 upgrade errors checks to Is() 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-18 22:53:38 -07:00
Tõnis Tiigi 462d7ceb41
Merge pull request #1434 from tonistiigi/dockerd-tests 
[carry] run integration tests against dockerd
 2020-04-14 09:05:36 -07:00
Edgar Lee 8ff11ce5c0 Protect defop map when marshalling concurrently 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-04-10 16:22:08 -07:00
Tonis Tiigi d64e417b1c hack: allow testing with dockerd 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-08 23:27:20 -07:00
Sam Whited c62983bbde all: run integration tests against dockerd 
Signed-off-by: Sam Whited <sam@samwhited.com>
 2020-04-08 23:27:17 -07:00
Edgar Lee d42b8bfc10 Set single export entry outside of session initialization 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-04-08 14:07:35 -07:00
Edgar Lee 423f5f8a97 Allow sessions to hijack the connection outside of a solve 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-04-08 12:13:42 -07:00
Tõnis Tiigi c44cb42a69
Merge pull request #1428 from tonistiigi/imagemeta-async 
llb: update image meta resolver to async callback
 2020-04-06 10:00:42 -07:00
Tonis Tiigi 0ef961eb87 llb: make generic runoptions stateoptions 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-05 22:30:58 -07:00
Tonis Tiigi 146ef63e4a llb: update image meta resolver to async callback 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-05 22:25:02 -07:00
Tonis Tiigi 1a9d366b49 llb: asyncronous llb graph generation support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-04-03 14:55:10 -07:00
Xiaofan Zhang 0947ed31ea vendor: update fsutil to c2c7d7b0e144 and symlink test 
Signed-off-by: Xiaofan Zhang <xiaofan.zhang@clinc.com>
 2020-03-31 02:59:34 +00:00
Akihiro Suda b42528249a
Merge pull request #1295 from zabio3/bugfix/issue1242 
support specifying a custom tag using --import-cache type=local
 2020-03-11 13:10:22 +09:00
Tonis Tiigi 09e8a06c04 client: improve cache key loop test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-03-02 22:35:59 -08:00
Anca Iordache 8c65b5bbd5 Tests build with local cache export 
- to detect infinite loop caused by circular dependencies

Signed-off-by: Anca Iordache <anca.iordache@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-03-02 22:35:59 -08:00
Tonis Tiigi a60ecfa4ae vendor: restore dependency versions 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2020-02-24 17:31:01 -08:00
Edgar Lee 5ec7bc7713 Change FrontendInputs in backend API definitions to *pb.Definition 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-02-24 14:38:01 -08:00
Edgar Lee b027a8572d Fix terminating op with non-zero output index, i.e. AddMount as final vertex 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-02-24 14:38:01 -08:00
Edgar Lee 7e0f923181 Implement CapFrontendInput to pass llb.States to frontends 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-02-24 14:38:01 -08:00
zabio3 807b20346e support specifying a non-latest tag using --import-cache type=local 
https://github.com/moby/buildkit/issues/1242

Signed-off-by: Tomohiro Kusumoto <zabio1192@gmail.com>
 2020-01-29 21:32:55 +09:00
Edgar Lee c83e8bf2a9 Add gateway apicap for CapReferenceOutput and move reference llb.Output to method ToOutput() 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-01-23 11:36:30 -08:00
Edgar Lee f937656b1a Verify platform is stored in state directly in defop test 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-01-23 11:02:40 -08:00
Edgar Lee 0a76749067 Add platform defop test, force renames in copy tests, and move forwarded solve test to dockerfile package 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-01-23 11:02:40 -08:00
Edgar Lee be7f1a36fa Return def bytes instead of remarshalling definition from pb.Op 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-01-23 11:02:40 -08:00
Edgar Lee 2edb431a0f Allow previous solve results to be used in new solves 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2020-01-23 11:02:40 -08:00
Akihiro Suda 6e62650fac
Merge pull request #1285 from jeffreyhuang23/issue-1240 
Fix issue #1240 (ignore non-existent local cache)
 2019-12-13 14:11:55 +09:00
Nikhil Pandeti f3dd3193b6 Only ignore errors if the src directory or index.json do not exist 
Error out if wrong parameters were passed or no src set

Signed-off-by: Nikhil Pandeti <nikhil.pandeti@utexas.edu>
 2019-12-12 13:06:14 -06:00
Nikhil Pandeti 6a9ce04908 Gofmt and goimports used on changed file 
Signed-off-by: Nikhil Pandeti <nikhil.pandeti@utexas.edu>
 2019-12-10 08:54:34 -06:00
Nikhil Pandeti 0206c71206 Fix issue #1240 
Allow build to run even if import cache of type=local is not found
Added an additional check in solve.go to catch errors thrown when
local import caches are not found. If none are found, continues as
if no import cache was specified.

Signed-off-by: Nikhil Pandeti <nikhil.pandeti@utexas.edu>
 2019-12-10 00:55:29 -06:00
Wei Fu 65388edc6c exporter: support compression for layer blob data 
Allow user to choose the compression type for layer data. Gzip is
default compression for layer exporter, which consume more Cpu resources
and take long time to export. With compression option, user can use
nocompressed option to export to save time. And future, zstd is one new
option for end-user.

Signed-off-by: Wei Fu <fuweid89@gmail.com>
 2019-12-09 23:51:38 +08:00
Edgar Lee c122adacb5 Implement file action remove for wildcards 
Signed-off-by: Edgar Lee <edgarl@netflix.com>
 2019-10-30 14:25:00 -07:00
Tonis Tiigi 08194783e0 cache: update unit tests to new cache manager constructor 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-10-16 10:34:50 -07:00
Tonis Tiigi fef0996045 dockerfile: skip cni tests on rootless 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-09-05 16:49:48 -07:00
Tonis Tiigi 1d7e7a6b51 llbsolver: fix error on multiple cache importers 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-08-29 15:21:17 -07:00
Tibor Vass 77ca4bce2e client: add exporter test for sockets 
Signed-off-by: Tibor Vass <tibor@docker.com>
 2019-08-27 20:53:37 +00:00
Tonis Tiigi ac8fd77049 client: update tests to new deps 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-08-20 17:03:05 -07:00
Michael Crosby 9290c15ffc Pin containerd runtime to v1 
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
 2019-08-20 17:03:04 -07:00
Tonis Tiigi 2cea1107d6 travis: add ci caching 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-08-17 09:13:06 -07:00
Tonis Tiigi fb3f2aeb63 client: add test for output target exists 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-07-31 10:21:26 -07:00
Tonis Tiigi 96b6a28312 exporter: allow oci exporters visibility to response metadata 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-07-31 10:21:19 -07:00
Tonis Tiigi 7b1bae7a42 solver: support no-cache for exec cache mounts 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-07-26 14:42:32 -07:00
Akihiro Suda d750cb188a
Merge pull request #1073 from tonistiigi/cni-net 
add cni networking support
 2019-07-19 10:47:27 +09:00
Andrey Smirnov df52fc7f9c Skip test in rootless mode 
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
 2019-07-18 22:50:53 +03:00
Andrey Smirnov fc9781a77c Add test for 'rw' sysfs/cgroup re-mount 
Signed-off-by: Andrey Smirnov <smirnov.andrey@gmail.com>
 2019-07-18 18:53:27 +03:00
Tonis Tiigi e7759a861b client: add cni network tests 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-07-10 18:00:06 -07:00
Tonis Tiigi 1ef979a7bb llb: fix excessive formatting 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-07-01 16:55:08 -07:00
Tonis Tiigi 8e47580f71 fix flaky TestBasicInlineCacheImportExport 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-06-10 14:16:42 -07:00
Tonis Tiigi d72c19071a imageutil: use leases on fetching config 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-05-28 14:39:05 -07:00
Akihiro Suda 8483d06bec connhelper/dockercontainer: support Docker context 
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
 2019-05-26 08:41:33 +09:00
Dave Chen 8e9cc2794e cleanup: no need to mkdir when importing cache 
Signed-off-by: Dave Chen <dave.chen@arm.com>
 2019-05-11 14:11:14 +08:00
Tonis Tiigi d106163985 llb: fix security mode cap check 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-04-27 08:57:36 -07:00
Tibor Vass e2f33ce4f0
Merge pull request #934 from tonistiigi/shared-session 
client: add shared session possibility
 2019-04-22 17:39:13 -07:00
Akihiro Suda fc044d0e6b new connhelper: kube-pod 
Fix #769
Relates to tonistiigi/buildx#22

Usage:

  $ kubectl run --generator=run-pod/v1 --image moby/buildkit:master-rootless bk -- --oci-worker-no-process-sandbox
  $ export BUILDKIT_HOST=kube-pod://bk

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
 2019-04-19 20:34:00 +09:00
Tonis Tiigi ef58b61d83 client: add shared session possibility 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-04-18 08:33:57 -07:00
Akihiro Suda 90b73ff6d2 connhelper: docker -> docker-container 
for consistency with tonistiigi/buildx#17

Close #935

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2019-04-14 21:14:40 +09:00
Tonis Tiigi e189eb7721 connhelper: refactor helpers to own packages 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-04-02 22:17:41 -07:00
Tonis Tiigi 15ef4e2b8a dockerfile: allow content cache for rw mounts 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-04-02 19:40:30 -07:00
Akihiro Suda 4f4bcade64
Merge pull request #907 from tonistiigi/tarball-exporter 
exporter: add tarball exporter
 2019-03-29 12:29:00 +09:00
Tõnis Tiigi 758c61e873
Merge pull request #570 from kunalkushwaha/entirlement-revised 
security entitlement support
 2019-03-28 16:14:06 -07:00
Tonis Tiigi c1a1d7033d exporter: add tar exporter 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-27 09:40:24 -07:00
Tonis Tiigi 01a8de7d74 integration: secmode switch support and security testcase 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Kunal Kushwaha <kunal.kushwaha@gmail.com>
 2019-03-27 13:57:03 +09:00
Kunal Kushwaha a2bbb5ff39 security entitlement support 
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
 2019-03-27 13:57:03 +09:00
Tibor Vass fe0b8a4ec7
Merge pull request #902 from tonistiigi/itegration-update 
integration: generalize pkg
 2019-03-26 16:35:12 -07:00
Tonis Tiigi 6081335ee5 client: allow setting custom dialer 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-26 11:33:56 -07:00
Akihiro Suda 2f007a47a9 client: add docker:// connhelper 
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2019-03-26 21:25:01 +09:00
Tonis Tiigi 7e52b33bf8 integration: generalize pkg 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-21 23:49:08 -07:00
Tibor Vass 9a9e755588
Merge pull request #887 from tonistiigi/push-by-digest 
exporter: allow push by digest in image exporter
 2019-03-20 19:13:09 -07:00
Akihiro Suda 92f1fb55d5
Merge pull request #888 from tonistiigi/fileop-cap 
client: set cap if fileop used
 2019-03-20 13:47:45 +09:00
Tonis Tiigi 8f174bcf52 client: add hardlinks local export test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-18 19:12:34 -07:00
Tonis Tiigi e8432d677b vendor: update fsutil to 1bdbf123ad 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-18 19:11:39 -07:00
Tonis Tiigi 210679d35f client: set cap if fileop used 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-18 16:35:30 -07:00
Tonis Tiigi 37bfc6546c exporter: allow push by digest in image exporter 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-18 16:00:48 -07:00
Tonis Tiigi 0d17ac323e fileop: updates with new fsutil copy pkg 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 17:49:58 -07:00
Tonis Tiigi 8a4674bab4 fileop: add dockerfile support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 17:49:50 -07:00
Tonis Tiigi 81a5fa5a2e llbsolver: fileop implementation 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 16:22:35 -07:00
Tonis Tiigi a443cfff05 fileop: resolve review comments 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 16:22:35 -07:00
Tonis Tiigi 89e6614b38 solver: change uid to uint 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 16:22:35 -07:00
Tonis Tiigi 431d11dda3 llb: add timestamp override to fileop 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 16:22:35 -07:00
Tonis Tiigi 5b4841f308 llb: initial fileop implementation 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-15 16:22:01 -07:00
Tonis Tiigi f4820681ce client: fix panic on exporting cache 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-03-12 11:19:29 -07:00
Akihiro Suda 5c9f7b8ff0 buildctl: new CLI ("Option C+") 
See https://github.com/moby/buildkit/pull/807#issuecomment-468146089

Close #774

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2019-03-06 13:20:21 +09:00
Iskander (Alex) Sharipov 6e363c2309
client: fix strings.Replace call args 
strings.Replace call with n=0 argument makes no sense
as it will do nothing. Probably -1 is intended.

Signed-off-by: Iskander Sharipov <quasilyte@gmail.com>
 2019-03-02 00:34:48 +03:00
Akihiro Suda 46bde0d6b1 test: make sure `--export-cache type=inline` is not needed for import 
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2019-02-23 21:01:27 +09:00
Akihiro Suda 3ba3f5b1ff
Merge pull request #777 from tonistiigi/export-cache-inline 
allow exporting cache metadata in the image config
 2019-01-27 06:40:59 +01:00
Tonis Tiigi c0371c908f client: avoid string formatting on custom vertex names 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-01-22 15:35:39 -08:00
Tonis Tiigi 5434a42a6a client: add inline remote cache tests 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2019-01-21 19:39:15 -08:00
Akihiro Suda 6e7617e889 support local cache exporter and importer 
Export:

  $ buildctl build ... --export-cache type=local,store=/path/to/output-dir

Import:

  $ buildctl build ... --import-cache type=local,store=/path/to/input-dir

Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`

Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2019-01-19 02:38:26 +09:00
Ian Campbell f9a436599a client/llb: factor out `State.WithImageConfig` from `Image`. 
Currently if a caller wants an `Image` (with config applied) and the
corresponding config (e.g. to use a base image and tailor the config to return
as `exptypes.ExporterImageConfigKey` in the build result) then they have to
`ResolveImageConfig` and then `llb.Image(..., llb.WithMetaResolver)`, which
ends up resolving the config twice.

Pulling this code out allows them to instead to the `config := ResolveImageConfig()`
once and to then do `llb.Image(...).WithImageConfig(config)`.

This is only really necessary if you want to run things in the image context
(and thus want the config applied) as part of tailoring the config before
returning the image and config. If you aren't running things on that `Image` as
part of the build then it doesn't matter if the config isn't applied to the
`Image` and `ResolveImageConfig` + `Image(...)` (without a metaresolver) is
fine.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-10-18 10:06:00 +01:00
Tonis Tiigi 1f51f5ea11 testutil: avoid excessive daemons in integration tests 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-10-11 16:47:35 -07:00
Tibor Vass c9b01072d0 solver: specify SSH key ID in error message when required key was not forwarded 
Signed-off-by: Tibor Vass <tibor@docker.com>
 2018-10-05 22:50:07 +00:00
Tõnis Tiigi ad7e104d02
Merge pull request #642 from ijc/readdir 
Add `ReadDir` to the gateway interface
 2018-10-04 14:10:38 -07:00
Ian Campbell b4454a169f implement `gateway.Reference.StatFile` method 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-10-03 12:16:45 +01:00
Ian Campbell b30ee91a53 implement `gateway.Reference.ReadDir` method 
The alternative is a helper container running e.g. find, capturing the output
to a scratch to be read back (with ref.ReadFile) and parsed.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-10-03 12:16:44 +01:00
Ian Campbell 8b69292087 Bump github.com/tonistiigi/fsutil dependency to f567071bed24 
7e391b0e78...f567071bed

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-10-03 11:46:35 +01:00
Tonis Tiigi cd451ec063 executor: make sure hostname is resolvable 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-10-02 23:24:37 -07:00
Tõnis Tiigi c9dc8352a4
Merge pull request #656 from AkihiroSuda/cap-ssh 
define CapMountSSH
 2018-10-02 23:21:17 -07:00
Akihiro Suda 758a267235 s/CapMount/CapExecMount/g 
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-10-03 15:03:36 +09:00
Akihiro Suda f542300a10 client/llb: fix cap marshalling 
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-10-03 15:02:08 +09:00
Akihiro Suda 900a1b96c4 define CapMountSSH 
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-10-03 15:00:10 +09:00
Akihiro Suda fa92effc4b dockerfile: add support for RUN --mount=type=ssh 
* Needs to be compiled with `dfrunmount dfssh`
* Implemented options:
 * `type`(required): needs to be `ssh`
 * `target`(optional): the socket path in the container
 * `id`(optional): id

Test script:

	#!/bin/bash
	set -exu -o pipefail
	REF=localhost:5000/dfssh:latest
	ssh-add -l
	sudo buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=frontend/dockerfile/cmd/dockerfile-frontend \
	  --frontend-opt "build-arg:BUILDTAGS=dfrunmount dfssh" \
	  --exporter=image --exporter-opt name=$REF --exporter-opt push=true
	mkdir -p /tmp/foo
	cd /tmp/foo
	cat << EOF > Dockerfile
	# syntax=$REF
	FROM alpine
	RUN apk add --no-cache openssh-client
	RUN mkdir -p -m 0700 ~/.ssh && ssh-keyscan gitlab.com >> ~/.ssh/known_hosts
	RUN --mount=type=ssh ssh git@gitlab.com
	# "Welcome to GitLab, @GITLAB_USERNAME_ASSOCIATED_WITH_SSHKEY" should be printed here
	EOF
	sudo buildctl build --ssh default=$SSH_AUTH_SOCK --progress=plain --frontend=dockerfile.v0 --local context=. --local dockerfile=.

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-10-03 14:53:43 +09:00
Ian Campbell 720d8327e2 Push setting of default PATH down into the executor(s) 
Setting the default `PATH` in the `llb.State` on the client side means it
depends on the `GOOS` of the buildkit client, rather than of the environment
where it will actually execute.

Instead defer this to execution time and insert the default PATH at that point
if one is not present. Doing this in solver/llbsolver/ops/exec covers all
executors and also avoids breaking the cache.

Client compatibility is handled via a new capability.

Fixes #604

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-09-28 14:16:47 +01:00
Ian Campbell 6e773ab708 Add `SetDefault` to `EnvList` 
This sets an envvar only if it is not already set.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-09-28 13:42:35 +01:00
Tonis Tiigi f6e104da5f gateway: allow access to apicaps 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-21 15:33:41 -07:00
Tõnis Tiigi 39404586a5
Merge pull request #622 from tonistiigi/df-modes-switch 
dockerfile: enable frontend selection in integration tests
 2018-09-20 14:44:11 -07:00
Tonis Tiigi a75983ddb9 integration: pass mirrored images from test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-18 22:07:38 -07:00
Tonis Tiigi 0940cdc6fe update golint comments 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-18 22:06:47 -07:00
Tibor Vass 36be95ea0e
Merge pull request #623 from tonistiigi/failfast 
buildctl: replace withblock with dial error check
 2018-09-17 11:44:57 -07:00
Akihiro Suda bf571a519e update Go to 1.11 
For consistency with Moby (moby/moby#37358)

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-09-15 14:25:11 +09:00
Tonis Tiigi 653fb12ecf buildctl: replace withblock with dial error check 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-13 17:29:01 -07:00
Tonis Tiigi f7120867f1 client: add regression test for closed stdin 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-11 14:26:46 -07:00
Tonis Tiigi 347420e77f sshprovider: allow keys from local files 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-10 14:12:41 -07:00
Tonis Tiigi 9337126e6e client: add ssh socket test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-07 08:57:11 -07:00
Tonis Tiigi 6888956557 sshforward: implement ssh socket forwarding 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-07 08:57:06 -07:00
Tõnis Tiigi 9a6e2a5a91
Merge pull request #607 from kunalkushwaha/fix-typo-1 
function name fixed SetMarhalDefaults -> SetMarshalDefaults
 2018-09-05 22:01:39 -07:00
Kunal Kushwaha 98c1f44151 function name fixed SetMarhalDefaults -> SetMarshalDefaults 
Signed-off-by: Kunal Kushwaha <kushwaha_kunal_v7@lab.ntt.co.jp>
 2018-09-06 13:28:39 +09:00
Tonis Tiigi 079c747c4e ontrol: support calling automatic GC 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-09-04 18:13:29 -07:00
Tõnis Tiigi 05a468d460
Merge pull request #595 from tonistiigi/uid 
client: reset user ID for synced dirs
 2018-08-31 12:14:06 -07:00
Tõnis Tiigi 2759005f6a
Merge pull request #575 from ijc/frontend-set-export-name 
exporters: allow frontend control of the exported image name
 2018-08-31 10:27:09 -07:00
Ian Campbell 03a9cb89af exporter: Use "*" to signify that client allows frontend to set image name 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-31 17:34:49 +01:00
Ian Campbell 0b20bcc8f3 The OCI image exporter does not support naming, so reject such attempts. 
Adjust the test case to not name in the OCI case.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-31 14:23:34 +01:00
Ian Campbell aa51b40649 Include exported image name in ExporterResponse metadata 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-31 11:07:32 +01:00
Tõnis Tiigi c9711fcc26
Merge pull request #574 from ijc/return-frontend-metadata 
Allow some Metadata to be return from frontend to the control client
 2018-08-30 10:08:51 -07:00
Ian Campbell 755bf5387e Moved some subtests into `t.Run` blocks. 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-30 16:28:19 +01:00
Ian Campbell 3db7441e06 clarify naming in testFrontendImageNaming 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-30 15:59:12 +01:00
Ian Campbell 8d114533c2 exports: require caller to request frontend-based naming, by passing name of `%s`. 
Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-30 15:59:06 +01:00
Tonis Tiigi 64ff38f163 client: reset user ID for synced dirs 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-08-28 14:22:22 -07:00
Akihiro Suda decb27618b containerd: set timeout explicitly (60s) 
The contained default value was shortened from 60s to 10s in containerd/containerd#2554

Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
 2018-08-23 18:14:57 +09:00
Ian Campbell fa39d134ee exporters: allow frontend control of the exported image name 
Returning a metadata item named "image.name" will set the name to use.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-17 13:20:36 +01:00
Ian Campbell fdf13cf96e Allow some Metadata to be return from frontend to the control client 
Propagate anything in the `frontend.*` namespace from the `frontend.Result`
into the `client.SolveResponse`.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-17 12:45:21 +01:00
Akihiro Suda af46188e9b
Merge pull request #533 from ijc/client-gateway 
access gateway API from client
 2018-08-17 05:12:23 +09:00
Ian Campbell 22f632f895 allow frontends to be run directly on the control client side 
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.

Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.

The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-08-14 11:50:08 +01:00
Tonis Tiigi 130f5f5ab0 solver: net host with basic entitlements support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-08-09 14:03:35 -07:00
Tõnis Tiigi 47c4888ab1
Merge pull request #551 from tonistiigi/gc-prune 
cache: add gc style pruning support
 2018-08-07 11:14:46 -07:00
Tonis Tiigi 1a37ee3bf4 cache: add gc style pruning support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-08-06 21:45:39 -07:00
Tonis Tiigi bf29f5bd3b client: add test for extra hosts 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-08-03 13:56:13 -07:00
Tonis Tiigi f7359f7093 llb: add extrahost support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-08-03 13:56:13 -07:00
Tonis Tiigi 4c44d2c6e4 dockerfile: custom names to vertexes 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-31 10:41:30 -07:00
Tonis Tiigi d1ad6961e3 cache: detect refs shared with image store 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-27 14:08:50 -07:00
Tonis Tiigi 97fa2b527a cache: add all option to prune 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-26 22:55:05 -07:00
Tonis Tiigi 489246dd28 cache: support for internal/frontend record type 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-26 22:54:53 -07:00
Tonis Tiigi 57b96a0ee5 cache: add record type field to usage record 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-26 22:54:39 -07:00
Tibor Vass aa81807f1c
Merge pull request #542 from tonistiigi/resolvemode 
llb: add resolvemode to image source
 2018-07-26 14:04:27 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-25 18:22:15 -07:00
Tonis Tiigi a3b5d22419 llb: add resolvemode to image source 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-25 18:20:04 -07:00
Tonis Tiigi 70e0b64482 client: generalize the filter passing 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-25 18:19:31 -07:00
Tonis Tiigi 0ba24f2a90 cache: implement filter for diskusage 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-25 17:20:59 -07:00
Tonis Tiigi f719aadcba dockerfile: add tests for multi-platform exports 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-25 16:01:12 -07:00
Tonis Tiigi 53f657a5db client: add secret mounts test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-19 22:02:11 -07:00
Tonis Tiigi 378df8cbaa llb: add secrets support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-19 22:02:11 -07:00
Tonis Tiigi d4c57aec5d llb: add caps support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-18 17:22:30 -07:00
Tonis Tiigi c9df92f4c5 imagemetaresolver: use platform in cache 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-17 18:36:32 -07:00
Tonis Tiigi 1456a3f165 llb: don't fill platform on static sources 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-16 15:27:03 -07:00
Akihiro Suda dca7bdcd4c
Merge pull request #495 from tonistiigi/unique-local 
llb: set unique ID to local sources without session
 2018-07-10 20:34:04 +09:00
Tonis Tiigi 946a12eb42 gateway: expose worker information 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-09 16:32:39 -07:00
Tonis Tiigi 40e270a2e6 llb: set unique ID to local sources without session 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-09 16:27:12 -07:00
Tonis Tiigi 3fa70c6b25 client: simplify parallel sources test 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-07-09 16:27:08 -07:00
Ian Campbell 0dd25df7e2 client: add test case for multiple parallel builds. 
It appears that multiple parallel builds suffer some cross talk between their
mount/snapshots e.g. this new case currently fails because iterations see files
in `/src/` which correspond to other iterations (i.e. iteration 1 might see a
file called "test2.txt"). The `ls -i` and `stat` output also confirms that
where tests see the same filename they are also apparently seeing the same
inode.

Note that each solve uses a `SolverOpt.SharedKey` of the (unique) source
directory so there should be no clashes. Use of `llb.sharedKeyHint()` is
avoided since using a unique one of these hides the issue (AIUI uniqueness of
`sharedKeyHint` is only required with a session, not between them).

The shell fragment within the test is a bit crazy, it tries to output as much
info about what it saw vs what was expected to aid debugging.

I have also been seeing cases in my own code where `cacheContext.checksum`
fails the `not found` test at the top, which seems like either the wrong thing
is mounted or it has gone away or otherwise been mutated unexpectedly. I had
been hoping that this test case would also reproduce this, but so far it has
not.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-07-09 15:44:28 -07:00
Tõnis Tiigi f3b743bd13
Merge pull request #467 from dmcgowan/export-oci-media-types 
Support creating and pushing OCI types
 2018-06-28 14:46:34 -07:00
Derek McGowan 618f34a1f9 Support creating and pushing OCI types 
Adds image and oci exporter option "oci-mediatypes"
Ensures that the images created in the content store
have the correct type which matches the manifest.
Sets the correct media type on the descriptor in push from
reading the type specified in the manifest.
Removes use of distribution manifest packages.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
 2018-06-28 11:23:35 -07:00
Tonis Tiigi e618878451 dockerfile: platform support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-25 11:06:08 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-25 10:39:17 -07:00
Akihiro Suda 19612b901f
Merge pull request #462 from tonistiigi/llbconstraints 
llb: force platform in llb and allow constraints
 2018-06-26 01:03:38 +09:00
Tonis Tiigi 242697af59 llb: add marshal roundtrip tests for platform 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-22 17:55:36 -07:00
Tonis Tiigi d34e4c68b5 llb: automatically carry platform with state chain 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-22 16:49:03 -07:00
Tonis Tiigi dc9de85069 llb: force platform in llb and allow constraints 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-22 16:49:03 -07:00
Tonis Tiigi a1bafadfd5 worker: add platforms support 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-22 11:46:46 -07:00
Tonis Tiigi ccbf185006 llbsolver: allow sharing modes for cache mounts 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-19 15:57:20 -07:00
Ian Campbell 97ffd6cce8 client: take a context.Context to New(). 
This allows two things:

- The caller to set a shorter timeout than previously hardcoded 30s. In
  `buildctl` reduce the timeout to 5s. Since the existing timeout has gone
  callers will need to arrange to pass one themselves.
- The caller can arrange for the context to be cancelled for other reasons, use
  this in `buildctl` to plumb through the Ctrl-C handling, meaning that
  `buildctl` now exits almost immediately on Ctrl-C instead of after several
  seconds.

Signed-off-by: Ian Campbell <ijc@docker.com>
 2018-06-11 14:28:08 +01:00
Tonis Tiigi 30461204da vendor: update fsutil to fix local source bug 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-10 09:22:59 -07:00
Akihiro Suda f381aaac05
Merge pull request #445 from tonistiigi/tmpfs 
llbsolver: add support for tmpfs mounts to exec
 2018-06-09 11:52:16 +09:00
Tonis Tiigi 596b03a8a0 llbsolver: allow tmpfs mounts 
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
 2018-06-08 13:05:14 -07:00