Commit Graph

2513 Commits (15275251102e6060e79b2f70a7e8adcf3919d865)

Author SHA1 Message Date
Tonis Tiigi 9cb0654ce4 integration: lock buffer for concurrent writes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-23 12:04:14 -07:00
Alex Couture-Beil 5382a2056e Treat unix sockets as regular files
This fix is similar to the fix in #1144; but was hit in a different code
path.

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-07-23 11:25:48 -07:00
Tõnis Tiigi 754bcfbe51
Merge pull request #1589 from thaJeztah/update_fsutil
vendor: update tonistiigi/fsutil v0.0.0-20200720184806-8f851f9d2045
2020-07-23 09:25:57 -07:00
Sebastiaan van Stijn 3890bbe6f7
vendor: update tonistiigi/fsutil v0.0.0-20200720184806-8f851f9d2045
full diff: ae3a8d7530...8f851f9d20

notable changes:

- walker: fix notadir error
- improving error returns
    - more typed errors
    - remove extra verbosity (eg. PathError already contains action and path)
    - ensure stack traces are added to errors

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-07-23 17:05:59 +02:00
Tõnis Tiigi b3cf4e508c
Merge pull request #1588 from TBBle/no_chopt_works_everywhere
Support not chowning files on all platforms
2020-07-22 10:39:02 -07:00
Paul "TBBle" Hampson 3c96d0b863 Support not chowning files on all platforms
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-23 03:18:30 +10:00
Tõnis Tiigi 5de894a607
Merge pull request #1585 from TBBle/windows_cni_support
CNI support for Windows
2020-07-20 23:57:38 -07:00
Paul "TBBle" Hampson 50c5d4ce51 Don't add loopback CNI configuration on Windows
There's no Loopback CNI plugin for Windows as far as I can see, so I'm
assuming it's not needed.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-21 16:31:32 +10:00
Paul "TBBle" Hampson 1fedff0e4b Reorder CNI config to set options before consumers
The `WithLoNetwork` and `WithConfFile` options consume the
`WithPluginDir` and `WithInterfacePrefix` settings, so they must come
first.

This fixes the issue that the provided plugin directory was not used for
the provided configuration file. This would have been more visible, but
the default `InterfacePrefix` is already "eth".

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-21 16:31:31 +10:00
Paul "TBBle" Hampson dc2574ceae Implement separate handling for Windows Network namespaces
The Windows API is completely different, so the common code cannot
create the unique identifier, but must accept what the OS-specific code
does.

For Linux, it's a pathname. In Windows, it's a GUID string.

This pulls in direct usage of Microsoft/hcsshim, which previously was
only an indirect usage via containerd calls from, e.g., localMounter on
Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-21 16:31:31 +10:00
Paul "TBBle" Hampson 1645a1e2ca Provide separate NetProviders map for Windows
This enforces the lack of host networking support on Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-21 16:31:31 +10:00
Akihiro Suda 40b0d17fc5
Merge pull request #1582 from tonistiigi/new-lint
replace gometalinter with golangci linter
2020-07-21 14:33:15 +09:00
Tonis Tiigi 94c202bfb9 handle error returns more properly
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi 5da4a40ae8 lint: enable more linters
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi dc40e5708c client: update dialer to contextDialer
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi 0ed9eb77ce contenthash: fix buffer reuse pool
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi bd3354fea3 session: avoid deprecated grpc.Stream type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi 57abacc68d contenthash: avoid deprecated Xattrs header
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi 3e9c84ff86 avoid returning private type from public function
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi e4aaec2fea package name fixes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi 06c4197c05 rename binfmt_misc to archutil
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tonis Tiigi c8190b1518 lint: enable golint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:24 -07:00
Tonis Tiigi 57a88b0afb lint: add misspell and unused
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-18 08:13:57 -07:00
Tonis Tiigi 04233f90b3 replace gometalinter with golangci linter
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-18 08:13:48 -07:00
Akihiro Suda 4d1f260e84
Merge pull request #1578 from tonistiigi/deploy-fix
hack: unblock master deploy
2020-07-18 12:27:43 +09:00
Tõnis Tiigi 27bea0af21
Merge pull request #1579 from tonistiigi/tests-debug
integration: tests flakiness fixes
2020-07-17 20:11:41 -07:00
Tonis Tiigi 55e29c1abe integration: fix rootless worker binaries closing after tests
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-17 19:30:40 -07:00
Tonis Tiigi 8247b7b50b integration: increase timeouts
Set the limit high enough to ensure failure in CI
signals a bug

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-17 19:30:40 -07:00
Tonis Tiigi adb68c2764 integration: add concurrent sandbox limit
Default parallelization seems off in CI

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-17 19:30:33 -07:00
Tonis Tiigi f825fea268 buildkitd: ensure stack traces on logging errors
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-17 19:18:49 -07:00
Tõnis Tiigi 97ca82f4c4
Merge pull request #1576 from coryb/frontendllbbridge-exec-refactor
add Run/Exec to FrontendLLBBridge interface
2020-07-16 20:45:30 -07:00
Tonis Tiigi 0089fbc454 hack: unblock master deploy
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-16 20:42:14 -07:00
Tõnis Tiigi e65986cbe8
Merge pull request #1577 from sipsma/faster-remotecache
remotecache: Only visit each item once when walking results.
2020-07-16 20:11:24 -07:00
Cory Bennett a23754fca1 Release needs context.TODO() so if ctx is cancelled the refs can be cleaned up
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-07-17 02:08:03 +00:00
Cory Bennett 441b76947b update FrontendLLBBridge to be composed of Executor
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-07-17 01:05:22 +00:00
Cory Bennett 6a56695d3a remove `Get` prefix from Executor/CacheManager funcs and update to avoid naming conflicts
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-07-17 00:54:15 +00:00
Erik Sipsma 806fddd7f7 remotecache: Only visit each item once when walking results.
Before this change, walkAllResults did not skip an item if it had already been
visited, so walking the graph of results actually followed every single possible
path in the graph instead of just visiting each item once.

On small graphs this wasn't noticeable, but on sufficiently large remote cache
imports this rapidly scaled out of control. For example, I first encountered
this when importing a max-cache from a build of a full linux rootfs from source
(which takes about 30 minutes to build w/out cache on an 18-core machine) and
after 30 minutes the cache import was still running with all CPUs pegged at 100%

To fix this, walkAllResults now keeps a map of already visited items (keyed by
their pointer) and skips visiting an item if it's already been visited, making
it usable on large remote cache imports.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-07-16 17:49:36 -07:00
Cory Bennett 453c76d29e add Run/Exec to FrontendLLBBridge interface
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-07-16 23:20:14 +00:00
Tõnis Tiigi 919ee62a78
Merge pull request #1574 from sipsma/fix-containerd-test
Use new containerd shim in integ tests; don't skip on requirement error.
2020-07-16 11:26:45 -07:00
Tõnis Tiigi 62d1fff289
Merge pull request #1573 from TBBle/small_fixes
Make Travis usable for forks
2020-07-16 11:23:47 -07:00
Erik Sipsma f828f65516 Use new containerd shim in integ tests; don't skip on require error.
A recent change updated the containerd shim built for integ tests to the newer
v2, but the integ test code itself was still hardcoded to look for the older
shim binary and, when that binary wasn't found, skip running the test. This
caused all containerd tests to be skipped.

The fix here updates the binary name to v2 and also removes the branch that
will skip a test when the requirements check fails.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-07-16 10:21:21 -07:00
Paul "TBBle" Hampson 7369864344 Move printLogs from oci.go to sandbox.go
It's not OCI-specific, so it makes more sense in the file that exports
it to the world.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-17 02:35:16 +10:00
Paul "TBBle" Hampson 25f50eb711 Wrap original error when failing to read download context
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-17 02:35:16 +10:00
Paul "TBBle" Hampson bbd372d00e Correctly build forks in Travis
The CI script goes looking for the branch to which a commit was pushed,
but was applying the branch name to moby/buildkit even if being built on
a different fork.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-17 02:35:16 +10:00
Tõnis Tiigi 94696710fa
Merge pull request #1571 from TBBle/windows_layer_mount_support
Split LocalMounter.Mount into Windows and Unix
2020-07-16 08:59:49 -07:00
Tõnis Tiigi cc8097127b
Merge pull request #1570 from TBBle/use-containerd-default-shim
Use containerd's default runtime shim instead of hard-coded runtime v1
2020-07-16 08:58:20 -07:00
Akihiro Suda b2336d60bf
Merge pull request #1568 from tonistiigi/inline-fix
inline cache: fix handling of duplicate blobs
2020-07-16 18:20:02 +09:00
Paul "TBBle" Hampson cd47186783 Split LocalMounter.Mount into Windows and Unix
Due to current deficiencies in the Windows implementation of
containerd's Mount.Mount, we need different behaviour for mounting on
Windows and Unix platforms.

Specifically:
* Windows mounts can only mount in-place, and hence only one mount
  should be in the list.
* BuildKit doesn't own the mount directory, so should not try and remove
  it on unmount.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-16 17:13:46 +10:00
Paul "TBBle" Hampson 97314e8567 Don't build the containerd-shim shim for tests
This is the shim used by the containerd Runtime V1 on Linux,
which was the default under containerd 1.2 and earlier.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-16 15:06:20 +10:00
Paul "TBBle" Hampson 156d66da02 Revert "Pin containerd runtime to v1"
This reverts commit 9290c15ffc.

This was pinned during the upgrade to containerd 1.3 series, which
changed the default runtime on Linux to io.containerd.runc.v2.

No specific rationale was listed for this pinning, and clearly it's the
wrong thing to do in the presence of Windows, which does not have this
runtime.

Instead, we rely on the containerd-internal defaults, which distinguish
the runtimes for Linux and Windows.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-16 15:06:20 +10:00