9f254e1836
Merge pull request #2178 from tonistiigi/metadata-lock
...
metadata: hold lock on storageitem update
2021-06-15 09:25:40 -07:00
976b7e9a97
metadata: hold lock on storageitem update
...
The locks usage is mixed up because two locks separate locks
are actually needed. With a specific lock, calls to SetValue
can be protected.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-14 20:27:01 -07:00
5429fe1b63
cache: avoid concurrent maps write on prune
...
remove() needs to be called while holding the manager lock
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-14 19:26:14 -07:00
1764b3b249
Enable stargz snapshotter to use session for snapshot auth
...
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2021-06-09 16:10:08 +09:00
167e0dc74a
cache: fix possible nil dereferences
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-07 20:26:05 -07:00
7c13af37a4
Revendor updated fsutils fork; use same logic for IncludePatterns and ExcludePatterns
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
5f90a18501
Add test case for added file which doesn't match include pattern
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
fc20bebf5e
Exclude source dir metadata from checksum if source path ends in separator
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
607dc1f2bf
return false, false, nil
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
b94fc7f4d7
Seek to path if !wildcard
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
32e52f15bc
Update vendored fsutil fork
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
7a35420c13
Include parent directories of include patterns in cache checksum
...
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
626744c9e7
Cache awareness of IncludePatterns/ExcludePatterns
...
Consider IncludePatterns and ExcludePattern when calculating content
hashes.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
c47972dc5a
Merge pull request #2018 from tonistiigi/wildcard-copy
...
contenthash: include basename in content checksum for wildcards
2021-04-14 15:58:43 +09:00
b62f436771
metadata: clone external data
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-30 13:30:29 -07:00
ba90a9ee33
remotecache: lock backlinks cache
...
Items may be reused by the cache resulting concurrent writes.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-29 22:13:40 -07:00
1982e1e285
contenthash: include basename in content checksum for wildcards
...
While we generally ignore the basename in this layer, for wildcards
there in no other place to add the basename to the checksum as they
can not be resolved earlier. Before the basename that was in the
checksum was the wildcard itself, so if the wildcard remained same,
content remained same but the file where wildcard pointed to was
renamed, the cache was not invalidated.
Unfortunately, this change breaks cache for all copy commands that
use a wildcard.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-23 17:15:50 -07:00
f3e88854a4
remotecache: validate remote cache for loops before export
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-16 23:08:42 -07:00
13ea18ad08
vendor containerd (requred for rootless overlayfs on kernel 5.11)
...
Required for rootless overlayfs on kernel 5.11
https://github.com/containerd/containerd/pull/5076
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-03-10 13:51:27 +09:00
29af1397e2
allow skipping empty layer detection on cache export
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-24 01:11:17 -08:00
d039647ea2
cache: remove unused CompareWithParent interface
...
CompareWithParent is not used in the current code.
Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2020-12-18 18:03:14 +09:00
717a4c72ae
Merge pull request #1825 from tonistiigi/prepareRemoteSnapshots-panic-fix
...
cache: handle nil result from flightcontrol
2020-11-25 15:27:04 -08:00
0bd43aa1dc
Check remote snapshot availability only for stargz
...
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-11-21 15:05:12 +09:00
54908ec03f
cache: handle nil result from flightcontrol
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-20 20:35:26 -08:00
a846452fa1
Merge pull request #1791 from tonistiigi/push-pull-retry
...
add retry handlers to push/pull
2020-11-16 13:28:07 +09:00
cf8babde54
progress: refactor logger handler to better reuse
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-15 13:33:58 -08:00
fad612e748
Allow worker to override GetRemote(), needed in moby integration
...
This fixes panics in the moby integration of buildkit when using cache.
Panics come from nil Differ in computeBlobChain which is called in GetRemote().
GetRemote() got moved from Worker to ImmutableRef during the lazy-pull refactor.
However, the ability to easily override GetRemote() got lost with that refactor.
This patch attempts to allow for the override while keeping changes minimal.
Signed-off-by: Tibor Vass <tibor@docker.com>
2020-11-13 21:30:35 +00:00
ce283f39b5
Merge pull request #1773 from tonistiigi/preprocess
...
solver: add input preprocess capability
2020-11-12 21:07:00 -08:00
9dabe0545f
Fixes needed to vendor buildkit into moby
...
Signed-off-by: Tibor Vass <tibor@docker.com>
2020-11-13 01:51:55 +00:00
aa29e77294
fix import cycle in integration test pulls
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-11 21:21:08 -08:00
e86356ddf8
solver: add input preprocess capability
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-03 14:49:44 -08:00
e3b05289d8
add session injection to remote loading
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-02 22:20:44 -08:00
fbe1346912
replace github.com/docker/docker/pkg/locker with moby/locker v1.0.1
...
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-30 09:25:05 +01:00
8adf951bc7
remotecache: allow oci mediatypes on exporting to registry
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-23 00:20:59 -07:00
c8b8d6ce63
exptypes: define empty gz layer digest
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-18 23:39:23 -07:00
85dd12d875
remotecache: don’t export cache for empty layers
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-18 23:39:23 -07:00
4f3fb6155a
Merge pull request #1659 from tonistiigi/loop-export-fix
...
remotecache: validate out broken links from loops
2020-10-18 22:01:54 -07:00
c95dd17ffe
remotecache: validate out broken links from loops
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-08-30 18:59:15 -07:00
c975424deb
Enable to use remote snapshots for refs
...
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-08-24 08:44:00 +09:00
b474dbf55f
resolver: clean up unused resolver pool
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-08-13 17:27:18 -07:00
f1cd79bf65
resolver: add better pooling and custom authenticator
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-08-13 17:27:11 -07:00
1b30fd146b
cache: Remove ImageRef from DescHandlers
...
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
cdcf49fd18
Fix progress in schema1 pull.
...
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
55cbd19dec
Add support for lazily-pulled blobs in cache manager.
...
This allows the layers of images to only be pulled if/once they are actually
required.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
6c824d68f1
Use filesystem-supplied modes for checksumming content
...
The existing code was spuriously failing unit tests, because it was
stripping non-permission bits off the file mode.
On examination, the behaviour was inherited from similar code in
moby/moby, which was intended to affect the resulting filemodes inside
a Linux-based container built from Windows. However, this code path does
have that effect in BuildKit as we are only calculating checksums here.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-29 02:50:24 +10:00
46995ec794
Don't silently ignore failed wildcard expansion
...
This was letting mount-related errors slip by, and instead returning
an empty hash.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-29 02:50:20 +10:00
b2d3473cba
Skip tests on Windows that attempt to bind-mount read-only
...
Containerd's mounter doesn't yet support bind-mounts on Windows.
BuildKit short-cuts this for read-write mounts, but not read-only
mounts.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-28 18:07:20 +10:00
a67af5b10c
Merge pull request #1581 from alexcb/tar-exporter-with-socket-copy-reproduce-error
...
fix socket handling during copy
2020-07-23 12:07:15 -07:00
5382a2056e
Treat unix sockets as regular files
...
This fix is similar to the fix in #1144 ; but was hit in a different code
path.
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-07-23 11:25:48 -07:00
5da4a40ae8
lint: enable more linters
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:36 -07:00
Tõnis Tiigi
ktock
Aaron Lehmann
Akihiro Suda
Tibor Vass
Tibor Vass
Sebastiaan van Stijn
Erik Sipsma
Paul "TBBle" Hampson
Alex Couture-Beil