Now, when a merge or diff ref is unlazied, the progress will show up
under the vertex for the merge/diff ref. Additionally, any ancestors of
the op that also need to be unlazied as part of unlazying the merge/diff
will show status updates under its vertex in the progress.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
switch to using newer MatchesUsingParentResults methods which were
introduced in https://github.com/moby/moby/pull/43037
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
This consists of just the base MergeOp with support for merging LLB
results that include deletions using hardlinks as the efficient path
and copies as fallback.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This is mostly just preparation for merge-op. The existing
Extract method is updated to be usable for unlazying any type of refs
rather than just lazy blobs. The way views are created is simplified and
centralized in one location.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
won't make sense soon due to other potential changes. For example,
there can now be multiple blobs associated with a ref (for different
compression types), so the fact that you could access the "Blob"
field from the Info method on Ref incorrectly implied there was just
a single blob for the ref. This is on top of the fact that there is
no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
Previously, many parts of the code outside the cache package could
obtain the bolt storage item for any ref and read/write it directly.
This made it hard to understand what fields are used and when. Now,
the Metadata method has been removed from the Ref interface and
replaced with getters+setters for metadata fields we want to expose
outside the package, which makes it much easier to track and
understand. Similar changes have been made to the metadata search
interface.
3. Use a consistent getter+setter interface for metadata, replacing
the mix of interfaces like Metadata(), Size(), Info() and other
inconsistencies.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This is important for two reasons:
1) Keeps caching logic consistent with recent fsutil changes to use
these functions (also vendored here).
2) Allows us to move forward with removal of the original buggy Matches
implementation in moby/moby.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
As discussed in #2300, includedPaths does not resolve symlinks when
looking up the source path in the prefix tree. If the user requests a
path that involves symlinks (for example, /a/foo when a symlink /a -> /b
exists), includedPaths will not find it, and will expect nothing to be
copied. This does not match the actual copy behavior implemented in
fsutil, which will follow symlinks in prefix components of a given path,
so it can end up caching an empty result even though the copy will
produce a non-empty result, which is quite bad.
To fix this, use getFollowLinks to resolve the path before walking it.
In the wildcard case, this is done to the non-wildcard prefix of the
path (if any), which matches the behavior in fsutil.
Fixes the repro case here:
https://gist.github.com/aaronlehmann/64054c9a2cff0d27e200cc107bba3d69Fixes#2300
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
The transformation to rootedPatterns seems very wrong and inconsistent
with what the copy logic did. Change it to match the copy logic, and add
more testing.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
This adds a little extra testing around ** patterns, and adds a
(currently skipped) test for copying directories under symlinks (#2300).
It removes an extra call to `filepath.FromSlash` in `shouldIncludePath`
and an unused argument to that function.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
I noticed this when building a Dockerfile that failed because a file didn't
exist, so went through error messages that looked like they had a duplicate
"not found" in the output;
[+] Building 0.9s (6/9)
=> [internal] load build definition from Dockerfile 0.2s
=> => transferring dockerfile: 306B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/alpine:latest 0.0s
=> CACHED [1/5] FROM docker.io/library/alpine 0.0s
=> [internal] load build context 0.6s
=> => transferring context: 701B 0.5s
=> ERROR [2/5] ADD no-such-file.txt / 0.0s
------
> [2/5] ADD no-such-file.txt /:
------
failed to compute cache key: "/no-such-file.txt" not found: not found
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
While we generally ignore the basename in this layer, for wildcards
there in no other place to add the basename to the checksum as they
can not be resolved earlier. Before the basename that was in the
checksum was the wildcard itself, so if the wildcard remained same,
content remained same but the file where wildcard pointed to was
renamed, the cache was not invalidated.
Unfortunately, this change breaks cache for all copy commands that
use a wildcard.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
The existing code was spuriously failing unit tests, because it was
stripping non-permission bits off the file mode.
On examination, the behaviour was inherited from similar code in
moby/moby, which was intended to affect the resulting filemodes inside
a Linux-based container built from Windows. However, this code path does
have that effect in BuildKit as we are only calculating checksums here.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Containerd's mounter doesn't yet support bind-mounts on Windows.
BuildKit short-cuts this for read-write mounts, but not read-only
mounts.
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
Kohei Tokunaga
Erik Sipsma
Tonis Tiigi
Alex Couture-Beil
CrazyMax
Aaron Lehmann
Akihiro Suda
Sebastiaan van Stijn
Paul "TBBle" Hampson