This change enables inline cache to work as expected with MergeOp by
supporting a new type of result, DiffResult, which enables results to be
specified as a specific ordered set of layers inside an image.
Previously, results could only be specified with a singe layer index,
which meant that they had to start at the image's base layer and end at
that index. That meant that merge inputs couldn't be specified as they
are often a subset of the image layers that don't begin at the base.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, you could return worker ref results from ops that have nil
refs but once they were attempted to be used, various nil exceptions
would get hit. Now, those cases should be handled.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This allows you to create refs that are single layers representing the
diff between any two arbitrary refs. The primary use case for this is
to allows users to extract the changes created by ops like Exec and
rebase them elsewhere through MergeOp. However, there is no restriction
on the inputs to DiffOp and the resulting ref's layer is simply the
layer created by running the differ on the two inputs refs
(specifically, the same differ used during exports).
A Diff ref can be mounted by itself, in which case it is defined as the
result of applying the diff to Scratch. Most use cases though will use
Diff refs as the input to a MergeOp, in which case the diff is just
applied on top of the lower merge inputs, as was the case before.
In cases like Diff(A, A->B->C) (i.e. cases where the diff is between two
refs where the lower is an ancestor of upper), the diff will be defined
as the layers separating the two refs. In other cases, the diff is just
a single layer, not re-used from the inputs, representing the diff
between the two refs (which can be defined as the layer "Diff(A,B)" that
satisfies "Merge(A, Diff(A,B)) == B").
Note that there is technically a meaningful difference between the
"unmerge" behavior of extracting the layers separating diffs and the
"simple diff" of just running the differ on the two refs. Namely, in the
case where there are "intermediate deletes" (i.e. deletes that only
exist in layers between A and B but not between A and B by themselves),
then the simple diff and unmerge can create different results when
plugged into a MergeOp. This is due to the fact that intermediate
deletes will apply to the merge when using the unmerge behavior, but not
when using the simple diff. This is on top of the fact that the simple
diff inherently has a "flattening" behavior where multiple layers are
squashed into a single one.
So, in the case where lower is an ancestor of upper, we choose to follow
the unmerge behavior, but it's possible users may prefer the simple diff
behavior. As of right now, they won't be able to do so, but if needed we
can add the ability to choose which behavior is followed in the future.
This could be done through a flag provided to DiffOp or possibly by
adapting llb.Copy to support this type of behavior with the same
efficiency as DiffOp.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
If the ref is invalid we are seeing a panic from `umask-git` because
the error type is not always a unix.WaitStatus:
```
#1 0.227 fatal: Not a valid object name 000111222333444555666777888999aaabbbcccd^{commit}
#1 0.229 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.229
#1 0.229 goroutine 1 [running]:
#1 0.229 github.com/moby/buildkit/source/git.gitMain()
#1 0.229 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.229 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.229 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.229 main.init.0()
#1 0.229 /src/cmd/buildkitd/main.go:76 +0xf6
#1 0.633 fatal: reference is not a tree: 000111222333444555666777888999aaabbbcccd
#1 0.635 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.635
#1 0.635 goroutine 1 [running]:
#1 0.635 github.com/moby/buildkit/source/git.gitMain()
#1 0.635 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.635 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.635 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.635 main.init.0()
#1 0.635 /src/cmd/buildkitd/main.go:76 +0xf6
```
This is from trying to solve:
```
llb.Git("https://github.com/moby/buildkit.git", "000111222333444555666777888999aaabbbcccd")
```
Signed-off-by: coryb <cbennett@netflix.com>
Stages and implicit stages from image names can be
redefined with build options.
This enables using more that one source directory
and reusing results from other builds. This can also
be used to use a local image from other build without
including a registry.
Contexts need to be defined as `context:name=` frontend
options. The value can be image, git repository,
URL, local directory or a frontend input.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
It looks like the intent is to keep track of the index in 'ret' where a
destination was written, but that's not what the current code is doing.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
HTTP/2(RFC7540) defines :authority pseudo header includes the authority portion
of target URI but it must not include userinfo part (i.e. url.Host).
However, when TLS certificate specified, grpc-go requires it must match
with its servername specified for certificate validation.
Signed-off-by: Shingo Omura <everpeace@gmail.com>
This breaks the giant blob that was the diffApply function into two
separate parts, a differ and an applier, which results in more modular
code that should be easier to follow and easier to make any future
updates to. For example, if we want to optimize by allowing differ and
applier to run in parallel in the future, that's straightforward now.
There are also some fixes that weren't needed for MergeOp, but will be
for DiffOp, such as correctly handling the case where a deletion is
applied that is under parent directories which don't exist yet (the
correct behavior is, surprisingly, to create the parent directories as
that is what the image import/export code ends up doing).
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this change, test cases were running with an env var that forces
the overlay differ to be on even when the native snapshotter was being
used, which resulted in failures. Now, that env var is skipped when
using the native snapshotter.
Additionally, this includes a related change to skip even trying to use
the overlay differ when the native snapshotter is in use. Previously,
the blob creation code first tried to use the overlay differ and then
failed and fell back to the double-walking differ. Now, it just jumps
right to the double-walking differ when the native snapshotter is in
use.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Using an interface instead of a func is more flexible while achieving
the same effect. It allows you to succintly define a large number of
test cases as structs, as is common in table-driven testing.
A helper func is added that converts the existing test funcs into the
interface, so the change is fairly seamless.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
update fsutils to 61a57076b9b065af88eb10f699926d7e8793910c
which is required to pull in moby/moby#43047
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
switch to using newer MatchesUsingParentResults methods which were
introduced in https://github.com/moby/moby/pull/43037
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
When vendored into moby, the local exporter uses a user map that results
in all files being given at most 755 permissions. This change updates
the test to use permissions less than or equal to that to make tests
equivalent whether running w/ dockerd worker or any other type.
Additionally, add assertions that exported images also have the expected
contents, which helps ensures consistency between images created by
dockerd and those created by vanilla buildkit.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Shiwei Zhang
Tõnis Tiigi
Edgar Lee
Erik Sipsma
Akihiro Suda
coryb
Kohei Tokunaga
CrazyMax
Christian Weichel
Hans van den Bogert
Aaron Lehmann
Shingo Omura
Alex Couture-Beil
Csaba Apagyi