Instead of doing a direct string comparison we should
use `platforms.Only` so that we can also detect the
variants that are compatible but don’t match directly.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Before this, there could be crash during a call to finalize a ref that
occured after the snapshot was committed but before committing the
metadata that indicates the immutable ref no longer had an
equalMutable. This resulted in a situation where any future calls to
finalize that ref would fail.
Now, if that situation happens, the cache will notice when it's
initially loaded that the ref has an equalMutable that's missing its
snapshot and that its own snapshot exists. It will then just use the
correctly committed snapshot and clear the equalMutable field.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This adds test coverage for ensuring the readonly parameter is honored
as expected in the ref Mount methods. There was a regression introduced
during #2335 that went unnoticed until identified and fixed in #2562.
This test coverage should help prevent similar regressions in the
future.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This change enables inline cache to work as expected with MergeOp by
supporting a new type of result, DiffResult, which enables results to be
specified as a specific ordered set of layers inside an image.
Previously, results could only be specified with a singe layer index,
which meant that they had to start at the image's base layer and end at
that index. That meant that merge inputs couldn't be specified as they
are often a subset of the image layers that don't begin at the base.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Before this, you could return worker ref results from ops that have nil
refs but once they were attempted to be used, various nil exceptions
would get hit. Now, those cases should be handled.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
This allows you to create refs that are single layers representing the
diff between any two arbitrary refs. The primary use case for this is
to allows users to extract the changes created by ops like Exec and
rebase them elsewhere through MergeOp. However, there is no restriction
on the inputs to DiffOp and the resulting ref's layer is simply the
layer created by running the differ on the two inputs refs
(specifically, the same differ used during exports).
A Diff ref can be mounted by itself, in which case it is defined as the
result of applying the diff to Scratch. Most use cases though will use
Diff refs as the input to a MergeOp, in which case the diff is just
applied on top of the lower merge inputs, as was the case before.
In cases like Diff(A, A->B->C) (i.e. cases where the diff is between two
refs where the lower is an ancestor of upper), the diff will be defined
as the layers separating the two refs. In other cases, the diff is just
a single layer, not re-used from the inputs, representing the diff
between the two refs (which can be defined as the layer "Diff(A,B)" that
satisfies "Merge(A, Diff(A,B)) == B").
Note that there is technically a meaningful difference between the
"unmerge" behavior of extracting the layers separating diffs and the
"simple diff" of just running the differ on the two refs. Namely, in the
case where there are "intermediate deletes" (i.e. deletes that only
exist in layers between A and B but not between A and B by themselves),
then the simple diff and unmerge can create different results when
plugged into a MergeOp. This is due to the fact that intermediate
deletes will apply to the merge when using the unmerge behavior, but not
when using the simple diff. This is on top of the fact that the simple
diff inherently has a "flattening" behavior where multiple layers are
squashed into a single one.
So, in the case where lower is an ancestor of upper, we choose to follow
the unmerge behavior, but it's possible users may prefer the simple diff
behavior. As of right now, they won't be able to do so, but if needed we
can add the ability to choose which behavior is followed in the future.
This could be done through a flag provided to DiffOp or possibly by
adapting llb.Copy to support this type of behavior with the same
efficiency as DiffOp.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
If the ref is invalid we are seeing a panic from `umask-git` because
the error type is not always a unix.WaitStatus:
```
#1 0.227 fatal: Not a valid object name 000111222333444555666777888999aaabbbcccd^{commit}
#1 0.229 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.229
#1 0.229 goroutine 1 [running]:
#1 0.229 github.com/moby/buildkit/source/git.gitMain()
#1 0.229 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.229 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.229 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.229 main.init.0()
#1 0.229 /src/cmd/buildkitd/main.go:76 +0xf6
#1 0.633 fatal: reference is not a tree: 000111222333444555666777888999aaabbbcccd
#1 0.635 panic: interface conversion: interface {} is syscall.WaitStatus, not unix.WaitStatus
#1 0.635
#1 0.635 goroutine 1 [running]:
#1 0.635 github.com/moby/buildkit/source/git.gitMain()
#1 0.635 /src/source/git/gitsource_unix.go:66 +0x27d
#1 0.635 github.com/docker/docker/pkg/reexec.Init(...)
#1 0.635 /src/vendor/github.com/docker/docker/pkg/reexec/reexec.go:26
#1 0.635 main.init.0()
#1 0.635 /src/cmd/buildkitd/main.go:76 +0xf6
```
This is from trying to solve:
```
llb.Git("https://github.com/moby/buildkit.git", "000111222333444555666777888999aaabbbcccd")
```
Signed-off-by: coryb <cbennett@netflix.com>
Stages and implicit stages from image names can be
redefined with build options.
This enables using more that one source directory
and reusing results from other builds. This can also
be used to use a local image from other build without
including a registry.
Contexts need to be defined as `context:name=` frontend
options. The value can be image, git repository,
URL, local directory or a frontend input.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
It looks like the intent is to keep track of the index in 'ret' where a
destination was written, but that's not what the current code is doing.
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
HTTP/2(RFC7540) defines :authority pseudo header includes the authority portion
of target URI but it must not include userinfo part (i.e. url.Host).
However, when TLS certificate specified, grpc-go requires it must match
with its servername specified for certificate validation.
Signed-off-by: Shingo Omura <everpeace@gmail.com>
This breaks the giant blob that was the diffApply function into two
separate parts, a differ and an applier, which results in more modular
code that should be easier to follow and easier to make any future
updates to. For example, if we want to optimize by allowing differ and
applier to run in parallel in the future, that's straightforward now.
There are also some fixes that weren't needed for MergeOp, but will be
for DiffOp, such as correctly handling the case where a deletion is
applied that is under parent directories which don't exist yet (the
correct behavior is, surprisingly, to create the parent directories as
that is what the image import/export code ends up doing).
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
Tonis Tiigi
Akihiro Suda
Erik Sipsma
Kohei Tokunaga
Shiwei Zhang
Edgar Lee
coryb
CrazyMax
Christian Weichel
Hans van den Bogert
Aaron Lehmann
Shingo Omura