Commit Graph

412 Commits (59d2f76e5ea1e6f696e4648691bde5d0edb6f781)

Author SHA1 Message Date
Akihiro Suda 817955dc8d
Merge pull request #2116 from tonistiigi/git-subdir
git: support subdir component
2021-06-08 14:27:50 +09:00
Tõnis Tiigi 8df5671844
Merge pull request #2082 from aaronlehmann/copy-include-exclude
Add IncludePatterns and ExcludePatterns options for Copy
2021-06-01 12:47:13 -07:00
Tõnis Tiigi e51dbe9fae
Merge pull request #2123 from hinshun/abs-mountpoint
Transform relative mountpoints for exec mounts in the executor
2021-05-26 19:37:52 -07:00
Aaron Lehmann b27fc6169b Combine include/exclude caps into a single cap
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
Aaron Lehmann 626744c9e7 Cache awareness of IncludePatterns/ExcludePatterns
Consider IncludePatterns and ExcludePattern when calculating content
hashes.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
Aaron Lehmann b545c281c1 Add capabilities for copy include/exclude patterns
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
Aaron Lehmann 7660af2f1d Add IncludePatterns and ExcludePatterns options for Copy
Allow include and exclude patterns to be specified for the "copy" op,
similarly to "local".

Depends on https://github.com/tonistiigi/fsutil/pull/101

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-05-26 13:48:03 -07:00
Edgar Lee 9aeed7b193 Transform relative mountpoints for exec mounts in the executor
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2021-05-20 15:17:27 -07:00
Tonis Tiigi 3bdb6b5159 git: support subdir component
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-05-19 22:13:04 -07:00
Vlad A. Ionescu 60d38f972c Move config under worker config. Add new integration test to GHA matrix.
Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
2021-05-13 14:55:36 +03:00
Vlad A. Ionescu b3cf7c43cf Switch to Acquire API.
Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
2021-05-12 13:41:33 +03:00
Vlad A. Ionescu 489e17aea9 Add a configuration item to limit parallelism.
Signed-off-by: Vlad A. Ionescu <vladaionescu@users.noreply.github.com>
2021-05-10 15:47:59 +03:00
Tõnis Tiigi f113b49307
Merge pull request #2086 from thaJeztah/add_all_proxy
Add "ALL_PROXY" to list of default args / proxy env-vars
2021-04-26 09:09:44 -07:00
Sebastiaan van Stijn a743d4ba02
Add "ALL_PROXY" to list of default args / proxy env-vars
Relates to a82fff6377/docs/packages.md (proxies)

> (..) the first four of these are the standard built-in build-arg options
> available for `docker build`
> (..) The last, `all_proxy`, is a standard var used for socks proxying. Since
> it is not built into `docker build`, if you want to use it, you will need to
> add the following line to the dockerfile:
>
>     ARG all_proxy

Given the we support all other commonly known proxy env-vars by default, it makes
sense to add this one as well.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-04-24 18:19:02 +02:00
Aaron Lehmann 61bb15a47f Add hack to preserve Dockerfile RUN cache compatibility after mount cache bugfix
Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-04-14 08:53:08 -07:00
Aaron Lehmann 0e106e6967 Fix missing mounts in execOp cache map
A bug in cloneExecOp prevented mounts from being included in the digest
computed for the execOp cache map. This could lead to an exec being
wrongly cached when a different mount was used for a subsequent
execution.

Repro case:
https://gist.github.com/aaronlehmann/cfeaefc028df8127fb85b9b5f9125f2d

In this example, pass2 should generate an empty diff because the /from
and /to mounts are the same busybox image. But before the fix, it uses
the cached result from pass1 (with different mounts) instead.

Signed-off-by: Aaron Lehmann <alehmann@netflix.com>
2021-04-13 22:04:28 -07:00
Tonis Tiigi 86154daec1 solver: check for input index existence
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-04-07 22:02:42 -07:00
Sebastiaan van Stijn 292685db37
solve: remove duplicate filepath.Join
I saw these, and there was no comment in the code explaining if there was a reason
for this; these were added in 2be999ba52, but couldn't
find a mention about these particular lines in the review comments, and I couldn't
think of a special reason for it ':-)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-03-29 13:24:55 +02:00
Tonis Tiigi bee77d81b9 solver: check for edge dependency before merging
In some cases edges that depend on each other
can have identical cache keys. This happens for
example when empty layers are optimized out by
the differ between identical commands. We need to
detect this case and avoid merge.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-03-18 19:55:35 -07:00
Akihiro Suda 13ea18ad08
vendor containerd (requred for rootless overlayfs on kernel 5.11)
Required for rootless overlayfs on kernel 5.11
https://github.com/containerd/containerd/pull/5076

Signed-off-by: Akihiro Suda <akihiro.suda.cz@hco.ntt.co.jp>
2021-03-10 13:51:27 +09:00
Akihiro Suda 19d31737fb
Merge pull request #1963 from tonistiigi/release-count-fixes 2021-02-19 08:22:54 +09:00
Tonis Tiigi 8d70777537 fileop: fix checksum to contain indexes of inputs
Cache mismatch can happen if fileop switches input indexes
between different actions.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-03 22:17:57 -08:00
Tonis Tiigi 42fb2a8771 avoid double release of same ref released multiple times
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-03 10:53:19 -08:00
Tonis Tiigi b4b0ece384 make sure result refs returned in errors are clone
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-03 10:49:38 -08:00
Tonis Tiigi 3660e5f9c8 don't commit cache mounts on error
Cache mount instances are shared between multiple vertextes/builds
so if one of the cloned instance gets committed reference count
will get corrupted as other parts of the code still see reference as
mountable.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-02 23:20:44 -08:00
Tonis Tiigi dae8f156da update qemu emulators
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-01-21 18:33:00 -08:00
Tõnis Tiigi dd7fe19951
Merge pull request #1830 from coryb/unknown-exit-status
move UnknownExitStatus to executor package from errdefs
2020-12-28 16:35:14 -08:00
Tonis Tiigi 3889ef509c allow nil return from frontend
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-14 17:19:41 -08:00
Tibor Vass bea286f94c
Merge pull request #1871 from tonistiigi/scheduler-fix-for-testing
solver: fix slow-cache status tracking with preprocess
2020-12-11 11:47:45 -08:00
Tonis Tiigi 2f247f0340 solver: fix slow-cache status tracking with preprocess
Preprocess should not be taken into account when calculating
cache-slow state. But it should be used to detect if dep is complete
so preprocess runs before parent's exec function.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-10 22:57:30 -08:00
Tonis Tiigi 1b271e0014 solver: delay before discarding job
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-09 20:53:03 -08:00
Cory Bennett 9d3f55c400 move ExitError from solver/errdefs to frontend/gateway/errdefs
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2020-12-07 01:09:59 +00:00
Tõnis Tiigi d5f179bb79
Merge pull request #1863 from tonistiigi/solver-slowcache
solver: improve slow cache detection logging
2020-12-02 16:47:35 -08:00
Tonis Tiigi 706129170d solver: remove scheduler fix
Asked in review.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-02 16:07:13 -08:00
Tonis Tiigi f38972a8ec mounts: handle optional ssh mount
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-02 14:28:51 -08:00
Tonis Tiigi 6a321dc574 solver: improve slow cache detection and add logging
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-01 22:38:15 -08:00
Tonis Tiigi e46cfab9fb fix invalid cancellation errors on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-01 00:27:31 -08:00
Tonis Tiigi df3a9cad23 make sure execerror is released on cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-25 15:50:10 -08:00
Edgar Lee 372df78cc8 Guarantee err results are released when result proxy is released
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-25 11:26:53 -08:00
Sebastiaan van Stijn 67bcfe699a
copy containerd.UnknownExitStatus to local const
Copy this const to a local constant to prevent importing the containerd
client in the front-end.

For consistency, I also updated the executor code to use the same const,
although not strictly needed.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-11-18 00:11:09 +01:00
Tõnis Tiigi 5e5f527e06
Merge pull request #1732 from hinshun/exec-error
Allow gateway exec-ing into a failed solve with an exec op
2020-11-16 19:13:11 -08:00
Edgar Lee fa8a02c861 Fix optional cast for WithOp when unit testing
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-16 13:37:30 -08:00
Edgar Lee 1240dd7795 Return committed readonly inputs and actives in exec error in MountIDs
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-16 13:21:23 -08:00
Tõnis Tiigi 8dc3a54a78
Merge pull request #1782 from alexcb/acb-other-git-user
SSH-based auth for llb.Git operations
2020-11-16 10:53:44 -08:00
Edgar Lee 3ba6cd7bc2 Fix ExecError.EachRef invoking callback with possibly nil solver.Results
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 23:20:41 -08:00
Edgar Lee c33bcd61a2 Rename OutputIDs to MountIDs
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 22:05:13 -08:00
Edgar Lee bcff7baf60 Allow scratch mounts with gateway.RunContainer
- Plumb default worker by adding GetDefault() to frontend.WorkerInfos
- To avoid cyclic dependency, refactor frontend.WorkerInfos to worker.Infos
- Refactor gateway.NewContainer to share code with llbsolver/ops/exec.go

Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 22:05:13 -08:00
Edgar Lee a459eb4927 Return exec error for errors returned before executor
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 22:05:13 -08:00
Edgar Lee 7e1dc9bec1 Refactor to file action indexed outputs
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 22:05:13 -08:00
Edgar Lee 2d23d0cc43 Fix lint and unit tests for fileopsolver
Signed-off-by: Edgar Lee <edgarl@netflix.com>
2020-11-13 22:05:13 -08:00