65388edc6c
exporter: support compression for layer blob data
...
Allow user to choose the compression type for layer data. Gzip is
default compression for layer exporter, which consume more Cpu resources
and take long time to export. With compression option, user can use
nocompressed option to export to save time. And future, zstd is one new
option for end-user.
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-12-09 23:51:38 +08:00
1955890445
cache: fix possible concurrent maps write on parent release
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-11-12 13:41:50 -08:00
565deba342
blobs: allow alternative compare-with-parent diff
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-11-06 17:34:42 -08:00
1d867ebdb1
cache: follow-up updates to storage leases
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-21 17:19:44 -07:00
339d4b2fef
leaseutil: mark temporary leases with timestamps
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:35:50 -07:00
31a9aeea88
cache: add migration flow to new lease based format
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:35:49 -07:00
d35d2c1c94
blobs: ensure lease is held on GetDiffPairs
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:35:48 -07:00
c6d0a57d51
cache: correct parent releasing
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:35:46 -07:00
3cedd2c0e5
cache: fix image share checker ref leak
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:34:53 -07:00
08194783e0
cache: update unit tests to new cache manager constructor
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:34:50 -07:00
c005fb08eb
cache: better error management
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:33:02 -07:00
6be136db6e
cleanup unused code
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:33:01 -07:00
a9209caa93
clean up temporary leases on startup after crash
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:57 -07:00
7390a41a5b
cache: track views with a lease
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:55 -07:00
688e2c2272
cache: update components to new lease based cache manager
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:32:04 -07:00
b9ec2bc4d4
cache: update manager to use leases instead of root labels
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-10-16 10:31:51 -07:00
481d39c176
add cross-repo push support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-22 17:25:13 -07:00
588c73e1e4
Merge pull request #1140 from tonistiigi/fix-symlink-parent
...
contenthash: fix symlinks with unscanned parents
2019-08-19 13:56:08 -07:00
3504fae6f2
contenthash: fix symlinks with unscanned parents
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-16 15:03:14 -07:00
a0dead0809
fix possible double release on mountable
...
Refactor the interface to avoid such issues in the future.
BuildKit own mounts are stateless and not affected but
a different mountable implementation could get confused.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-16 08:03:16 -07:00
0d45d77756
remotecache: fix concurrent write
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-14 17:02:16 -07:00
c114e438f5
remotecache: fix inline cache in manifest lists
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-13 16:25:08 -07:00
3c3dac0a5b
remotecache: other fix to record sorting
...
Signed-off-by: Mark Gordon <msg555@gmail.com>
2019-08-08 12:33:31 -04:00
27990cc6a3
remotecache: fix records sorting
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-08-06 12:25:15 -07:00
4b7741dc29
Merge pull request #1100 from tonistiigi/overlay-readonly
...
cache: force overlay readonly by removing upperdir
2019-07-30 15:11:48 -07:00
92c3fd477b
worker: remove unreferenced cache mount after release
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-29 15:41:49 -07:00
7beb0bb724
cache: force overlay readonly by removing upperdir
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-29 11:00:19 -07:00
7b1bae7a42
solver: support no-cache for exec cache mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-07-26 14:42:32 -07:00
e2dcafa5ca
Removing wrapf for review
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-07 12:04:47 -07:00
b087d06adb
cache: error tracing on cache importer
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-06 21:15:21 -07:00
ed6bbb2bf5
cache: add more error tracing
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-06-06 21:15:21 -07:00
5aa12caefe
Merge pull request #1023 from tonistiigi/local-cancel
...
local: handle canceled uploads
2019-05-29 18:33:54 -07:00
669e65e6fe
local: handle canceled uploads
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 14:06:17 -07:00
6defbb2356
cache: fix releasing parent on error
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-27 21:58:25 -07:00
ffad8ada79
exporter: support unpack opt for image exporter
...
It is enhancement which allows to unpack image into containerd
snapshotter storage by `--output type=image,<.>=<.>,unpack=true`.
In order to support this feature, we needs to extend the Snapshotter
witwh `Name() string` function. Because we needs to set gc label for
snapshotter which need snapshotter name.
fix : #908
Signed-off-by: Wei Fu <fuweid89@gmail.com>
2019-04-24 21:24:06 +08:00
894da35fed
fix unit tests for idmapping support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Tibor Vass <tibor@docker.com>
2019-04-02 20:35:18 -07:00
9f53ea3d78
userns support for sources and executor
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 18:26:04 -07:00
6921dbe0f6
snapshot: base for userns remapping
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-04-02 18:23:46 -07:00
e5b647a1a3
Merge pull request #881 from tonistiigi/contenthash-hardlink
...
contenthash: fix hardlinks checksum on handlechange
2019-03-20 16:13:12 -07:00
7f4d96c648
contenthash: add hardlinks testcase
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-18 18:58:25 -07:00
0d17ac323e
fileop: updates with new fsutil copy pkg
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 17:49:58 -07:00
4ffd79735b
fileop: connect with contenthash
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:31:00 -07:00
81a5fa5a2e
llbsolver: fileop implementation
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 16:22:35 -07:00
23e1cd35d9
contenthash: fix hardlinks checksum on handlechange
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-15 15:47:08 -07:00
d9cdaf8003
remotecache: allow uncompressed digest in inline export
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-13 19:19:59 -07:00
5c9f7b8ff0
buildctl: new CLI ("Option C+")
...
See https://github.com/moby/buildkit/pull/807#issuecomment-468146089
Close #774
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-03-06 13:20:21 +09:00
632797cef0
vendor: udpate fsutil to b4281fa
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-05 17:34:01 -08:00
54bd828a10
contenthash: fix symlink parent reference
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-04 18:10:40 -08:00
8bcd59a546
contenthash: add nofollow support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-01 23:41:05 -08:00
1814a0163a
contenthash: add wildcard support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-03-01 17:39:34 -08:00
7d41912f36
remotecache: keep layer description
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
5be7870549
load parent results on combined cachemanager
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
d77d950053
remotecache: walklinks support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
d514afdb20
remotecache: support createdat and fix order
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
97eff70c5e
inline remote cache support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-21 19:39:15 -08:00
ac4037a1d9
solver: keep original cache creation date when copying between stores
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:34 -08:00
1ec543e384
solver: exclude randomized cache keys from exporter
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-01-18 11:19:25 -08:00
6e7617e889
support local cache exporter and importer
...
Export:
$ buildctl build ... --export-cache type=local,store=/path/to/output-dir
Import:
$ buildctl build ... --import-cache type=local,store=/path/to/input-dir
Impact on CLI:
* Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max`
* New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max`
Impact on API:
* New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
* While new API can be used for `registry` caches, the client continues
to use the legacy API for `registry` caches to ensure compatibility with
old daemons.
* To import `local` caches with a frontend, the frontend needs to support
a new frontend opt `cache-imports`.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-01-19 02:38:26 +09:00
5fa4642e1d
contenthash: clean readlink value
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-30 22:14:11 -07:00
89e8097d4c
fix setting uncompressed label on content
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-10-08 23:11:25 -07:00
b4454a169f
implement `gateway.Reference.StatFile` method
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:45 +01:00
df76f6d380
Move cache/fsutil into a separate cache/util subpackage
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
b30ee91a53
implement `gateway.Reference.ReadDir` method
...
The alternative is a helper container running e.g. find, capturing the output
to a scratch to be read back (with ref.ReadFile) and parsed.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
531ee8d4ea
cache: refactor snapshot management out of ReadFile
...
Allowing for other operations within a mount.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-10-03 12:16:44 +01:00
8b69292087
Bump github.com/tonistiigi/fsutil dependency to f567071bed24
...
7e391b0e78...f567071bed
2018-10-03 11:46:35 +01:00
9bf5431b75
contenthash: fix issues on symlinks on parent paths
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-21 13:16:52 -07:00
a93a96b0e3
contenthash: use require in all tests
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-21 11:19:15 -07:00
cb0d51cca3
Merge pull request #630 from tonistiigi/cleanup
...
golint and misspell cleanup
2018-09-19 12:01:45 -07:00
ae321cc9cf
cache: remove the unused old gc primitives
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-18 22:06:47 -07:00
aa73951164
Merge pull request #631 from jhowardmsft/boltdb
...
Revendoring to move boltdb to bbolt
2018-09-19 13:59:08 +09:00
2de2c04c8e
Revendoring to move boltdb to bbolt
...
Signed-off-by: John Howard <jhoward@microsoft.com>
2018-09-18 11:18:08 -07:00
80f959c7ef
Merge pull request #626 from AkihiroSuda/update-golang
...
update Go to 1.11
2018-09-17 10:35:26 -07:00
bf571a519e
update Go to 1.11
...
For consistency with Moby (moby/moby#37358 )
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-09-15 14:25:11 +09:00
757795e556
cache: fix usage counters
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-14 18:44:44 -07:00
ecd4a22546
resolver: add config support for mirrors/plainhttp
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 18:44:45 -07:00
079c747c4e
ontrol: support calling automatic GC
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-04 18:13:29 -07:00
acab0f6873
cache: correct prune size calculation cache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-31 12:00:13 -07:00
28fec2b9cb
cache: fix possible prune deadlock
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-31 11:59:53 -07:00
1a37ee3bf4
cache: add gc style pruning support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-08-06 21:45:39 -07:00
d1ad6961e3
cache: detect refs shared with image store
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-27 14:08:50 -07:00
97fa2b527a
cache: add all option to prune
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:55:05 -07:00
489246dd28
cache: support for internal/frontend record type
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
57b96a0ee5
cache: add record type field to usage record
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
87f43efda5
cache: implementation for prune filtering
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 19:35:49 -07:00
0ba24f2a90
cache: implement filter for diskusage
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 17:20:59 -07:00
56fa35725a
vendor: update containerd
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-25 16:37:42 +09:00
bc765861be
diff: implement windows layer support for linux
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 16:33:21 -07:00
57006ea333
cache: avoid commit on finalize when possible
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-11 16:21:27 -07:00
80d2f820f9
decopule cache/remotecache from registry
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-07-04 17:38:09 +09:00
618f34a1f9
Support creating and pushing OCI types
...
Adds image and oci exporter option "oci-mediatypes"
Ensures that the images created in the content store
have the correct type which matches the manifest.
Sets the correct media type on the descriptor in push from
reading the type specified in the manifest.
Removes use of distribution manifest packages.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2018-06-28 11:23:35 -07:00
ccbf185006
llbsolver: allow sharing modes for cache mounts
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-19 15:57:20 -07:00
6ce604740f
vendor: replace locker pkg for legal
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 11:00:37 -07:00
47a00b84fe
update containerd library
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-06-08 21:44:17 +09:00
e33d2ac17c
contenthash: normalize symlink mode
...
Mode is not implemented in platform compatible way
so normalize when calculating the checksum.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-01 23:34:51 -07:00
6b25a56906
dockerfile: support building from a Dockerfile URL
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-30 13:20:27 -07:00
71f5e6dd21
solver: rename solver-next to solver
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
a66885e70c
cache: reenable last used metadata
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
3040783254
cache: helper methods for immutableref
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
f00ba89a30
llbsolver: support detecting existing remotes
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
f617d38aaa
rename cacheimport to remotecache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Wei Fu
Tonis Tiigi
Tibor Vass
Mark Gordon
Akihiro Suda
Ian Campbell
Akihiro Suda
John Howard
Derek McGowan