e8dbd3aeb4
Merge pull request #302 from ijc/improved-schema1-support
...
Improved schema1 support
2018-03-07 09:41:34 -08:00
ae0293c4bf
Add an integration test for pulling schema1 images
...
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-07 15:56:22 +00:00
18280e73cc
Merge pull request #289 from ijc/allow-local-image-for-gateway-fe
...
Use local images if pull fails
2018-03-06 09:59:17 -08:00
ac5091c10b
Use local images if pull fails
...
When using `--frontend=gateway.v0` in order to use a tagged image as the
frontend it is currently required to push the tag to a registry, so that
buildkit can pull it. One can use the `image@sha256:digest` form but this is
inconvenient for the development cycle.
Instead, introduce a proxy `Resolver` which if the pull fails checks the local
image store (if there is one, which is worker dependent) for a suitable image
and if so uses that.
With this I can now:
buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --exporter=image --exporter-opt=name=docker.io/ijc25/fe:dev
builtctl build --frontend=gateway.v0 --frontend-opt=source=ijc25/fe:dev --exporter=image --exporter-opt=name=hello-world:dev
Without any push to a registry.
I am using the containerd worker.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-06 12:19:34 +00:00
b7632ad036
Correct content-store management after schema1 conversion
...
The previous code had 3 issues:
- The original schema 1 manifest remained in `ongoing.added` but was not added
to the content store by the conversion. #296 tried to address this in an
incorrect way and was reverted by the previous commit. Instead switch in the
schema2 manifest, which was otherwise missed.
- Empty layers in the schema 1 manifest are not propagated to the schema 2
version, meaning they are not referenced by anything and hence are never
freed up and therefore leak
- The new schema 2 image config is generated by the converter and not fetched,
so it never passes through the `HandlerFunc` which adds things to
`ongoing.added`.
Two address these last two bullets when processing a schema 1 image walk over
the converted image and together with `ongoing.added` produce lists of used and
unused blobs, use the list of used blobs to perform the GC updates and simply
delete everything on the unused list.
Closes : #301 .
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:26:47 +00:00
0ff79ec71c
Revert "Swallow a NotFound error when deleting content, to support schema 1 manifests"
...
This reverts commit 92101de156#296 .
This was the wrong approach a subsequent commit will fix this properly.
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:10:19 +00:00
92101de156
Swallow a NotFound error when deleting content, to support schema 1 manifests
...
The behaviour of `github.com/containerd/containerd/remotes/docker/schema1` is
such that the manifest is not actually in the content store, so attempting to
delete it fails with `NotFound`, which with the `gateway.v0` frontend results
in:
time="2018-02-26T17:01:15Z" level=error msg="fatal error: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found"
panic: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found
It seems safe to just ignore these errors, since `NotFound` is what we wanted
after delete anyway.
This allows support for e.g. `llb.Image("docker.io/docker/whalesay:latest")`
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-02-26 17:08:52 +00:00
7906fb3085
vendor: update containerd to v1.0.2
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
79e346f59a
Add deadcode to the linter
...
it tends to help keeping uneeded code out O:)
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
08373f2381
Clean some small dependencies on docker/docker package
...
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 16:05:24 -08:00
6e40e83d35
Remove net/context dependencies
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
29b72a3912
solver: add some trace points
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
b94704e55c
session: fix session closing and tracing
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
00a5729229
pull: fix content labels after pull
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
b0679c66db
snapshot: clean up snapshot interface
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
367752bf1c
dockerfile: add test for building from git repo
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-17 21:09:20 -08:00
2d3f36d359
source: add sharedkeyhint to local source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:13:15 -08:00
08e1c2990c
dockerfile: add dockerignore support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:03:35 -08:00
ad7c6a905c
source: add function for parsing llb to identifier
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:43:49 -08:00
2db636479c
source: avoid parallel resolve of same config
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 15:02:15 -08:00
2ce418ae7b
git: fix checking out submodules
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-10 18:22:32 -08:00
e4f6a5e752
set more tests to parallel
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-08 18:19:08 -08:00
fdfa6bcb59
vendor: update containerd to 1.0.0
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 23:34:25 -08:00
e0af9d1218
source: add more properties to http source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 12:00:46 -08:00
c9f45dbc22
add unit tests for http source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-02 19:52:15 -08:00
59628fb20f
add http source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-02 19:52:15 -08:00
e3a2aa149e
containerimage: add schema1 pull support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-27 21:42:08 -08:00
6c076f5d6a
cleanup linter warnings
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-16 18:09:35 -08:00
c71a1ca768
bump up containerd (v1.0.0-beta.3)
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-13 05:36:59 +00:00
51d1c4166f
session: expose image auth
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-20 12:56:06 +02:00
5f4b51d0c9
blobmap: fix diffid storing on pull
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:38:08 -07:00
f41283caed
solver: fix shared request progress and cancellation
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 12:45:47 -07:00
05c39612d8
dockerfile: resolve to digest ref in frontend
...
Less hacky than keeping the cached resolver.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 15:49:37 -07:00
b36df4738a
frontend: add more commands to dockerfile
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 18:09:28 -07:00
024fd0afaa
source: fix multi-platform pull
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 13:57:40 -07:00
279940b60d
vendor containerd d1e11f17ec7b325f89608dd46c128300b8727d50
...
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-09-14 19:06:31 +00:00
6d335c3c86
imagesource: share config resolver with frontend
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-05 13:11:27 -07:00
2d290db465
frontend: base of dockerfile frontend
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-31 23:27:25 -07:00
8738929b8c
solver: implement content based cache support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-08-04 10:03:26 -07:00
39b9dec52f
Merge pull request #87 from tonistiigi/snapshot-metadata
...
cache: add more metadata to snapshots
2017-07-27 09:36:10 +09:00
5ecab5e142
Merge pull request #11 from dmcgowan/containerd-worker
...
Containerd worker
2017-07-26 14:36:51 -07:00
38ac00090c
cache: add more metadata values to snapshots
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-26 11:28:20 -07:00
9ce5053a8b
Update containerd version
...
Fix for logrus rename, use fork until fixed in moby.
Removed unused tar stream.
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-07-26 11:07:45 -07:00
93757aef7f
nested llb builds support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-24 18:09:16 -07:00
f0424c367f
Merge pull request #80 from tonistiigi/local-source-progress
...
source: add progress to local sources
2017-07-21 10:28:50 -07:00
09d7a2b9fa
source: add progress to local sources
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-21 10:08:45 -07:00
492c631e05
Merge pull request #79 from tonistiigi/retain-immutable-refs
...
solver: avoid removing references required for future cache
2017-07-21 16:47:52 +09:00
cb168bc954
source: add multiple dir support for local source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-20 21:17:23 -07:00
8b09e89dbb
solver: avoid removing references required for future cache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-20 17:05:37 -07:00
2ef0971822
cache: remove unused branches
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-19 19:02:58 -07:00
ff951eecd9
llb: add readonly mounts support
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-18 16:07:37 -07:00
eac79f7c7e
cache: refactor reference reuse and caching
...
Replaces previous mutable.Freeze logic with
commits that can live together with mutable data.
Finalize method is added if the implementation
needs to make sure that the immutable ref is
flushed to the driver. Refs are automaitcally
finalized when writable layers are created on
top of them.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-18 16:01:01 -07:00
2e48745f57
session: remove confusing uuid variable
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-17 16:08:26 -07:00
6896b5c414
Merge pull request #71 from tonistiigi/local-session
...
source: add local file source fundamentals
2017-07-13 15:56:19 +09:00
5c2e675d18
source: add local file source fundamentals
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-12 23:23:20 -07:00
9e9df712ec
Updates for containerd alpha0
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-12 23:15:12 -07:00
fa7ee2e2a8
source: add keepgitdir option for git source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 22:19:05 -07:00
b65989a5e4
source: add git source
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 22:19:05 -07:00
25ba9d72de
solver: first pass of instruction cache
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-06 20:42:17 -07:00
c9e2493f06
source: update interface for cache calls
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-06 12:12:17 -07:00
f0c14879c7
solver: remove llb dependency
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-05 22:42:33 -07:00
1f5addddac
vendor: update containerd
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-01 08:37:36 -07:00
b573060eea
project: remove poc label
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-23 09:37:36 -07:00
bd32090372
progress: temp fix for context canceling
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-22 12:14:39 -07:00
4ed7fa3280
progress: refactor progress IDs to items
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-19 14:32:50 -07:00
8e7d0904ce
client: handle status log in client
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-16 16:08:53 -07:00
4b41158252
solver: control vertexstatus for pull
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-15 16:08:45 -07:00
94d62fd432
Add context to functions which call snapshotter
...
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-13 16:10:31 -07:00
5a0f803917
imagesource: fill blob map on pull
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-31 15:36:04 -07:00
fdde46f7ff
cachemanager: check pulled snapshot
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-30 14:17:04 -07:00
ae05d02379
Renames and refactoring
...
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-26 23:12:13 -07:00
Tõnis Tiigi
Ian Campbell
Vincent Demeester
Akihiro Suda
Akihiro Suda
Derek McGowan