Commit Graph

48 Commits (c2a35a5c6fa41c837764f96b7d5d87a930aadf14)

Author SHA1 Message Date
Ian Campbell ef1a79a1c1 Allow caller to specify platform for use with imagemetaresolver.
This needs to be plumbed through to `imageutil.Config` which is also used from
the `containerimage` package. Since that is used in the context of a Worker the
`platforms.Default()` (corresponding to the platform of the worker itself) is
appropriate so apply a default at that level.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-05-24 11:23:35 +01:00
Tonis Tiigi 8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Akihiro Suda d22e5a1721 split puller from source/containerimage into util/pull
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-05-18 06:14:06 +09:00
Tonis Tiigi 9c044db670 llbsolver: update pull for multiple cache keys
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi c0442a1c0d cache: avoid reloading snapshots in gc queue
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-10 15:51:36 -07:00
Tonis Tiigi 566e28c174 snapshot: update mounts to mountable interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-04-27 17:58:24 -07:00
Akihiro Suda 126f91dea7 update containerd (binary: v1.0.3, library: Apr 3, 2018)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-04 17:22:20 +09:00
Tõnis Tiigi e8dbd3aeb4
Merge pull request #302 from ijc/improved-schema1-support
Improved schema1 support
2018-03-07 09:41:34 -08:00
Ian Campbell ae0293c4bf Add an integration test for pulling schema1 images
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-07 15:56:22 +00:00
Tõnis Tiigi 18280e73cc
Merge pull request #289 from ijc/allow-local-image-for-gateway-fe
Use local images if pull fails
2018-03-06 09:59:17 -08:00
Ian Campbell ac5091c10b Use local images if pull fails
When using `--frontend=gateway.v0` in order to use a tagged image as the
frontend it is currently required to push the tag to a registry, so that
buildkit can pull it. One can use the `image@sha256:digest` form but this is
inconvenient for the development cycle.

Instead, introduce a proxy `Resolver` which if the pull fails checks the local
image store (if there is one, which is worker dependent) for a suitable image
and if so uses that.

With this I can now:

    buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --exporter=image --exporter-opt=name=docker.io/ijc25/fe:dev
    builtctl build --frontend=gateway.v0 --frontend-opt=source=ijc25/fe:dev --exporter=image --exporter-opt=name=hello-world:dev

Without any push to a registry.

I am using the containerd worker.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-06 12:19:34 +00:00
Ian Campbell b7632ad036 Correct content-store management after schema1 conversion
The previous code had 3 issues:

- The original schema 1 manifest remained in `ongoing.added` but was not added
  to the content store by the conversion. #296 tried to address this in an
  incorrect way and was reverted by the previous commit. Instead switch in the
  schema2 manifest, which was otherwise missed.
- Empty layers in the schema 1 manifest are not propagated to the schema 2
  version, meaning they are not referenced by anything and hence are never
  freed up and therefore leak
- The new schema 2 image config is generated by the converter and not fetched,
  so it never passes through the `HandlerFunc` which adds things to
  `ongoing.added`.

Two address these last two bullets when processing a schema 1 image walk over
the converted image and together with `ongoing.added` produce lists of used and
unused blobs, use the list of used blobs to perform the GC updates and simply
delete everything on the unused list.

Closes: #301.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:26:47 +00:00
Ian Campbell 0ff79ec71c Revert "Swallow a NotFound error when deleting content, to support schema 1 manifests"
This reverts commit 92101de156 from #296.

This was the wrong approach a subsequent commit will fix this properly.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:10:19 +00:00
Ian Campbell 92101de156 Swallow a NotFound error when deleting content, to support schema 1 manifests
The behaviour of `github.com/containerd/containerd/remotes/docker/schema1` is
such that the manifest is not actually in the content store, so attempting to
delete it fails with `NotFound`, which with the `gateway.v0` frontend results
in:

    time="2018-02-26T17:01:15Z" level=error msg="fatal error: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found"
    panic: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found

It seems safe to just ignore these errors, since `NotFound` is what we wanted
after delete anyway.

This allows support for e.g. `llb.Image("docker.io/docker/whalesay:latest")`

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-02-26 17:08:52 +00:00
Tonis Tiigi 7906fb3085 vendor: update containerd to v1.0.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
Vincent Demeester 79e346f59a
Add deadcode to the linter
it tends to help keeping uneeded code out O:)

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
Tonis Tiigi 6e40e83d35 Remove net/context dependencies
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
Tonis Tiigi 29b72a3912 solver: add some trace points
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
Tonis Tiigi b94704e55c session: fix session closing and tracing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
Tonis Tiigi 00a5729229 pull: fix content labels after pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi b0679c66db snapshot: clean up snapshot interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi 2db636479c source: avoid parallel resolve of same config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 15:02:15 -08:00
Tonis Tiigi fdfa6bcb59 vendor: update containerd to 1.0.0
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 23:34:25 -08:00
Tonis Tiigi e3a2aa149e containerimage: add schema1 pull support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-27 21:42:08 -08:00
Akihiro Suda c71a1ca768 bump up containerd (v1.0.0-beta.3)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-13 05:36:59 +00:00
Tonis Tiigi 51d1c4166f session: expose image auth
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-20 12:56:06 +02:00
Tonis Tiigi 5f4b51d0c9 blobmap: fix diffid storing on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:38:08 -07:00
Tonis Tiigi f41283caed solver: fix shared request progress and cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 12:45:47 -07:00
Tonis Tiigi 05c39612d8 dockerfile: resolve to digest ref in frontend
Less hacky than keeping the cached resolver.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 15:49:37 -07:00
Tonis Tiigi b36df4738a frontend: add more commands to dockerfile
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 18:09:28 -07:00
Tonis Tiigi 024fd0afaa source: fix multi-platform pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 13:57:40 -07:00
Akihiro Suda 279940b60d vendor containerd d1e11f17ec7b325f89608dd46c128300b8727d50
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-09-14 19:06:31 +00:00
Tonis Tiigi 6d335c3c86 imagesource: share config resolver with frontend
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-05 13:11:27 -07:00
Tonis Tiigi 38ac00090c cache: add more metadata values to snapshots
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-26 11:28:20 -07:00
Tonis Tiigi 9e9df712ec Updates for containerd alpha0
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-12 23:15:12 -07:00
Tonis Tiigi b65989a5e4 source: add git source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-10 22:19:05 -07:00
Tonis Tiigi 25ba9d72de solver: first pass of instruction cache
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-06 20:42:17 -07:00
Tonis Tiigi c9e2493f06 source: update interface for cache calls
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-06 12:12:17 -07:00
Tonis Tiigi 1f5addddac vendor: update containerd
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-01 08:37:36 -07:00
Tonis Tiigi b573060eea project: remove poc label
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-23 09:37:36 -07:00
Tonis Tiigi bd32090372 progress: temp fix for context canceling
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-22 12:14:39 -07:00
Tonis Tiigi 4ed7fa3280 progress: refactor progress IDs to items
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-19 14:32:50 -07:00
Tonis Tiigi 8e7d0904ce client: handle status log in client
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-16 16:08:53 -07:00
Tonis Tiigi 4b41158252 solver: control vertexstatus for pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-06-15 16:08:45 -07:00
Derek McGowan 94d62fd432 Add context to functions which call snapshotter
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
2017-06-13 16:10:31 -07:00
Tonis Tiigi 5a0f803917 imagesource: fill blob map on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-31 15:36:04 -07:00
Tonis Tiigi fdde46f7ff cachemanager: check pulled snapshot
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-30 14:17:04 -07:00
Tonis Tiigi ae05d02379 Renames and refactoring
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-05-26 23:12:13 -07:00