Commit Graph

63 Commits (763379fb2c521e5804e7ce713ca5a59a2007d81e)

Author SHA1 Message Date
Tibor Vass 914d86758c
Merge pull request #1024 from tonistiigi/gcr-workaround
pull: add cache for gcr bug
2019-06-17 10:36:49 -07:00
Tonis Tiigi ab23a1ab1b pull: add cache for gcr bug
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 18:51:55 -07:00
Tonis Tiigi d72c19071a imageutil: use leases on fetching config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 14:39:05 -07:00
Tonis Tiigi 8e25187cc7 pull: fix content blobs deletion on pull race
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-05-28 14:39:05 -07:00
Tonis Tiigi 37567fc826 pull: fix second cachekey for schema1
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2019-02-25 17:16:43 -08:00
Akihiro Suda 26010a4864 decouple SessionManager from Worker
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2019-02-23 22:55:17 +09:00
Tonis Tiigi ecd4a22546 resolver: add config support for mirrors/plainhttp
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-09-07 18:44:45 -07:00
Tonis Tiigi 489246dd28 cache: support for internal/frontend record type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi e971b7cddf pull: handle resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:55 -07:00
Tonis Tiigi 6be1257f5d export: manifest list support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 16:01:12 -07:00
Akihiro Suda 56e2ea083a
Merge pull request #510 from tonistiigi/win-differ-support
diff: implement windows layer support for linux
2018-07-17 11:29:54 +09:00
Tonis Tiigi bc765861be diff: implement windows layer support for linux
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 16:33:21 -07:00
Tonis Tiigi b3572c6648 image: fix platform passing on config resolve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 13:51:58 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Ian Campbell ef1a79a1c1 Allow caller to specify platform for use with imagemetaresolver.
This needs to be plumbed through to `imageutil.Config` which is also used from
the `containerimage` package. Since that is used in the context of a Worker the
`platforms.Default()` (corresponding to the platform of the worker itself) is
appropriate so apply a default at that level.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-05-24 11:23:35 +01:00
Tonis Tiigi 8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Akihiro Suda d22e5a1721 split puller from source/containerimage into util/pull
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-05-18 06:14:06 +09:00
Tonis Tiigi 9c044db670 llbsolver: update pull for multiple cache keys
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi c0442a1c0d cache: avoid reloading snapshots in gc queue
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-10 15:51:36 -07:00
Tonis Tiigi 566e28c174 snapshot: update mounts to mountable interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-04-27 17:58:24 -07:00
Akihiro Suda 126f91dea7 update containerd (binary: v1.0.3, library: Apr 3, 2018)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-04 17:22:20 +09:00
Tõnis Tiigi e8dbd3aeb4
Merge pull request #302 from ijc/improved-schema1-support
Improved schema1 support
2018-03-07 09:41:34 -08:00
Ian Campbell ae0293c4bf Add an integration test for pulling schema1 images
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-07 15:56:22 +00:00
Tõnis Tiigi 18280e73cc
Merge pull request #289 from ijc/allow-local-image-for-gateway-fe
Use local images if pull fails
2018-03-06 09:59:17 -08:00
Ian Campbell ac5091c10b Use local images if pull fails
When using `--frontend=gateway.v0` in order to use a tagged image as the
frontend it is currently required to push the tag to a registry, so that
buildkit can pull it. One can use the `image@sha256:digest` form but this is
inconvenient for the development cycle.

Instead, introduce a proxy `Resolver` which if the pull fails checks the local
image store (if there is one, which is worker dependent) for a suitable image
and if so uses that.

With this I can now:

    buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --exporter=image --exporter-opt=name=docker.io/ijc25/fe:dev
    builtctl build --frontend=gateway.v0 --frontend-opt=source=ijc25/fe:dev --exporter=image --exporter-opt=name=hello-world:dev

Without any push to a registry.

I am using the containerd worker.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-06 12:19:34 +00:00
Ian Campbell b7632ad036 Correct content-store management after schema1 conversion
The previous code had 3 issues:

- The original schema 1 manifest remained in `ongoing.added` but was not added
  to the content store by the conversion. #296 tried to address this in an
  incorrect way and was reverted by the previous commit. Instead switch in the
  schema2 manifest, which was otherwise missed.
- Empty layers in the schema 1 manifest are not propagated to the schema 2
  version, meaning they are not referenced by anything and hence are never
  freed up and therefore leak
- The new schema 2 image config is generated by the converter and not fetched,
  so it never passes through the `HandlerFunc` which adds things to
  `ongoing.added`.

Two address these last two bullets when processing a schema 1 image walk over
the converted image and together with `ongoing.added` produce lists of used and
unused blobs, use the list of used blobs to perform the GC updates and simply
delete everything on the unused list.

Closes: #301.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:26:47 +00:00
Ian Campbell 0ff79ec71c Revert "Swallow a NotFound error when deleting content, to support schema 1 manifests"
This reverts commit 92101de156 from #296.

This was the wrong approach a subsequent commit will fix this properly.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:10:19 +00:00
Ian Campbell 92101de156 Swallow a NotFound error when deleting content, to support schema 1 manifests
The behaviour of `github.com/containerd/containerd/remotes/docker/schema1` is
such that the manifest is not actually in the content store, so attempting to
delete it fails with `NotFound`, which with the `gateway.v0` frontend results
in:

    time="2018-02-26T17:01:15Z" level=error msg="fatal error: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found"
    panic: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found

It seems safe to just ignore these errors, since `NotFound` is what we wanted
after delete anyway.

This allows support for e.g. `llb.Image("docker.io/docker/whalesay:latest")`

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-02-26 17:08:52 +00:00
Tonis Tiigi 7906fb3085 vendor: update containerd to v1.0.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
Vincent Demeester 79e346f59a
Add deadcode to the linter
it tends to help keeping uneeded code out O:)

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
Tonis Tiigi 6e40e83d35 Remove net/context dependencies
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
Tonis Tiigi 29b72a3912 solver: add some trace points
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
Tonis Tiigi b94704e55c session: fix session closing and tracing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
Tonis Tiigi 00a5729229 pull: fix content labels after pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi b0679c66db snapshot: clean up snapshot interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi 2db636479c source: avoid parallel resolve of same config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 15:02:15 -08:00
Tonis Tiigi fdfa6bcb59 vendor: update containerd to 1.0.0
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-04 23:34:25 -08:00
Tonis Tiigi e3a2aa149e containerimage: add schema1 pull support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-11-27 21:42:08 -08:00
Akihiro Suda c71a1ca768 bump up containerd (v1.0.0-beta.3)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-11-13 05:36:59 +00:00
Tonis Tiigi 51d1c4166f session: expose image auth
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-20 12:56:06 +02:00
Tonis Tiigi 5f4b51d0c9 blobmap: fix diffid storing on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-10-13 15:38:08 -07:00
Tonis Tiigi f41283caed solver: fix shared request progress and cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-27 12:45:47 -07:00
Tonis Tiigi 05c39612d8 dockerfile: resolve to digest ref in frontend
Less hacky than keeping the cached resolver.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-22 15:49:37 -07:00
Tonis Tiigi b36df4738a frontend: add more commands to dockerfile
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 18:09:28 -07:00
Tonis Tiigi 024fd0afaa source: fix multi-platform pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-15 13:57:40 -07:00
Akihiro Suda 279940b60d vendor containerd d1e11f17ec7b325f89608dd46c128300b8727d50
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2017-09-14 19:06:31 +00:00
Tonis Tiigi 6d335c3c86 imagesource: share config resolver with frontend
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-09-05 13:11:27 -07:00
Tonis Tiigi 38ac00090c cache: add more metadata values to snapshots
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-26 11:28:20 -07:00
Tonis Tiigi 9e9df712ec Updates for containerd alpha0
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-07-12 23:15:12 -07:00