Commit Graph

100 Commits (785436a312230fcc79b41aa044c1643528c91913)

Author SHA1 Message Date
Tonis Tiigi 489246dd28 cache: support for internal/frontend record type
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:53 -07:00
Tonis Tiigi 57b96a0ee5 cache: add record type field to usage record
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-26 22:54:39 -07:00
Tonis Tiigi 9f5e5c7dc7 metaresolver: refactor to option struct and resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:22:15 -07:00
Tonis Tiigi e971b7cddf pull: handle resolvemode
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:55 -07:00
Tonis Tiigi a3b5d22419 llb: add resolvemode to image source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 18:20:04 -07:00
Tonis Tiigi 6be1257f5d export: manifest list support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-25 16:01:12 -07:00
Tonis Tiigi c824098183 session: better error on missing local
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-21 11:49:30 -07:00
Tonis Tiigi 132d08a1cc local: add followpaths to the cache key
FollowPaths is a special case of IncludePatterns and
should behave same way.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-17 15:38:50 -07:00
Akihiro Suda 56e2ea083a
Merge pull request #510 from tonistiigi/win-differ-support
diff: implement windows layer support for linux
2018-07-17 11:29:54 +09:00
Tonis Tiigi bc765861be diff: implement windows layer support for linux
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 16:33:21 -07:00
Tonis Tiigi b3572c6648 image: fix platform passing on config resolve
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-07-16 13:51:58 -07:00
Tonis Tiigi f9383c48a3 source: image source platform support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-25 10:39:17 -07:00
Tonis Tiigi 287b831542 source: remove debug
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 13:45:07 -07:00
Tonis Tiigi 6ce604740f vendor: replace locker pkg for legal
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-08 11:00:37 -07:00
Tonis Tiigi 055dcb6c09 local: allow followpaths for local source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-06-07 23:23:13 -07:00
Tonis Tiigi 1067c24df2 git: support all git transports
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-29 10:56:08 -07:00
Tonis Tiigi 6b23ab01ce exec: make stdio debugging optional
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-25 11:15:32 -07:00
Tõnis Tiigi 6a103ab0ac
Merge pull request #403 from tonistiigi/modtime
Fixes for cache invalidation on modtime change
2018-05-24 13:51:50 -07:00
Ian Campbell ef1a79a1c1 Allow caller to specify platform for use with imagemetaresolver.
This needs to be plumbed through to `imageutil.Config` which is also used from
the `containerimage` package. Since that is used in the context of a Worker the
`platforms.Default()` (corresponding to the platform of the worker itself) is
appropriate so apply a default at that level.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-05-24 11:23:35 +01:00
Tonis Tiigi 03d9b00d9c http: include modtime in cache hash
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 22:24:30 -07:00
Tonis Tiigi 8084e8cde9 llbsolver: allow empty returns for opts
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-22 15:46:32 -07:00
Tonis Tiigi 179366acc7 http: allow setting transport from options
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-17 16:11:48 -07:00
Akihiro Suda d22e5a1721 split puller from source/containerimage into util/pull
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-05-18 06:14:06 +09:00
Tonis Tiigi a2072e6a47 git: fix cachemap bug on sha identifier
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi 9c044db670 llbsolver: update pull for multiple cache keys
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-14 09:53:34 -07:00
Tonis Tiigi c0442a1c0d cache: avoid reloading snapshots in gc queue
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-05-10 15:51:36 -07:00
Tonis Tiigi 566e28c174 snapshot: update mounts to mountable interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-04-27 17:58:24 -07:00
Tonis Tiigi 641fdea51b git: fix cancellation on blocking remotes
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-04-10 17:55:18 -07:00
Akihiro Suda 126f91dea7 update containerd (binary: v1.0.3, library: Apr 3, 2018)
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
2018-04-04 17:22:20 +09:00
Tõnis Tiigi e8dbd3aeb4
Merge pull request #302 from ijc/improved-schema1-support
Improved schema1 support
2018-03-07 09:41:34 -08:00
Ian Campbell ae0293c4bf Add an integration test for pulling schema1 images
Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-07 15:56:22 +00:00
Tõnis Tiigi 18280e73cc
Merge pull request #289 from ijc/allow-local-image-for-gateway-fe
Use local images if pull fails
2018-03-06 09:59:17 -08:00
Ian Campbell ac5091c10b Use local images if pull fails
When using `--frontend=gateway.v0` in order to use a tagged image as the
frontend it is currently required to push the tag to a registry, so that
buildkit can pull it. One can use the `image@sha256:digest` form but this is
inconvenient for the development cycle.

Instead, introduce a proxy `Resolver` which if the pull fails checks the local
image store (if there is one, which is worker dependent) for a suitable image
and if so uses that.

With this I can now:

    buildctl build --frontend=dockerfile.v0 --local context=. --local dockerfile=. --exporter=image --exporter-opt=name=docker.io/ijc25/fe:dev
    builtctl build --frontend=gateway.v0 --frontend-opt=source=ijc25/fe:dev --exporter=image --exporter-opt=name=hello-world:dev

Without any push to a registry.

I am using the containerd worker.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-06 12:19:34 +00:00
Ian Campbell b7632ad036 Correct content-store management after schema1 conversion
The previous code had 3 issues:

- The original schema 1 manifest remained in `ongoing.added` but was not added
  to the content store by the conversion. #296 tried to address this in an
  incorrect way and was reverted by the previous commit. Instead switch in the
  schema2 manifest, which was otherwise missed.
- Empty layers in the schema 1 manifest are not propagated to the schema 2
  version, meaning they are not referenced by anything and hence are never
  freed up and therefore leak
- The new schema 2 image config is generated by the converter and not fetched,
  so it never passes through the `HandlerFunc` which adds things to
  `ongoing.added`.

Two address these last two bullets when processing a schema 1 image walk over
the converted image and together with `ongoing.added` produce lists of used and
unused blobs, use the list of used blobs to perform the GC updates and simply
delete everything on the unused list.

Closes: #301.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:26:47 +00:00
Ian Campbell 0ff79ec71c Revert "Swallow a NotFound error when deleting content, to support schema 1 manifests"
This reverts commit 92101de156 from #296.

This was the wrong approach a subsequent commit will fix this properly.

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-03-02 16:10:19 +00:00
Ian Campbell 92101de156 Swallow a NotFound error when deleting content, to support schema 1 manifests
The behaviour of `github.com/containerd/containerd/remotes/docker/schema1` is
such that the manifest is not actually in the content store, so attempting to
delete it fails with `NotFound`, which with the `gateway.v0` frontend results
in:

    time="2018-02-26T17:01:15Z" level=error msg="fatal error: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found"
    panic: rpc error: code = Unknown desc = content digest sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b: not found

It seems safe to just ignore these errors, since `NotFound` is what we wanted
after delete anyway.

This allows support for e.g. `llb.Image("docker.io/docker/whalesay:latest")`

Signed-off-by: Ian Campbell <ijc@docker.com>
2018-02-26 17:08:52 +00:00
Tonis Tiigi 7906fb3085 vendor: update containerd to v1.0.2
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-02-13 16:00:09 -08:00
Vincent Demeester 79e346f59a
Add deadcode to the linter
it tends to help keeping uneeded code out O:)

Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 17:59:04 -08:00
Vincent Demeester 08373f2381
Clean some small dependencies on docker/docker package
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
2018-02-01 16:05:24 -08:00
Tonis Tiigi 6e40e83d35 Remove net/context dependencies
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-17 09:11:22 -08:00
Tonis Tiigi 29b72a3912 solver: add some trace points
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:12 -08:00
Tonis Tiigi b94704e55c session: fix session closing and tracing
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-08 18:03:06 -08:00
Tonis Tiigi 00a5729229 pull: fix content labels after pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi b0679c66db snapshot: clean up snapshot interface
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2018-01-04 23:09:05 -08:00
Tonis Tiigi 367752bf1c dockerfile: add test for building from git repo
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-17 21:09:20 -08:00
Tonis Tiigi 2d3f36d359 source: add sharedkeyhint to local source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:13:15 -08:00
Tonis Tiigi 08e1c2990c dockerfile: add dockerignore support
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-15 15:03:35 -08:00
Tonis Tiigi ad7c6a905c source: add function for parsing llb to identifier
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-13 16:43:49 -08:00
Tonis Tiigi 2db636479c source: avoid parallel resolve of same config
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-12 15:02:15 -08:00
Tonis Tiigi 2ce418ae7b git: fix checking out submodules
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2017-12-10 18:22:32 -08:00