Commit Graph

16 Commits (ee1ad6893a0a7a2c0a72d82b3ba7a86c46794122)

Author SHA1 Message Date
Kohei Tokunaga da821a471c Fix estargz compression loses the original tar metadata
Currently, eStargz compression doesn't preserve the original tar metadata
(header bytes and their order). This causes failure of `TestGetRemote` because
an uncompressed blob converted from a gzip blob provides different digset
against the one converted from eStargz blob even if their original tar (computed
by differ) are the same.
This commit solves this issue by fixing eStargz to preserve original tar's
metadata that is modified by eStargz.

Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-09-22 10:50:57 +09:00
Erik Sipsma a9f1980ebb Refactor cache metadata interface.
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
   won't make sense soon due to other potential changes. For example,
   there can now be multiple blobs associated with a ref (for different
   compression types), so the fact that you could access the "Blob"
   field from the Info method on Ref incorrectly implied there was just
   a single blob for the ref. This is on top of the fact that there is
   no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
   Previously, many parts of the code outside the cache package could
   obtain the bolt storage item for any ref and read/write it directly.
   This made it hard to understand what fields are used and when. Now,
   the Metadata method has been removed from the Ref interface and
   replaced with getters+setters for metadata fields we want to expose
   outside the package, which makes it much easier to track and
   understand. Similar changes have been made to the metadata search
   interface.
3. Use a consistent getter+setter interface for metadata, replacing
   the mix of interfaces like Metadata(), Size(), Info() and other
   inconsistencies.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-25 19:15:09 +00:00
Kohei Tokunaga f8d30d567e Add `estargz` compression type
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-08-24 14:10:09 +09:00
Erik Sipsma 23ec51611f cache: Fix flightcontrol use in computeBlobChain.
Previously, the flightcontrol group was being given a key just set to
the ref's ID, which meant that concurrent calls using different values
of compressionType, createIfNeeded and forceCompression would
incorrectly be de-duplicated.

The change here splits up the flightcontrol group into a few separate
calls and ensures that all the correct input variables are put into the
flightcontrol keys.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-16 17:26:36 +00:00
Koichi Shiraishi 671ae38fb7
all: unify the specs-go package import alias to ocispecs
ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1
                      opencontainers          /image-spec/specs-go/v1

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
2021-08-11 08:29:09 +09:00
Tonis Tiigi a558ac4cce add per domain semaphore to limit concurrent connections
This is a safer alternative until we figure out why
http.Transport based limiting fails.

Some connections like cache export/import do not have a
domain key atm and these connections use global pool.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-07-13 14:08:03 -07:00
ktock 3152eab58f Enable to forcefully specify compression type
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2021-07-07 12:00:06 +09:00
Akihiro Suda a846452fa1
Merge pull request #1791 from tonistiigi/push-pull-retry
add retry handlers to push/pull
2020-11-16 13:28:07 +09:00
Tonis Tiigi cf8babde54 progress: refactor logger handler to better reuse
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-15 13:33:58 -08:00
Tibor Vass fad612e748 Allow worker to override GetRemote(), needed in moby integration
This fixes panics in the moby integration of buildkit when using cache.
Panics come from nil Differ in computeBlobChain which is called in GetRemote().

GetRemote() got moved from Worker to ImmutableRef during the lazy-pull refactor.
However, the ability to easily override GetRemote() got lost with that refactor.

This patch attempts to allow for the override while keeping changes minimal.

Signed-off-by: Tibor Vass <tibor@docker.com>
2020-11-13 21:30:35 +00:00
Tonis Tiigi aa29e77294 fix import cycle in integration test pulls
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-11 21:21:08 -08:00
Tonis Tiigi e3b05289d8 add session injection to remote loading
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-02 22:20:44 -08:00
ktock c975424deb Enable to use remote snapshots for refs
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-08-24 08:44:00 +09:00
Erik Sipsma 1b30fd146b cache: Remove ImageRef from DescHandlers
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma cdcf49fd18 Fix progress in schema1 pull.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma 55cbd19dec Add support for lazily-pulled blobs in cache manager.
This allows the layers of images to only be pulled if/once they are actually
required.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00