Commit Graph

188 Commits (78e88560e1a6643534ee668f1e45a72fe7794066)

Author SHA1 Message Date
Tõnis Tiigi 509b73d91e
Merge pull request #2356 from crazy-max/git-umask
Standard user umask for git process
2021-09-27 18:44:13 -07:00
CrazyMax 3f1152f657
Standard user umask for git process
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-09-27 20:10:19 +02:00
CrazyMax 5fcc94454f
Generate and embed build sources
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-09-18 01:39:35 +02:00
CrazyMax 8bb242e7ac
Refactor url redacting util
Signed-off-by: CrazyMax <crazy-max@users.noreply.github.com>
2021-09-14 17:39:07 +02:00
Erik Sipsma a9f1980ebb Refactor cache metadata interface.
There are a few goals with this refactor:
1. Remove external access to fields that no longer make sense and/or
   won't make sense soon due to other potential changes. For example,
   there can now be multiple blobs associated with a ref (for different
   compression types), so the fact that you could access the "Blob"
   field from the Info method on Ref incorrectly implied there was just
   a single blob for the ref. This is on top of the fact that there is
   no need for external access to blob digests.
2. Centralize use of cache metadata inside the cache package.
   Previously, many parts of the code outside the cache package could
   obtain the bolt storage item for any ref and read/write it directly.
   This made it hard to understand what fields are used and when. Now,
   the Metadata method has been removed from the Ref interface and
   replaced with getters+setters for metadata fields we want to expose
   outside the package, which makes it much easier to track and
   understand. Similar changes have been made to the metadata search
   interface.
3. Use a consistent getter+setter interface for metadata, replacing
   the mix of interfaces like Metadata(), Size(), Info() and other
   inconsistencies.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-25 19:15:09 +00:00
Kohei Tokunaga f8d30d567e Add `estargz` compression type
Signed-off-by: Kohei Tokunaga <ktokunaga.mail@gmail.com>
2021-08-24 14:10:09 +09:00
Erik Sipsma 4237175ac7 pull: use resolvemode in flightcontrol key.
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2021-08-16 17:26:36 +00:00
Koichi Shiraishi 671ae38fb7
all: unify the specs-go package import alias to ocispecs
ocispecs means "O"pen "C"ontainer "I"nitiative image-spec/"specs"-go/v1
                      opencontainers          /image-spec/specs-go/v1

Signed-off-by: Koichi Shiraishi <zchee.io@gmail.com>
2021-08-11 08:29:09 +09:00
coryb de0019bf35 fix dropped pull progress output due to canceled context
fixes #2248

Signed-off-by: coryb <cbennett@netflix.com>
2021-07-14 09:11:44 -07:00
Tonis Tiigi a558ac4cce add per domain semaphore to limit concurrent connections
This is a safer alternative until we figure out why
http.Transport based limiting fails.

Some connections like cache export/import do not have a
domain key atm and these connections use global pool.

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-07-13 14:08:03 -07:00
Morlay 18b49fd7dc refactor to use util/bklog instead of using logurs directly
Signed-off-by: Morlay <morlay.null@gmail.com>
2021-07-13 11:42:31 +08:00
Tonis Tiigi fb32dad95b git: fix default branch detection
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-07-06 16:34:43 -07:00
Levi Harrison 2a8e0b11cd Fix tests
Unfortunately, further test cases will have to be removed because gitindentifier will now leave the branch blank instead of filling it in

Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
2021-07-06 15:38:12 -07:00
Levi Harrison ccc9f20895 Moved getDefaultBranch function to gitsource
It is my suspecion that the tests were failing on previous commits because of the lack of authentication and other stuff like that available in gitidentifier as compared to gitsource

Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
2021-07-06 15:38:00 -07:00
Levi Harrison 8604d7cef2 Automatically detect default git branch
Instead of just assuming that the default branch is master, use ls-remote to find out. Also removed tests that didn't specifiy a branch but required authentication, because those will fail now that the repo is actually checked.

Signed-off-by: Levi Harrison <levisamuelharrison@gmail.com>
2021-07-06 15:37:31 -07:00
Akihiro Suda be8ab28a0a
Merge pull request #2081 from tonistiigi/local-differ-none
Allow none differ on local source to avoid false Dockerfile matches
2021-07-01 09:42:24 +09:00
Cory Bennett b1d441b175 [#2112] progress.FromContext returns a writer factory
this allows progress.Controller to manage the writer lifecycle

Signed-off-by: Cory Bennett <cbennett@netflix.com>
2021-06-28 18:58:29 +00:00
Cory Bennett 1da8b671c4 [#2112] progress.Controller should own the progress.Writer to prevent leaks
Signed-off-by: Cory Bennett <cbennett@netflix.com>
2021-06-28 18:56:45 +00:00
Tonis Tiigi a7b155e30c llb: don't allow content diff for now
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-11 21:15:10 -07:00
Tonis Tiigi baa4fcdb0f add differ support for local source
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-06-11 21:15:07 -07:00
Tõnis Tiigi 03af083def
Merge pull request #2144 from thaJeztah/bump_golang
update to go 1.16
2021-06-08 10:30:33 -07:00
Sebastiaan van Stijn 22dd74ae93
update to go 1.16
This updates all occurrences of Go 1.13 to Go 1.16; also updated
the code that's used to redact credentials in URLs to use the Go
implementation.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-06-02 00:31:57 +02:00
Tonis Tiigi 3bdb6b5159 git: support subdir component
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-05-19 22:13:04 -07:00
Alex Couture-Beil 5d2fd7eb45 Redact credentials from URLs before returning errors
this is to prevent errors such as

    failed to fetch remote https://user:password@github.com/user/private-repo-failure.git: exit status 128

from leaking the password; now it will be displayed like:

    failed to fetch remote https://user:xxxxx@non-existant-host/user/private-repo.git: exit status 128

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2021-04-28 10:39:15 -07:00
Pierre Fenoll a471427331 Fix potential crash on nil value
Signed-off-by: Pierre Fenoll <pierrefenoll@gmail.com>
2021-04-15 15:01:02 +02:00
Tonis Tiigi 5bf64293f8 git: set token only for main remote access
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2021-02-19 00:25:35 -08:00
Alex Couture-Beil 3a5bdf0710 Move prefix check to isGitTransport
- rename IsSSHTransport to IsImplicitSSHTransport
- add testing for NewGitIdentifier

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2021-01-04 11:09:56 -08:00
Alex Couture-Beil 3fa063dcf3 move ssh regexp test to common location
- include testing
- improve hostname checking

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-12-10 17:10:38 -08:00
Alex Couture-Beil 23af9dcb86 require a username for git over ssh
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-12-10 17:10:38 -08:00
Alex Couture-Beil b64f992917 Allow any ssh-user while cloning from git
- this allows one to use a non-standard ssh username like repo@host
rather than assuming it must be git@host

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-12-10 17:10:38 -08:00
Tonis Tiigi a6baa678ea fix building from git url without a protocol
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-09 18:40:34 -08:00
Tonis Tiigi e46cfab9fb fix invalid cancellation errors on pull
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-12-01 00:27:31 -08:00
Alex Couture-Beil 985bd6e7fb disable host key checking when none are given
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-11-12 19:26:24 -08:00
Alex Couture-Beil 92e89a3374 Fixes related to the PR
- back out changes to changing the git url
 - fix gid
 - ignore global ssh config option when specifying known hosts

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-11-12 17:32:29 -08:00
Alex Couture-Beil 486c5fd5c6 change ssh auth sock perms to 0700
Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-11-11 08:20:33 -08:00
Alex Couture-Beil f2c8eb177c Support cloning from non-standard git repos
- fixes assumption that ssh git clones must be via the `git` user.
- allows passing the SSH_AUTH_SOCK from the client to GitSource
- allows passing a known_host entry for ssh

Signed-off-by: Alex Couture-Beil <alex@earthly.dev>
2020-11-10 15:55:17 -08:00
Tonis Tiigi e3b05289d8 add session injection to remote loading
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-11-02 22:20:44 -08:00
Sebastiaan van Stijn fbe1346912
replace github.com/docker/docker/pkg/locker with moby/locker v1.0.1
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-10-30 09:25:05 +01:00
Tonis Tiigi 6a3d6e983b source: fix puller error caching on cancellation
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-10-27 13:21:49 -07:00
ktock a9ba84fd5e Limit size of additional label for avoiding preparation failure
In containerd, there is a size limit for label size (4096 chars).
If an image has many layers (> (4096-43)/72 > 55),
`containerd.io/snapshot/remote/stargz.layers` will hit the limit of
label size and the remote snapshot preparation will fail.
This commit fixes this by limiting the size of the label.

Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-09-11 11:40:56 +09:00
ktock c975424deb Enable to use remote snapshots for refs
Signed-off-by: ktock <ktokunaga.mail@gmail.com>
2020-08-24 08:44:00 +09:00
Tonis Tiigi f1cd79bf65 resolver: add better pooling and custom authenticator
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-08-13 17:27:11 -07:00
Erik Sipsma 1b30fd146b cache: Remove ImageRef from DescHandlers
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma 926ca1804c pull: de-pointer non-optional platform field
Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Erik Sipsma 55cbd19dec Add support for lazily-pulled blobs in cache manager.
This allows the layers of images to only be pulled if/once they are actually
required.

Signed-off-by: Erik Sipsma <erik@sipsma.dev>
2020-08-05 17:18:43 -07:00
Paul "TBBle" Hampson 32a080f0f3 Skip source tests on Windows that depend on read-write bind-mounts
These were previously failing due to use of sh.exe on Windows, and with
that fixed, have been revealed to depend on bind-mounts.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-28 18:07:20 +10:00
Paul "TBBle" Hampson 31195373cf Don't rely on sh being in the PATH on Windows for tests
Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-28 18:07:20 +10:00
Paul "TBBle" Hampson b2d3473cba Skip tests on Windows that attempt to bind-mount read-only
Containerd's mounter doesn't yet support bind-mounts on Windows.
BuildKit short-cuts this for read-write mounts, but not read-only
mounts.

Signed-off-by: Paul "TBBle" Hampson <Paul.Hampson@Pobox.com>
2020-07-28 18:07:20 +10:00
Tonis Tiigi c8190b1518 lint: enable golint
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-19 09:28:24 -07:00
Tonis Tiigi 05013a663b pull: fix session updating on resolver
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
2020-07-05 00:15:33 -07:00