This will produce a "requested feature is not supported" style error message
with some more specific details about the build server configuration rather
than a more generic gRPC "unsupported" error message.
Ensure that the tests pass the caps to `Marshal` so they see the new
functionality.
Signed-off-by: Ian Campbell <ijc@docker.com>
Previously, the output said "load Dockerfile from " + filename when the
file is not named Dockerfile. With this patch, it is changed to always
show "load build definition from " + filename.
This is because `# syntax = ...` can be put in non-Dockerfiles.
Signed-off-by: Tibor Vass <tibor@docker.com>
The alternative is a helper container running e.g. find, capturing the output
to a scratch to be read back (with ref.ReadFile) and parsed.
Signed-off-by: Ian Campbell <ijc@docker.com>
```
FROM alpine
RUN echo "Hey $name"
```
When building the above Dockerfile, the RUN command used to show: `echo Hey `
With this patch it now shows `echo "Hey $name"`.
However, because it is very useful for debugging to see the values variables can take,
they are expanded if set.
```
FROM alpine
ENV name=Jude
RUN echo "Hey $name"
```
Building the above Dockerfile makes the RUN command show: `echo "Hey Jude"`.
Signed-off-by: Tibor Vass <tibor@docker.com>
Most accesses to this map are protected. Add the missing locks (some of which
are in code paths where we wouldn't normally expect concurrent accesses, but
better to be safe).
Signed-off-by: Ian Campbell <ijc@docker.com>
Tests such as TestNoSnapshotLeak were failing in client mode (e.g. using #522)
because we weren't releasing the intermediate refs.
Resolve this by refactoring the existing code which frees the intermediate refs
from `gatewayFrontend.Solve` into a method on `llbBridgeForwarder` and as well
as the original site also call from the solver when the top-level solve (in
clientside frontend mode) completes. The original call (which is via a defer)
could likely sensibly be moved either earlier or later if desired but leave it
here it is to minimise the scope of the change.
The previous code used the `retErr` named return but the code between that
point and the end of the function already ensured that `lbf.err` is the same as
`retErr`, thus the only change in the code which has moved is
`s/retErr/lbf.err/`. Note that the `res` named return was previously unused by
name.
Fixes#581.
Signed-off-by: Ian Campbell <ijc@docker.com>
This allows builder code to be written which can be built as either a gateway
container or in a purely client side configuration, giving implementors more
flexibility.
Now when `Solve` sees a request with neither a definition nor a frontend
specified it will make the job available via new LLBBridge endpoints on the
control socket which the client can then use. These end points require the job
id to be present in the gRPC metadata and a client side object is added to
facilitate this.
The `llbBridgeForwarder` type is now exposed as a public `interface
LLBBridgeForwarder` which satisfies the underlying gRPC server interface
(`pb.LLBBridgeServer`) as well as a new `Done()` & `Result()` pair which can be
used to wait for the client to call `Return()` (using a model similar to
`context.Context`).
Signed-off-by: Ian Campbell <ijc@docker.com>
Reorder the code slightly so that an error returned by the call to the
`BuildFunc` is picked up by the `defer`d error handling.
Signed-off-by: Ian Campbell <ijc@docker.com>
Refactor the setting/getting of the result for a `llbBridgeForwarder` in order
to check that `Return` is only called once and only with the correct options.
Signed-off-by: Ian Campbell <ijc@docker.com>
Provide a `New` method and a new `Run` method on the `grpcClient` to allow
manual creation of a client in addition to the current ability to initialise
from the environment. Accordingly the existing `Run` method becomes
`RunFromEnvironment`.
Signed-off-by: Ian Campbell <ijc@docker.com>
On this code `dispatchEnv` is called with `commit = false`,
so this call do these 2 things
* `AddEnv` to `d.state`
* `addEnv` to update `d.image.Config.Env`
But updating `d.image.Config.Env` has no effect on `d.image.Config.Env`
because this is in for-loop of `range d.image.Config.Env`.
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
In the case when the second argument of `toEnvList` is `nil`,
we can use `ProcessWordWithMap` and avoid string contatination
and string split.
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
`KeyValuePairOptional` is enough in this case. We can avoid create `ArgCommand` out of Parser
by replacing them with `KeyValuePairOptional`.
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
`dispatchStates` manages a collection of `dispatchState`.
By `dispatchStates`, we can aggregate how to find `dispatchState`
in one place and `dispatchStatesByName` local variable in `Dockerfile2LLB`
can be removed.
Signed-off-by: Yuichiro Kaneko <spiketeika@gmail.com>
This allows two things:
- The caller to set a shorter timeout than previously hardcoded 30s. In
`buildctl` reduce the timeout to 5s. Since the existing timeout has gone
callers will need to arrange to pass one themselves.
- The caller can arrange for the context to be cancelled for other reasons, use
this in `buildctl` to plumb through the Ctrl-C handling, meaning that
`buildctl` now exits almost immediately on Ctrl-C instead of after several
seconds.
Signed-off-by: Ian Campbell <ijc@docker.com>
Detect Special parameters as defined in
http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html#tag_02_05_02
Treat these as parameters that are not set, instead of
producing an error that a modifier is missing.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby 9654e9b6f80e1b931763c04400a19596e256c99a
`${}`, `${:}` and so on are invalid because there's
no parameter within the brackets; fix detection for
this situation and add/update tests.
There were some existing test-cases that were testing
for the wrong behavior, which are now updated.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby 334bf3ea76004d0abe02dd1698989f9eaf87a86a
The builder did not detect syntax errors in substitusions in the
Dockerfile, causing those values to be processed incorrectly instead of
producing an error.
Example 1: missing `}`
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${aaa:-bbb
RUN echo $var
EOF
Before:
Step 3/3 : RUN echo $var
---> Running in f06571e77146
bbb
After:
Step 2/3 : ARG var=${aaa:-bbb
failed to process "${aaa:-bbb": syntax error: missing '}'
Example 2: missing closing `}`, no default value
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${aaa
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${aaa
failed to process "${aaa": missing ':' in substitution
After:
Step 2/3 : ARG var=${aaa
failed to process "${aaa": syntax error: missing '}'
Example 3: double opening bracket (`{`)
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${{aaa:-bbb}
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${{aaa:-bbb}
failed to process "${{aaa:-bbb}": missing ':' in substitution
After:
Step 2/3 : ARG var=${{aaa:-bbb}
failed to process "${{aaa:-bbb}": syntax error: bad substitution
Example 4: double opening bracket (`{`), no default value
docker build --no-cache -<<'EOF'
FROM busybox
ARG var=${{aaa}
RUN echo $var
EOF
Before:
Step 2/3 : ARG var=${{aaa}
failed to process "${{aaa}": missing ':' in substitution
After:
Step 2/3 : ARG var=${{aaa}
failed to process "${{aaa}": syntax error: bad substitution
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby 955a6ad95f7891a45692d975793abf1eeb07cdd5
remove unnescessary import aliases, brackets, and so on.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby f23c00d8701e4bd0f2372a586dacbf66a26f9a51
This hasn't been the way to release Docker for the past year so let's
just remove them altogether
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
rewritten from github.com/moby/moby 80bc8b4766cc808186af91930152d78afb22aef8
Since we use `NewLabelCommand()` instead of `addNodesForLabelOption()`
to create the 'LABEL' commands from '--label' options, so all the related
functions should be removed.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
rewritten from github.com/moby/moby c7b543164daed58fbea36471592438b4e53ab748
This PR is tring to fix issue #36996.
Currently for multi-stage build, if `--target` specified, the `--label` option
will be ignored. The root cause is the last stage build will remove the `LABEL`
command(s) node created from the `--label` option. In order to address this issue,
we can create `LABEL` command(s) and add it/tem to the last stage.
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
rewritten from github.com/moby/moby 9c238ebd55e4105ad7f7edc04231ea61bb278ae8
This PR is trying to fix issue #36956.
The stage name is case-insensitive by design, so we should use
`strings.EqualFold()` as the comparison method to eliminate the
case sensitive noise.
Also we need to return a pre-defined error code order to avoid below
message like:
"FIXME: Got an API for which error does not match any expected type!!!:
failed to reach build target dev in Dockerfile"
Signed-off-by: Dennis Chen <dennis.chen@arm.com>
rewritten from github.com/moby/moby 7c0570473cfa181aeb3278072cc9af4f9298cb98
gty-migrate-from-testify --ignore-build-tags
Signed-off-by: Daniel Nephin <dnephin@docker.com>
rewritten from github.com/moby/moby 6be0f709830113966f295401327b027ec2f0bbca
Moves builder/shell_parser and into its own subpackage at builder/shell since it
has no dependencies other than the standard library. This will make it
much easier to vendor for downstream libraries, without pulling all the
dependencies of builder/.
Fixes#36154
Signed-off-by: Matt Rickard <mrick@google.com>
rewritten from github.com/moby/moby a634526d14639c6b98c509a069ef29e2b69c0ef0
Signed-off-by: John Howard <jhoward@microsoft.com>
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
rewritten from github.com/moby/moby d2c41f06c36dc24b2170fef365563d64cc5bd8ec
Signed-off-by: John Howard <jhoward@microsoft.com>
This PR has the API changes described in https://github.com/moby/moby/issues/34617.
Specifically, it adds an HTTP header "X-Requested-Platform" which is a JSON-encoded
OCI Image-spec `Platform` structure.
In addition, it renames (almost all) uses of a string variable platform (and associated)
methods/functions to os. This makes it much clearer to disambiguate with the swarm
"platform" which is really os/arch. This is a stepping stone to getting the daemon towards
fully multi-platform/arch-aware, and makes it clear when "operating system" is being
referred to rather than "platform" which is misleadingly used - sometimes in the swarm
meaning, but more often as just the operating system.
rewritten from github.com/moby/moby 0380fbff37922cadf294851b1546f4c212c7f364
Commit 8d1ae76dcbbb73d8e20c6a14a7d3fe2410b95f55 added
deprecation warnings for empty continuation lines,
but also treated comment-only lines as empty.
This patch distinguishes empty continuation lines
from comment-only lines, and only outputs warnings
for the former.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby 2fd736ac10c1c46d1001373d887cb99b3d8ee824
This is a work base to introduce more features like build time
dockerfile optimisations, dependency analysis and parallel build, as
well as a first step to go from a dispatch-inline process to a
frontend+backend process.
Signed-off-by: Simon Ferquel <simon.ferquel@docker.com>
rewritten from github.com/moby/moby 669c0677980b04bcbf871bb7c2d9f07caccfd42b
This fix is related to 29005 and 24693. Currently in `Dockerfile`
empty lines will continue as long as there is line escape before.
This may cause some issues. The issue in 24693 is an example.
A non-empty line after an empty line might be considered to be a
separate instruction by many users. However, it is actually part
of the last instruction under the current `Dockerfile` parsing
rule.
This fix is an effort to reduce the confusion around the parsing
of `Dockerfile`. Even though this fix does not change the behavior
of the `Dockerfile` parsing, it tries to deprecate the empty line
continuation and present a warning for the user. In this case,
at least it prompt users to check for the Dockerfile and avoid
the confusion if possible.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
rewritten from github.com/moby/moby 7815c8f8754d5473eda7cd80277a4ea3c59e3c29
I noticed that we're using a homegrown package for assertions. The
functions are extremely similar to testify, but with enough slight
differences to be confusing (for example, Equal takes its arguments in a
different order). We already vendor testify, and it's used in a few
places by tests.
I also found some problems with pkg/testutil/assert. For example, the
NotNil function seems to be broken. It checks the argument against
"nil", which only works for an interface. If you pass in a nil map or
slice, the equality check will fail.
In the interest of avoiding NIH, I'm proposing replacing
pkg/testutil/assert with testify. The test code looks almost the same,
but we avoid the confusion of having two similar but slightly different
assertion packages, and having to maintain our own package instead of
using a commonly-used one.
In the process, I found a few places where the tests should halt if an
assertion fails, so I've made those cases (that I noticed) use "require"
instead of "assert", and I've vendored the "require" package from
testify alongside the already-present "assert" package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
rewritten from github.com/moby/moby 6052f2b3969feadb01662d8e2f30337d9c7f61af
text does not appear to contain a placeholder
Signed-off-by: Helen Xie <chenjg@harmonycloud.cn>
rewritten from github.com/moby/moby 2a8d6368d4a930203b93f75914173ab65bf3b0bc
This drops support for migrations from pre-1.10 Docker versions, which
should be done via an external tool or an intermediate upgrade.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
rewritten from github.com/moby/moby f8119bb7a76b5c42defb6e0a2dc67bd77ad29a5e
This reverts commit 105bc63295a7126798d3722a0e205c5ead4e2b1c,
which (although correct), resulted in a backward incompatible
change.
We can re-implement this in future, after this changes goes
through a deprecation cycle
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby c67550414b762ab32ea3e9645ebcafb79cb9059d
This reverts commit 3e1b539e8d0ed4abf695b0a8c42346fba6d5a6b0.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
rewritten from github.com/moby/moby 6e0666e622a8b1fc19c19eb490d218eebca15edc
Came from looking at issue #27545
Signed-off-by: Doug Davis <dug@us.ibm.com>
rewritten from github.com/moby/moby eaf0b5708fc91ab0c78eeee1a2c9c1251439d4a8
This fix tries to fix the bug reported by #24693 where an empty
line after escape will not be stopped by the parser.
This fix addresses this issue by stop the parser from continue
with an empty line after escape.
An additional integration test has been added.
This fix fixes#24693.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
rewritten from github.com/moby/moby 3e1b539e8d0ed4abf695b0a8c42346fba6d5a6b0
While look at #27039 I noticed that we allow for whitespace after
the continuation char (\\) which is wrong. It needs to be the very
last char in the line.
Signed-off-by: Doug Davis <dug@us.ibm.com>
rewritten from github.com/moby/moby 105bc63295a7126798d3722a0e205c5ead4e2b1c
We attached the JSON flag to the wrong AST node, causing Docker to treat
the exec form ["binary", "arg"] as if the shell form "binary arg" had
been used. This failed if "ls" was not present.
Added a test to detect this.
Fixes#26174
Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
rewritten from github.com/moby/moby e95b6b51daed868094c7b66113381d5088e831b4
Signed-off-by: John Howard <jhoward@microsoft.com>
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
rewritten from github.com/moby/moby 755be795b4e48b3eadcdf1427bf9731b0e97bed1
`TestParseWords` needs to use the `tokenEscape` for one of the test
cases, but `tokenEscape` was not being set unless tests ran in a
specific order.
This sets a default value for `tokenEscape`... `\`... so that tests that
rely on this global are not affected by test ordering.
This is the simplest fix for these cases. Ideally the token should not
be set as a global but rather passed down, which is a much larger
change.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
rewritten from github.com/moby/moby df167d3ff04cdc90012c8ca39647662ad69e6715
This fix tries to address issues in #23221 where Dockerfile
may consists of UTF-8 BOM. This likely happens when Notepad
tries to save a file as UTF-8 in Windows.
This fix skips the UTF-8 BOM bytes from the beginning of the
Dockerfile if exists.
Additional tests has been added to cover the changes in this
fix.
This fix fixes#23221.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
rewritten from github.com/moby/moby 678c80f9256021ce74184fdd6b612d9dea377fba
This PR adds support for user-defined health-check probes for Docker
containers. It adds a `HEALTHCHECK` instruction to the Dockerfile syntax plus
some corresponding "docker run" options. It can be used with a restart policy
to automatically restart a container if the check fails.
The `HEALTHCHECK` instruction has two forms:
* `HEALTHCHECK [OPTIONS] CMD command` (check container health by running a command inside the container)
* `HEALTHCHECK NONE` (disable any healthcheck inherited from the base image)
The `HEALTHCHECK` instruction tells Docker how to test a container to check that
it is still working. This can detect cases such as a web server that is stuck in
an infinite loop and unable to handle new connections, even though the server
process is still running.
When a container has a healthcheck specified, it has a _health status_ in
addition to its normal status. This status is initially `starting`. Whenever a
health check passes, it becomes `healthy` (whatever state it was previously in).
After a certain number of consecutive failures, it becomes `unhealthy`.
The options that can appear before `CMD` are:
* `--interval=DURATION` (default: `30s`)
* `--timeout=DURATION` (default: `30s`)
* `--retries=N` (default: `1`)
The health check will first run **interval** seconds after the container is
started, and then again **interval** seconds after each previous check completes.
If a single run of the check takes longer than **timeout** seconds then the check
is considered to have failed.
It takes **retries** consecutive failures of the health check for the container
to be considered `unhealthy`.
There can only be one `HEALTHCHECK` instruction in a Dockerfile. If you list
more than one then only the last `HEALTHCHECK` will take effect.
The command after the `CMD` keyword can be either a shell command (e.g. `HEALTHCHECK
CMD /bin/check-running`) or an _exec_ array (as with other Dockerfile commands;
see e.g. `ENTRYPOINT` for details).
The command's exit status indicates the health status of the container.
The possible values are:
- 0: success - the container is healthy and ready for use
- 1: unhealthy - the container is not working correctly
- 2: starting - the container is not ready for use yet, but is working correctly
If the probe returns 2 ("starting") when the container has already moved out of the
"starting" state then it is treated as "unhealthy" instead.
For example, to check every five minutes or so that a web-server is able to
serve the site's main page within three seconds:
HEALTHCHECK --interval=5m --timeout=3s \
CMD curl -f http://localhost/ || exit 1
To help debug failing probes, any output text (UTF-8 encoded) that the command writes
on stdout or stderr will be stored in the health status and can be queried with
`docker inspect`. Such output should be kept short (only the first 4096 bytes
are stored currently).
When the health status of a container changes, a `health_status` event is
generated with the new status. The health status is also displayed in the
`docker ps` output.
Signed-off-by: Thomas Leonard <thomas.leonard@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
rewritten from github.com/moby/moby b6c7becbfe1d76b1250f6d8e991e645e13808a9c
This fix tries to address the inconsistency in #22036 where labels
set on the command line will not override labels specified in
Dockerfile, but will override labels inherited from `FROM` images.
The fix add a LABEL with command line options at the end of the
processed Dockerfile so that command line options labels always
override the LABEL in Dockerfiles (or through `FROM`).
An integration test has been added for test cases specified in #22036.
This fix fixes#22036.
NOTE: Some changes are from #22266 (@tiborvass).
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
rewritten from github.com/moby/moby 5844736c14b29860ea03b040e9a052e59ad75bfc
Most of them were found and fixed by codespell.
Signed-off-by: Stefan Weil <sw@weilnetz.de>
rewritten from github.com/moby/moby 2eee613326fb59fd168849618d14a9054a40f9f5
The LXC driver was deprecated in Docker 1.8.
Following the deprecation rules, we can remove a deprecated feature
after two major releases. LXC won't be supported anymore starting on Docker 1.10.
Signed-off-by: David Calavera <david.calavera@gmail.com>
rewritten from github.com/moby/moby 3b5fac462d21ca164b3778647420016315289034
This commit fixes the case when "!" is provided alone as a dockerignore
pattern resulting in buildkit hanging. An integration test is added to
guard the bug.
The bug was due to incorrect error propagation in the fsutil package.
Thus this commit vendors a newer, fixed version of fsutil.
Signed-off-by: Tibor Vass <tibor@docker.com>
e.g. with busybox image:
OCI runtime create failed: container_linux.go:348:
starting container process caused "process_linux.go:402:
container init caused \"rootfs_linux.go:58:
mounting \\\"proc\\\" to rootfs \\\"/.../rootfs\\\" at \\\"/proc\\\"
caused \\\"mkdir /.../rootfs/proc: read-only file system\\\"\"": unknown
This is because we were setting the underlying snapshot readonly so the various
mountpoints (here /proc) cannot be created. This would not be necessary if
those mountpoints were present in images but they typically are not.
The right way to get around this (used e.g. by `ctr`) is to use a writeable
snapshot but to set root readonly in the OCI spec. In this configuration the
rootfs is writeable when mounts are processed but is then made readonly by the
runtime (runc) just before entering the user specified binary within the
container.
This involved a surprising amount of plumbing.
Use this new found ability in the dockerfile converter's `dispatchCopy`
function.
Signed-off-by: Ian Campbell <ijc@docker.com>
Modelled after the vendor support provide a validator and an updator for files
produced by `go generate` (which today just means `*.pb.go`).
Main difference from the vendor support is that we are no longer simply nuking
and replacing a single directory, so I ended up hardcoding `*.pb.go` in a bunch
of places which I don't like but cannot see a way around which doesn't risk
nuking people's other local changes.
The generated files are placed in an unpacked form in a `FROM scratch`
container for update. Use a subdirectory and `tar --strip-components` (portable
to MacOS and Linux according to `tar(1)`) since trying to do a `docker export`
of just the root ends up adding `.dockerenv`, `sys`, `proc` and `dev` to the
source tree.
The validate container is not `FROM scratch` because we want `cat`.
The run in `frontend/gateway/pb/generate.go` was missing an include so fix
that.
The versions of `protoc` and the gogo plugins were chosen to regenerate the
existing code as closely as possible. The updates to `*.pg.go` here are all the
result of regenerating with go1.9 which fixed
https://github.com/golang/go/issues/17663 and replaced an invalid timestamp in
the gzip header of the data encoded in `fileDescriptor*`, and adopted a new
standard for marking generated files.
Finally, I noticed that my `docker run`s were missing an `--rm` which I
inherited from `validate-vendor`, so fix all those.
Closes: #322
Signed-off-by: Ian Campbell <ijc@docker.com>
Tõnis Tiigi
Akihiro Suda
Akihiro Suda
Ian Campbell
Tibor Vass
Tibor Vass
Kunal Kushwaha
Yuichiro Kaneko
Anda Xu
Vincent Demeester
Sebastiaan van Stijn
Eli Uriegas
Dennis Chen
John Howard
Daniel Nephin
bin liu
Matt Rickard
Anthony Sottile
Ri Xu
Simon Ferquel
Yong Tang
Jean-Pierre Huynh
Aaron Lehmann
fate-grand-order
Aaron.L.Xu
Justin Cormack
Doug Davis
allencloud
Thomas Leonard
Brian Goff
Shev Yan
Tiffany Jernigan
Jonathan Stoppani
Jessica Frazelle
Stefan Weil
Justas Brazauskas
perhapszzy@sina.com
David Calavera
Alexander Morozov