With this commit the `profiles.json` contain both the target specific
`default_packages` as well as the device specific `device_packages` as a
array of strings.
This information is required for downstream projects like the various
web-based interactive firmware generators.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
Signed-off-by: Paul Spooren <mail@aparcar.org>
(cherry picked from commit 263f7e5bbd)
JSON info files contain machine readable information of built profiles
and resulting images. These files were added in commit 881ed09ee6
("build: create JSON files containing image info").
They are useful for firmware wizards and script checking for
reproducibility.
Currently all JSON files are stored next to the built images, resulting
in up to 168 individual files for the ath79/generic target.
This patch refactors the JSON creation to store individual per image
(not per profile) files in $(BUILD_DIR)/json_info_files and create an
single overview file called `profiles.json` in the target directory.
Storing per image files and not per profile solves the problem of
parallel file writes. If a profiles sysupgrade and factory image are
finished at the same time both processes would write to the same JSON
file, resulting in randomly broken outputs.
Some target like x86/64 do not use the image code yet, resulting in
missing JSON files. If no JSON info files were created, no
`profiles.json` files is created as it would be empty anyway.
As before, this creation is enabled by default only if `BUILDBOT` is set.
Tested via buildroot & ImageBuilder on ath79/generic, imx6 and x86/64.
Signed-off-by: Paul Spooren <mail@aparcar.org>
[json_info_files dir handling in Make, if case refactoring]
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(backported from commit 07449f692c)
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
Commit 881ed09ee6 ("build: create JSON files containing image info")
has removed the crucial empty new line from the image copy step
resulting in the following errors during make function expansion:
GZ_SUFFIX :=
bash: GZ_SUFFIX: command not found
Makefile:86: recipe for target 'openwrt-ath79-generic-tplink_archer-c7-v5-squashfs-sysupgrade.bin' failed
Fixes: 881ed09ee6 ("build: create JSON files containing image info")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
(cherry picked from commit 0fb23d67f0)
The JSON info files contain details about the created firmware images
per device and are stored next to the created images.
The JSON files are stored as "$(IMAGE_PREFIX).json" and contain some
device/image meta data as well as a list of created firmware images.
An example of openwrt-ramips-rt305x-aztech_hw550-3g.json
{
"id": "aztech_hw550-3g",
"image_prefix": "openwrt-ramips-rt305x-aztech_hw550-3g",
"images": [
{
"name": "openwrt-ramips-rt305x-aztech_hw550-3g-squashfs-sysupgrade.bin",
"sha256": "db2b34b0ec4a83d9bf612cf66fab0dc3722b191cb9bedf111e5627a4298baf20",
"type": "sysupgrade"
}
],
"metadata_version": 1,
"supported_devices": [
"aztech,hw550-3g",
"hw550-3g"
],
"target": "ramips/rt305x",
"titles": [
{
"model": "HW550-3G",
"vendor": "Aztech"
},
{
"model": "ALL0239-3G",
"vendor": "Allnet"
}
],
"version_commit": "r10920+123-0cc87b3bac",
"version_number": "SNAPSHOT"
}
Signed-off-by: Paul Spooren <mail@aparcar.org>
(backported from commit 881ed09ee6)
Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
This reverts commit 1c0290c5cc.
Dropping the nopad can make the padding overflow into the next erase
block on devices using a non-aligned rootfs start. This breaks the jffs2
overlay partition with the following messages:
[ 30.343877] jffs2_scan_eraseblock(): End of filesystem marker found at 0x10000
[ 30.376512] jffs2: Cowardly refusing to erase blocks on filesystem with no valid JFFS2 nodes
[ 30.385253] jffs2: empty_blocks 196, bad_blocks 0, c->nr_blocks 197
Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
(cherry picked from commit f11d90a76b)
While the -nopad option prevents mksquashfs from padding the
image to an arbitrary 4k. It does not take into consideration
that squashfs is programmed to have this 4k padding when it's
being used on on a block device... which is its main "use-case".
Now, after a week long discussion on the ML that included a
back-and-forth between some of the possible options.
But this is likely the best KISS patch to deal with the issue
right away given the limited resources.
From squashfs code point of view, be warned. The 4k padding is
not enough when dealing with devices that have a PAGE_SIZE
bigger than 4k.
if it turns out to be affecting you, then please look-up either:
"FS#2460 - kernel panic reading squashfs from ubi volume" bug
Or the discussion on the OpenWrt-Devel ML in
"amp821xx: use newly added pad-squashfs for Meraki MR24" and
"Squashfs breakage lottery with UBI..."
before making an educated guess.
Note: This will not affect the "tiny"/small flash devices as
much as it seems at first. This is because the the rootfs_data
partition that follows uses jffs2. And it requires to be aligned
to the flash block-size in order to work at all.
So either the involved FSes will meet in the middle as before,
or not at all. But in that latter case the image was already
hoping for the "undefined behaviour" gamble to turn out in its
favour and this is probably why this was unnoticed for so long.
Fixes: FS#2460
Reported-by: Russell Senior <russell@personaltelco.net>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
(cherry picked from commit 1c0290c5cc)
Allow overriding the default selection state for Devices, similar to
setting a default for packages.
E.g. by setting DEFAULT to n, they won't be selected by default anymore
when enabling all device in the multi device profile.
This allows preventing images being built by the default config for
known broken devices, devices without enough RAM/flash, or devices not
working with a certain kernel versions.
This does not prevent the devices from being manually selected or images
being built by the ImageBuilder. These devices often still have worth
with a reduced package-set, or as a device for regression testing, when
no better device is available.
Signed-off-by: Jonas Gorski <jonas.gorski@gmail.com>
(cherry picked from commit 7546be6007)
The helper shared Build/append-uboot in include/image-commands.mk
uses it, so include this variable in DEFAULT_DEVICE_VARS.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
Otherwise tar will keep the sgid bit when running
from a sgid-set directory, resulting in a different
file being generated.
Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
[reworked commit message, removed DMARC]
It was reported to me on IRC today, that my change causes issues with
kernel versions between 4.14 and 4.19.
It's because I've wrongly used `git describe` in order to get kernel
version where we should disable noisy DTC checks, but I should've used
`git tag --contains` instead.
Fixes: cbbef976e2 ("build: dtc: Disable noisy warnings by default")
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Having image metadata (and signature) appended is a condition for
semi-automated sysupgrade, hence IB needs to be able to tell which
images will end up with metadata.
Signed-off-by: Daniel Golle <daniel@makrotopia.org>
squashfskit is a fork of the squashfs-tools.
squashfskit creates reproducible filesystems and includes
many of the distro patches.
Signed-off-by: Alexander Couzens <lynxis@fe80.eu>
Tested-by: Paul Spooren <mail@aparcar.org>
BIN_DIR can be set to overwrite the output path for new images. This is an
advertised feature for the imagebuilder and is used by systems like
LibreMesh's chef.
The legacy images are build using a new sub-make which doesn't receive the
variable overwrites of the parent make process. As result, the BIN_DIR is
automatically defined to the default value from rules.mk. The images will
therefore not be placed in the output path which was selected by the user.
Providing BIN_DIR as an explicit variable override to the sub-make works
around this problem.
Fixes: 26c771452c ("image.mk: add LegacyDevice wrapper to allow legacy image building code to be used for device profiles")
Reported-by: Paul Spooren <mail@aparcar.org>
Signed-off-by: Sven Eckelmann <sven@narfation.org>
The KERNEL_ENTRY was missing from the DEFAULT_DEVICE_VARS.
This bug was discovered while preparing alternative images
for the mpc85xx's TP-Link WDR4900-V1, which all failed to
boot due to this:
|## Booting kernel from Legacy Image at 02000000 ...
| Image Name: POWERPC OpenWrt Linux-4.14.96
| Image Type: PowerPC Linux Kernel Image (uncompressed)
| Data Size: 2056568 Bytes = 2 MiB
| Load Address: 01000000
| Entry Point: 00000000
| Verifying Checksum ... OK
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
It was reported to me today on IRC, that building of artifacts doesn't
work properly if the concat_cmd references DEVICE_NAME variable. I've
found out, that it's due to missing call of Device/Export in artifacts
building code path, so this patch adds the missing Device/Export call
which in turn exports DEFAULT_DEVICE_VARS into the artifacts
environment.
Fixes: 493c9a3551 ("build: Introduce building of artifacts")
Tested-by: Oskari Lemmela <oskari@lemmela.net>
Reported-by: Oskari Lemmela <oskari@lemmela.net>
Signed-off-by: Petr Štetiar <ynezz@true.cz>
While helping with review and build testing of a few 4.19 pull requests,
I've noticed, that dtc compiler in OpenWrt uses different options then
upstream kernel, which is leading to a very noisy output[1]. It wouldn't
be that bad per se, but a lot of such warnings aren't easily fixable so
I think, that we should follow what upstream does and simply ignore^W
silence those noisy warnings.
So this patch tries to syncs dtc compiler flags with upstream kernel
till version 4.19.13, disabling those warnings as they were added in
upstream kernel:
v4.6-rc1-2-gbc55398 dtc: turn off dtc unit address warnings by default
The newly added dtc warning to check DT unit-address without reg
property and vice-versa generates lots of warnings. Turn off the check
unless building with W=1 or W=2.
v4.11-rc2-11-g8654cb8 dtc: update warning settings for new bus and node/property
dtc gained new warnings checking PCI and simple buses, unit address
formatting, and stricter node and property name checking. Disable the
new dtc warnings by default as there are 1000s. As before, warnings are
enabled with W=1 or W=2. The strict node and property name checks are a
bit subjective, so they are only enabled for W=2.
v4.16-rc3-9-g4fd98e3 scripts: turn off some new dtc warnings by default
The latest dtc update adds some new noisy warnings, so turn them off by
default. Disable 'avoid_unnecessary_addr_size' and 'alias_paths'. They
can be re-enabled by building with 'W=1'.
v4.17-rc1-27-g74656b6 kbuild: disable new dtc graph and unit-address warnings
dtc gained some new warnings for OF graphs and unique unit addresses,
but they are currently much too noisy. So turn off
'graph_child_address', 'graph_port', and 'unique_unit_address' warnings
by default. They can be enabled by building dtbs with W=1.
Build tested on imx6 and ath79 with 4.14 and 4.19.
1. https://github.com/openwrt/openwrt/pull/1694#issuecomment-450864335
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Add the ABI_VERSION source makefile variable to the binary package basename
and resolve source dependencies on packages with ABI_VERSION set to such
expanded names.
If for example a package specifies DEPENDS:=libopenssl while the OpenSSL
Makefile specifies ABI_VERSION:=1.0.0, the resulting ipk control data
dependency will be "Depends: libopenssl1.0.0" and the libopenssl ipk file
will be called "libopenssl1.0.0_<version>_<arch>.ipk".
The next time a library such as OpenSSL is updated to an incompatible
version, the ABI_VERSION shall be changed accordingly to prevent opkg from
simply upgrading to an incompatible library without considering the
dependencies of already installed packages.
Also introduce another "SourceName" control field which is required by
the newly introduced "scritps/ipkg-remove" to determine the proper related
.ipk files to delete upon buildroot package clean operations.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
We currently could (ab)use IMAGES for this task, but the downside is,
that the filenames has filesystem tied to the filename, which might be
confusing as the artifact itself don't has to be used with that specific
filesystem. Another downside is, that the artifacts built with IMAGES
target are build for every FILESYSTEMS filesystem.
Consider following use case:
define Device/apalis
...
FILESYSTEMS := ext4 squashfs
IMAGES := spl-uboot.bin recovery.scr
IMAGE/spl-uboot.bin := append-uboot-spl | pad-to 68k | append-uboot
IMAGE/recovery.scr := recovery-scr
endef
Where we would get target binaries with following filenames:
openwrt-imx6-apalis-squashfs.recovery.scr
openwrt-imx6-apalis-squashfs.spl-uboot.bin
openwrt-imx6-apalis-ext4.recovery.scr
openwrt-imx6-apalis-ext4.spl-uboot.bin
With proposed patch, we could now just do:
define Device/apalis
...
ARTIFACTS := spl-uboot.bin recovery.scr
ARTIFACT/spl-uboot.bin := append-uboot-spl | pad-to 68k | append-uboot
ARTIFACT/recovery.scr := recovery-scr
endef
Which would produce target binaries with following filenames:
openwrt-imx6-apalis-recovery.scr
openwrt-imx6-apalis-spl-uboot.bin
Signed-off-by: Petr Štetiar <ynezz@true.cz>
Acked-by: Hauke Mehrtens <hauke@hauke-m.de>
Some devices only boot when a special config is found in the image and
completely ignore the default entry during the selection. These devices can
now use the variable DEVICE_DTS_CONFIG in their device image definition.
Signed-off-by: Sven Eckelmann <sven@narfation.org>
This reverts commit 43be5087a9.
The change is incompatible with the image builder code.
Luckily the RT-AC58U is no longer depending on the initramfs
being available for the target's image generation rules.
Reported-by: Venitex Aveon <aveon@aenote.com>
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
The image production rules does not have the initramfs-image
as a dependency. So, from make’s perspective initramfs
creation can run independently/in parallel with the image
generation code in the target's Makefile.
This is a problem for devices that have to use the initramfs
for the image creation and can lead to broken images.
Signed-off-by: Christian Lamparter <chunkeey@gmail.com>
When generating per-device rootfs directories, the ./etc/opkg/ directory
is moved away prior to calling opkg install, opkg remove and rootfs_prepare.
After the opkg invocations and the rootfs_prepare macro call, the saved opkg
config directory is supposed to be moved back to its previous ./etc/opkg
location.
The mv command however can fail to properly restore the directory under
certain circumstances, e.g. when the prior opkg or files/ overlay copy
operations caused a new ./etc/opkg/ directory to be created.
In this case, the backed up directory (named target-dir-$hash.opkg) will be
moved into the preexisting ./etc/opkg/ directory instead, causing the opkg
configuration to be located in a wrong path on the final rootfs, e.g. in
/etc/opkg/target-dir-$hash.opkg/distfeeds.conf instead of
/etc/opkg/distfeeds.conf.
Solve this problem by replacing the naive "mv" command with a recursive
"cp -T" invocation which causes the backed up directory tree to get merged
with the destination directory in case it already exists.
Also perform the rootfs_prepare macro call after restoring the opkg
configuration, to allow users to override it again by using the files/
overlay mechanism.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
mpc85xx uses this for firmware image files, since the dtb data is not
directly part of the kernel image. This causes build failures in the
image builder.
Fix this by adding a separate build step that runs this call earlier,
reusing the generated file for any calls from kernel or image build
commands.
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Starting with commit d5d332d3f7e8 ("devicetree: Move include prefixes
from arch to separate directory") included in 4.12 and newer relocated
the dt-bindings directory, so account for that while passing CPPFLAGS
before DTC runs.
Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
Some files (e.g. /etc/dropbear) need to be owned by root. Add cpio
option to ensure that.
Other image types (at least targz and squashfs) already have this.
Signed-off-by: Michal Sojka <sojkam1@fel.cvut.cz>
This reduces the amount of hacks in the makefile code.
Remove the apm821xx code to do the same - it was broken and left both
compressed and uncompressed images in $(BIN_DIR)
Signed-off-by: Felix Fietkau <nbd@nbd.name>
The generated 'its' is passed to mkimage which expects linux arch
strings rather than the full arch (e.g. mips not mipsel).
It currently works in some cases where LINUX_KARCH == ARCH but
otherwise you get an unknown arch build error.
Signed-off-by: Ian Pozella <Ian.Pozella@imgtec.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>
Now that the VERSION_NUMBER variable holds the human friendly name and not
the commit ID anymore, we need to support adding the revision ID as well.
Introduce a new config variable CONFIG_VERSION_CODE_FILENAMES which, if set,
causes the resulting file names to contain a commit ID designation as printed
by scripts/getver.sh.
Also sanitize the input variables to ensure that the resulting strings are
lowercased and no not contain spaces.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
There is very little practical use to limit the number of available inodes on
an ext4 filesystem and the make_ext4fs utility is able to calculate useful
defaults by itself.
Drop the option to make resulting ext4 filesystems more flexible by default.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Acked-by: Michael Heimpold <mhei@heimpold.de>
Allow CONFIG_TARGET_EXT4_RESERVED_PCT to be empty as make_ext4fs is usually
able to figure out a suitable default.
Signed-off-by: Jo-Philipp Wich <jo@mein.io>
Reviewed-by: Michael Heimpold <mhei@heimpold.de>
Especially --force-overwrite and --force-depends will often lead to broken
images; it's better to fail the build in such cases than to silently ignore
the errors.
Instead, ignore errors in the per-device rootfs opkg remove command, so
the build doesn't break when packages can't be removed because of
dependencies.
Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Paul Spooren
Petr Štetiar
Jonas Gorski
Christian Lamparter
Eneas U de Queiroz
Daniel Golle
Alexander Couzens
Sven Eckelmann
Jo-Philipp Wich
Felix Fietkau
Matthias Schiffer
Zoltan HERPAI
Florian Fainelli
Michal Sojka
John Crispin
Ian Pozella