On Wed, May 19, 2021 at 12:47 PM Konrad Weihmann [off-list ref]
wrote:

On 19.05.21 12:23, Martin Jansa wrote:

quoted

* all known issues in this area were fixed, make it fatal that it
   cannot be overlooked if someone triggers this issue again

Signed-off-by: Martin Jansa <redacted>
---
  meta/lib/oe/sstatesig.py | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/lib/oe/sstatesig.py b/meta/lib/oe/sstatesig.py
index 6cd6e11acc..e86a09b332 100644
--- a/meta/lib/oe/sstatesig.py
+++ b/meta/lib/oe/sstatesig.py

@@ -453,7 +453,7 @@ def find_sstate_manifest(taskdata, taskdata2,

taskname, d, multilibcache):

quoted

          manifest = d2.expand("${SSTATE_MANIFESTS}/manifest-%s-%s.%s" %

(pkgarch, taskdata, taskname))

quoted

          if os.path.exists(manifest):
              return manifest, d2
-    bb.error("Manifest %s not found in %s (variant '%s')?" % (manifest,

d2.expand(" ".join(pkgarchs)), variant))

quoted

+    bb.fatal("Manifest %s not found in %s (variant '%s')?" % (manifest,

d2.expand(" ".join(pkgarchs)), variant))

quoted

      return None, d2

doesn't that make the return statement obsolete? also how can an error
be overlooked? It does make the task triggering this issue fail, so how
is a fatal better in this case (besides abnormally terminating the task)

bb.error doesn't make the task to fail, it only shows an error (which
results in non-zero exit code), but lets depending tasks to continue and
fail with other (often more misleading) errors, because their dependency is
really missing in RSS

without this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_configure: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_install: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: adwaita-icon-theme-3.34.3-r0 do_package_write_ipk: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
NOTE: Tasks Summary: Attempted 2819 tasks of which 2803 didn't need to be
rerun and all succeeded.
Summary: There were 6 ERROR messages shown, returning a non-zero exit code.

with this patch:
ERROR: adwaita-icon-theme-3.34.3-r0 do_prepare_recipe_sysroot: Manifest
/OE/build/oe-core/tmp-glibc/sstate-control/manifest-x86_64_ubuntu-21.04-at-spi2-core-native.populate_sysroot
not found in x86_64 x86_64_ubuntu-21.04 (variant '')?
ERROR: Logfile of failure stored in:
/OE/build/oe-core/tmp-glibc/work/all-oe-linux/adwaita-icon-theme/3.34.3-r0/temp/log.do_prepare_recipe_sysroot.1638704
ERROR: Task
(/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:do_prepare_recipe_sysroot)
failed with exit code '1'
NOTE: Tasks Summary: Attempted 2805 tasks of which 2800 didn't need to be
rerun and 1 failed.
Summary: 1 task failed:

/OE/build/oe-core/openembedded-core/meta/recipes-gnome/gnome/adwaita-icon-theme_3.34.3.bb:
do_prepare_recipe_sysroot
Summary: There was 1 ERROR message shown, returning a non-zero exit code.

Without this fix:
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479

I was seeing various native dependencies missing and as you can see in
the adwaita-icon-theme example, missing at-spi2-core-native isn't fatal for
the build, but were adwaita-icon-theme packages and sstate built correctly?
If you don't investigate all these ERROR messages and non-zero exit code
(e.g. just because you have thousands of builds per day populating the same
sstate mirror), you might miss some of these breakages. While using fatal
prevents adwaita-icon-theme to even start building and producing possibly
broken packages, images, sstate.

In some other cases e.g. qtwayland-native missing when building target
qtwayland you will get do_compile failure complaining that the the binary
from qtwayland-native is missing, but the root cause of this issue is
written many lines above do_compile log. You can see some other examples in
https://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=817f1c14cec4d66504a5028908a4ebac147f3b52
and before changing this from bb.warn to bb.error in previous patch most of
these issues were just "some random builddir breakage which was resolved by
doing clean build from sstate".

And this happens in multi-machine builds with dunfell/gatesgarth/hardknott
and only master is safe now. I plan to ask Steve to backport this and
related fixes to dunfell after a bit more time in master:
https://git.openembedded.org/openembedded-core/commit/?id=0eb95212a7e6b6bdc1243094072dea432cb48f0f
https://git.openembedded.org/openembedded-core/commit/?id=63da9a4f889c5b0e41bc8ec08abe0acea1546479
https://git.openembedded.org/openembedded-core/commit/?id=95607a26854d873399d2b9d7e5fcbffc0cbdba4c
and then this one.

Regards,