artifact mismatch while using tools: target-toolchain: host-toolchain

[mahaase]

basement/recipes/libs/pango.yaml

Lines 8 to 11 in fe037ed

	- name: libs::glib-tools
	use: [tools]
	tools:
	target-toolchain: host-toolchain

Hey,

there is an issue for matching artifacts, if we use the tool replace feature.
the artifacts built on jenkins doesn't match with the host-machine using --sandbox.
the second part of the artifact-name isn't correct.

jenkins:

curl -sSg --fail -o libs_glib-tools_x86_64-bob_compat-linux-gnu_dist_1-cfd5-94ae.tgz http://fdt-c-pcs-0004.fdtech.intern/artifact/76/9f/39e5bb610ab473c20ce2405dc07623cdb843f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz

local:

>> x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools
   DOWNLOAD  dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/2/workspace  from http://fdt-c-pcs-0004.fdtech.intern/artifact/76/9f/39e5bb610ab473c20ce2405dc07623cdb843193205b3d955a0351162413094dcae51bf3de1f2-1.tgz .. not found
Build error: Downloading artifact failed

[jkloetzke]

I've quickly built it on current basement master (fe037ed) with the following minimal recipe:

inherit: ["basement::rootrecipe"]

depends:
  - libs::pango-dev

buildScript: |
  true
packageScript: |
  true

There I get the same fingerprint like your Jenkins (f0104983ee64093f2c7ecdf8ea68538690cfa842, the trailing part of the build-id):

[ 153] PACKAGE   libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace
[ 158] AUDIT     libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace .. ok
[ 159] UPLOAD    libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace  to /home/jkloetzk/src/private/mismatch/cache/3a/f1/b16448697d072325eee71e5bcd7a5d710de1f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz .. ok

It looks like your local build is somehow modified. Do you use the same sandbox? Could you supply some recipes where it can be reproduced?

[mahaase]

now i tested to remove the libs::glib-tools from pango.yaml and added it to the rootrecipe.yaml.

that works...!

should have both variants the same artifact?

UPDATE:
some more information:

in case of current variant:

curl -sSg --fail -o libs_glib-tools_x86_64-bob_compat-linux-gnu_dist_1-e25d-89b0.tgz http://fdt-c-pcs-0004.fdtech.intern/artifact/76/9f/39e5bb610ab473c20ce2405dc07623cdb843f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz

same pc, where jenkins is installed:

bob dev x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools/  -vv
--sandbox --download forced
>> x86_64::linux/devel::sandbox/devel::bootstrap-sandbox/devel::cross-toolchain/devel::gcc-cross/devel::binutils
   FNGRPRNT  devel::binutils .. 'x86_64\nglibc 2.27\nx86_64\nlibstdc++ 2...
>> x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools
   DOWNLOAD  dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/2/workspace  from http://fdt-c-pcs-0004.fdtech.intern/artifact/76/9f/39e5bb610ab473c20ce2405dc07623cdb843193205b3d955a0351162413094dcae51bf3de1f2-1.tgz .. not found
Build error: Downloading artifact failed

my laptop with WSL2:

bob dev x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools/  -vv --s
andbox --download forced
>> x86_64::linux/devel::sandbox/devel::bootstrap-sandbox/devel::cross-toolchain/devel::gcc-cross/devel::binutils
   FNGRPRNT  devel::binutils .. 'x86_64\nglibc 2.27\nx86_64\nlibstdc++ 2...
>> x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools
   DOWNLOAD  dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/2/workspace  from http://fdt-c-pcs-0004.fdtech.intern/artifact/76/9f/39e5bb610ab473c20ce2405dc07623cdb843193205b3d955a0351162413094dcae51bf3de1f2-1.tgz .. not found
Build error: Downloading artifact failed

#######
variant with glib-tools in rootrecipe:

WSL2:

bob dev x86_64::linux/devel::graphviz/libs::pango-dev/  -vv --sandbox --download forced
>> x86_64::linux/devel::sandbox/devel::bootstrap-sandbox/devel::cross-toolchain/devel::gcc-cross/devel::binutils
   FNGRPRNT  devel::binutils .. 'x86_64\nglibc 2.27\nx86_64\nlibstdc++ 2...
>> x86_64::linux/devel::graphviz/libs::pango-dev
   DOWNLOAD  dev/dist/libs/pango-dev/x86_64-bob_compat-linux-gnu/2/workspace  from http://fdt-c-pcs-0004.fdtech.intern/artifact/b1/b3/7cfeddf62bc2c622f935bf2573682f12bd21ae6e377d4ca6f0edb039e2e089c4c911e40d860c-1.tgz .. ok

jenkins pc:

bob dev x86_64::linux/devel::graphviz/libs::pango-dev/  -vv --sandbox --download forced
>> x86_64::linux/devel::sandbox/devel::bootstrap-sandbox/devel::cross-toolchain/devel::gcc-cross/devel::binutils
   FNGRPRNT  devel::binutils .. 'x86_64\nglibc 2.27\nx86_64\nlibstdc++ 2...
>> x86_64::linux/devel::graphviz/libs::pango-dev
   DOWNLOAD  dev/dist/libs/pango-dev/x86_64-bob_compat-linux-gnu/2/workspace  from http://fdt-c-pcs-0004.fdtech.intern/artifact/b1/b3/7cfeddf62bc2c622f935bf2573682f12bd21ae6e377d4ca6f0edb039e2e089c4c911e40d860c-1.tgz .. ok

[jkloetzke]

What other toolchain are you using in x86_64::linux? Maybe that makes a difference?

[jkloetzke]

Another thing: did you set BASEMENT_HOST_COMPAT_TOOLCHAIN to 0?

[jkloetzke]

Could you provide the audit.json.gz from the Jenkins build artifact (76/9f/39e5bb610ab473c20ce2405dc07623cdb843f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz), build x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools on the Jenkins pc and provide the audit.json.gz too? Maybe we can spot some difference...

[mahaase]

What other toolchain are you using in x86_64::linux? Maybe that makes a difference?

i don't think so. that is just stuff much far behind the current problem. the current problem occures in scope of the basement.

Another thing: did you set BASEMENT_HOST_COMPAT_TOOLCHAIN to 0?

nope! i know this switch. 👍

Could you provide the audit.json.gz from the Jenkins build artifact (76/9f/39e5bb610ab473c20ce2405dc07623cdb843f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz), build x86_64::linux/devel::graphviz/libs::pango-dev/libs::glib-tools on the Jenkins pc and provide the audit.json.gz too? Maybe we can spot some difference...

yep:
audit-artifact.json.gz
audit-build.json.gz

how do u compare them? my meld crashs/stucks while loading the files :/

[jkloetzke]

What other toolchain are you using in x86_64::linux? Maybe that makes a difference?

i don't think so. that is just stuff much far behind the current problem. the current problem occures in scope of the basement.

I was asking to understand the environment a bit better to come up with some theory what could go wrong. So the question is, do you configure another toolchain in your x86_64::linux recipe or do you stay with the default devel::host-compat-toolchain which is configured by basement::rootrecipe? If you use another toolchain, is it configured before or after the devel::graphviz dependency?

how do u compare them? my meld crashs/stucks while loading the files :/

Unfortunately there is no good tooling yet. Comparing it on text level won't work. I hope I'll find the time to dig into it this evening...

[mahaase]

I was asking to understand the environment a bit better to come up with some theory what could go wrong. So the question is, do you configure another toolchain in your x86_64::linux recipe or do you stay with the default devel::host-compat-toolchain which is configured by basement::rootrecipe? If you use another toolchain, is it configured before or after the devel::graphviz dependency?

x86_64/linux.yaml

inherit: ["toolchain::x86_64"]

toolchian/x86_64.yaml

inherit: [rootrecipe]

root: !expr |
  "${BOB_HOST_PLATFORM}" == "linux"

depends:
    - name: devel::cross-toolchain-x86_64-linux-gnu
      use: [tools]
      forward: True

rootrecipe.yaml

inherit: ["basement::rootrecipe", "pipython3::rootrecipe"]

depends:
    - name: devel::cuda
      use: [tools]
      forward: True
    - name: fdt::cmake-export
      use: [tools]
      forward: True
    - if: !expr |
        "${BOB_HOST_PLATFORM}" == "msys"
      name: devel::win::graphviz
      use: [tools]
      forward: True
    - if: !expr |
        "${BOB_HOST_PLATFORM}" == "linux"
      name: devel::graphviz
      use: [tools]
      forward: True

devel/cuda.yaml:

shared: True

metaEnvironment:
    PKG_VERSION: "10.2.89"

checkoutSCM:
    scm: url
    url: ${FILE_SERVER}/cuda/cuda-${PKG_VERSION}-${BOB_HOST_PLATFORM}-${ARCH}.tgz
    digestSHA256: "$(if-then-else,$(eq,${BOB_HOST_PLATFORM},msys),5263fd45bdcbeeac41664ef4006fd30bbdad7707dd43589fd72143de1264e255,\
                   $(if-then-else,$(eq,${ARCH},x86_64),63b258e6f9a375db334b63c5642634900ef283885e137b8f571f7e7b5a584c17,4238785e559f88b97b201603ad6235a13ed714e8cd37781ab33c6950652f63cc))"
    extract: False

buildScript: |
    tar -xf $1/*.tgz

packageScript: |
     rsync -a "$1/" .
     for i in $(find . -name bin) ; do
        chmod +x $i/* -R
     done

provideTools:
    cuda:
        path: "bin"
        environment:
            CUDACXX: "nvcc"
            CUDAFLAGS: "-Xcompiler='\"'$(if-then-else,$(eq,$BOB_HOST_PLATFORM,msys),-O$(if-then-else,$(eq,${BASEMENT_OPTIMIZE},0),d,${BASEMENT_OPTIMIZE})$(if-then-else,${BASEMENT_DEBUG}, -Zi,) -MD$(if-then-else,${BASEMENT_DEBUG},d,) -W3,-O${BASEMENT_OPTIMIZE}$(if-then-else,${BASEMENT_DEBUG}, -g,) -pipe)'\"'"

cmake-export.yaml

inherit: [cmake]

metaEnvironment:
    PKG_VERSION: "v1.0.1"

checkoutSCM:
    scm: git
    url: ${GIT_SERVER}/fdt/fdt.cmake.export.git
    tag: ${PKG_VERSION}

buildScript: |
    mkdir -p install/usr
    rsync -a --delete $1/bin install/usr/

packageScript: |
    rsync -a --delete $1/install/* .

provideTools:
    cmake-export: "usr/bin"

I guess, that is the full chain before. nothing strang, isn't it?
I will try to reproduce it, just with basement repository.

[mahaase]

I've quickly built it on current basement master (fe037ed) with the following minimal recipe:

inherit: ["basement::rootrecipe"]

depends:
  - libs::pango-dev

buildScript: |
  true
packageScript: |
  true

There I get the same fingerprint like your Jenkins (f0104983ee64093f2c7ecdf8ea68538690cfa842, the trailing part of the build-id):

[ 153] PACKAGE   libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace
[ 158] AUDIT     libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace .. ok
[ 159] UPLOAD    libs::glib-tools - dev/dist/libs/glib-tools/x86_64-bob_compat-linux-gnu/1/workspace  to /home/jkloetzk/src/private/mismatch/cache/3a/f1/b16448697d072325eee71e5bcd7a5d710de1f0104983ee64093f2c7ecdf8ea68538690cfa842-1.tgz .. ok

It looks like your local build is somehow modified. Do you use the same sandbox? Could you supply some recipes where it can be reproduced?

i can't upload this recipe to jenkins server:

Parse error: Jobs are cyclic

[jkloetzke]

i can't upload this recipe to jenkins server:

Parse error: Jobs are cyclic

Ok, now we have two problems. 😒 I have the feeling that this is still somehow related to the original problem...

[mahaase]

But i also tested this with clean basement master, does this work for u?

[jkloetzke]

No, I get the same error on vanilla basement with the minimal root recipe.

[daxcore]

i also gone back to 28th of december and tried with kernel::linux-image. still the same issue. so i guess, target-toolchain: host-toolchain is the trigger of the problem.

OH: i guess the sandbox is the problem. not the target-toolchain: host-toolchain stuff. this also happens, if i just have a dependency to e.g. libs::expat-dev.

[jkloetzke]

I think I found a smoking gun: libs::pango depends on libs::glib-dev and libs::glib-tools. They differ only in the packageStep. Both are built with the devel::host-compat-toolchain but the former with the target-toolchain and the latter with the renamed host-toolchain. Content wise they are the same but the host-toolchain has a fingerprintScript while the target-toolchain has none. Depending on which is built first the fingerprint part of the buildStep will be different...

Currently the build logic only looks on the variant-id. Two packages with the same variant-id but different fingerprintScript are treated as the same. Right now I have no good idea how to solve the issue...

[daxcore]

oh. maybe u should give u and the problem a night, ... or two ;)

UPDATE: the problem "cyclic jobs" also persists, if there is no pango recipe. i checked out b9874d09bea5de1ca5e3fb7b0d470c20ea126479, there was no pango.

UPDATE2: i tested the stuff at work by replacing target-toolchain: host-toolchain by adding glib-tools in rootrecipe, that worked.