Skip to content

Conversation

@cladmi
Copy link
Contributor

@cladmi cladmi commented Nov 23, 2018

Contribution description

When building in parallel, clean is done at the same time as the
container starts building.

So the parallel clean all handling should also be done for the
..in-docker-container target.

Testing procedure

I added a test commit to increase the probability of showing the issue and should be removed before merging. The issue is also present and was found without.

Testing should do a BUILD_IN_DOCKER=1 make -j clean all. It could be shown reliably with any example that takes more than 5 seconds to build on your machine.

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all        

With this PR, (should be run when sudo does not ask for the password anymore).

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all                                                                                                                                                                                                  
make: Entering directory '/home/harter/work/git/RIOT/examples/gnrc_networking'                                                                                                                                                                                                            
sleep 2                                                                                                                                                                                                                                                                                   
for i in  ; do "make" -C /home/harter/work/git/RIOT/pkg/$i clean ; done                                                                                                                                                                                                                   
rm -rf /home/harter/work/git/RIOT/examples/gnrc_networking/bin/native                                                                                                                                                                                                                     
rm -rf scan-build/native
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/RIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/RIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/RIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache  \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
"make" -C /data/riotbuild/riotbase/sys/net/crosslayer/inet_csum
"make" -C /data/riotbuild/riotbase/sys/net/gnrc
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netapi
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif/hdr
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netreg
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6/echo
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/icmpv6/error
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6/hdr
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ipv6/nib
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/network_layer/ndp
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pkt
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktbuf
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktbuf_static
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/pktdump
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/routing/rpl
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/transport_layer/udp
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/icmpv6
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/ipv6/addr
"make" -C /data/riotbuild/riotbase/sys/net/network_layer/ipv6/hdr
"make" -C /data/riotbuild/riotbase/sys/net/transport_layer/udp
"make" -C /data/riotbuild/riotbase/sys/od
"make" -C /data/riotbuild/riotbase/sys/ps
"make" -C /data/riotbuild/riotbase/sys/random
"make" -C /data/riotbuild/riotbase/sys/random/tinymt32
"make" -C /data/riotbuild/riotbase/sys/shell
"make" -C /data/riotbuild/riotbase/sys/shell/commands
"make" -C /data/riotbuild/riotbase/sys/trickle
"make" -C /data/riotbuild/riotbase/sys/xtimer
"make" -C /data/riotbuild/riotboard/native
"make" -C /data/riotbuild/riotboard/native/drivers
"make" -C /data/riotbuild/riotcpu/native
"make" -C /data/riotbuild/riotcpu/native/netdev_tap
"make" -C /data/riotbuild/riotcpu/native/periph
"make" -C /data/riotbuild/riotcpu/native/vfs
   text    data     bss     dec     hex filename
 215663     641  102788  319092   4de74 /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_networking.elf
make: Leaving directory '/home/harter/work/git/RIOT/examples/gnrc_networking'

With only the test commit, clean happens during the build in docker:

BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -C examples/gnrc_networking -j clean all
make: Entering directory '/home/harter/work/git/RIOT/examples/gnrc_networking'
sleep 2
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotbase' \
    -v '/home/harter/work/git/RIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/harter/work/git/RIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/harter/work/git/RIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/harter/work/git/RIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/harter/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache  \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all 
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
for i in  ; do "make" -C /home/harter/work/git/RIOT/pkg/$i clean ; done
rm -rf /home/harter/work/git/RIOT/examples/gnrc_networking/bin/native
rm -rf scan-build/native
/data/riotbuild/riotbase/sys/luid/luid.c:62:1: fatal error: opening dependency file /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.d: No such file or directory
 }
 ^
compilation terminated.
/data/riotbuild/riotbase/Makefile.base:83: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.o' failed
make[3]: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/luid/luid.o] Error 1
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/luid' failed
make[2]: *** [ALL--/data/riotbuild/riotbase/sys/luid] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys' failed
make[1]: *** [ALL--/data/riotbuild/riotbase/sys] Error 2
/data/riotbuild/riotbase/Makefile.include:449: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a' failed
make: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a] Error 2
/home/harter/work/git/RIOT/makefiles/docker.inc.mk:105: recipe for target '..in-docker-container' failed
make: *** [..in-docker-container] Error 2
make: Leaving directory '/home/harter/work/git/RIOT/examples/gnrc_networking'

Issues/PRs references

Was found while testing #10344 and also mentioned in #10459 (comment)

@cladmi cladmi added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable Area: build system Area: Build system CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … labels Nov 23, 2018
@cladmi cladmi requested review from jcarrano and jnohlgard November 23, 2018 17:33
Copy link
Contributor

@jcarrano jcarrano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested:

Without this fix:

RIOT/examples/gnrc_networking $BUILD_IN_DOCKER=1 DOCKER="sudo docker" make -j clean all
sleep 2
Launching build container using image "riot/riotbuild:latest".
sudo docker run --rm -t -u "$(id -u)" \
    -v '/home/jcarrano/source/masterRIOT:/data/riotbuild/riotbase' \
    -v '/home/jcarrano/source/masterRIOT/cpu:/data/riotbuild/riotcpu' \
    -v '/home/jcarrano/source/masterRIOT/boards:/data/riotbuild/riotboard' \
    -v '/home/jcarrano/source/masterRIOT/makefiles:/data/riotbuild/riotmake' \
    -v '/home/jcarrano/source/masterRIOT:/data/riotbuild/riotproject' \
    -v /etc/localtime:/etc/localtime:ro \
    -e 'RIOTBASE=/data/riotbuild/riotbase' \
    -e 'CCACHE_BASEDIR=/data/riotbuild/riotbase' \
    -e 'RIOTCPU=/data/riotbuild/riotcpu' \
    -e 'RIOTBOARD=/data/riotbuild/riotboard' \
    -e 'RIOTMAKE=/data/riotbuild/riotmake' \
    -e 'RIOTPROJECT=/data/riotbuild/riotproject' \
    -v /home/jcarrano/.gitcache:/data/riotbuild/gitcache -e GIT_CACHE_DIR=/data/riotbuild/gitcache -v /home/jcarrano/source/vanillaRIOT/.git:/home/jcarrano/source/vanillaRIOT/.git \
     \
    -w '/data/riotbuild/riotproject/examples/gnrc_networking/' \
    'riot/riotbuild:latest' make all 
Building application "gnrc_networking" for "native" with MCU "native".

"make" -C /data/riotbuild/riotbase/core
"make" -C /data/riotbuild/riotbase/drivers
"make" -C /data/riotbuild/riotbase/drivers/netdev_eth
"make" -C /data/riotbuild/riotbase/drivers/periph_common
"make" -C /data/riotbuild/riotbase/sys
"make" -C /data/riotbuild/riotbase/sys/auto_init
"make" -C /data/riotbuild/riotbase/sys/auto_init/netif
"make" -C /data/riotbuild/riotbase/sys/div
"make" -C /data/riotbuild/riotbase/sys/evtimer
"make" -C /data/riotbuild/riotbase/sys/fmt
"make" -C /data/riotbuild/riotbase/sys/iolist
"make" -C /data/riotbuild/riotbase/sys/luid
"make" -C /data/riotbuild/riotbase/sys/net/crosslayer/inet_csum
"make" -C /data/riotbuild/riotbase/sys/net/gnrc
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netapi
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif
"make" -C /data/riotbuild/riotbase/sys/net/gnrc/netif/hdr
for i in  ; do "make" -C /home/jcarrano/source/masterRIOT/pkg/$i clean ; done
rm -rf /home/jcarrano/source/masterRIOT/examples/gnrc_networking/bin/native
rm -rf scan-build/native
/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr/gnrc_netif_hdr_print.c:62:1: fatal error: opening dependency file /data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.d: No such file or directory
 }
 ^
compilation terminated.
/data/riotbuild/riotbase/Makefile.base:83: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.o' failed
make[4]: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/gnrc_netif_hdr/gnrc_netif_hdr_print.o] Error 1
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr' failed
make[3]: *** [ALL--/data/riotbuild/riotbase/sys/net/gnrc/netif/hdr] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys/net/gnrc' failed
make[2]: *** [ALL--/data/riotbuild/riotbase/sys/net/gnrc] Error 2
/data/riotbuild/riotbase/Makefile.base:20: recipe for target 'ALL--/data/riotbuild/riotbase/sys' failed
make[1]: *** [ALL--/data/riotbuild/riotbase/sys] Error 2
/data/riotbuild/riotbase/Makefile.include:449: recipe for target '/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a' failed
make: *** [/data/riotbuild/riotproject/examples/gnrc_networking/bin/native/application_gnrc_networking.a] Error 2
make: *** [/home/jcarrano/source/masterRIOT/makefiles/docker.inc.mk:106: ..in-docker-container] Error 2

With the fix it works. Squash and we merge.

@jcarrano
Copy link
Contributor

@cladmi While PRing #10492 I realized it may make more sense if you cherry-pick that commit into this PR (I couldn't figure out which PR depended on which, so for me that means they should go together.)

@cladmi
Copy link
Contributor Author

cladmi commented Dec 5, 2018

Rebased now that #10492 is merged

@cladmi
Copy link
Contributor Author

cladmi commented Dec 5, 2018

Build is now working in murdock, tell me when I should remove the test commits.

@jcarrano
Copy link
Contributor

jcarrano commented Dec 5, 2018

Remove the test commits and let's go forward.

@cladmi cladmi force-pushed the pr/make/clean_all_docker_container branch from 5e0a193 to c0d89ec Compare December 5, 2018 19:11
@cladmi cladmi removed the CI: needs squashing Commits in this PR need to be squashed; If set, CI systems will mark this PR as unmergable label Dec 5, 2018
@cladmi
Copy link
Contributor Author

cladmi commented Dec 5, 2018

Removed the test commit.

When building in parallel, clean is done at the same time as the
container starts building.

So the parallel `clean all` handling should also be done for the
`..in-docker-container` target.
@cladmi cladmi force-pushed the pr/make/clean_all_docker_container branch from c0d89ec to 135d2f7 Compare December 6, 2018 11:34
@cladmi
Copy link
Contributor Author

cladmi commented Dec 6, 2018

It failed because of the native test failure (now disabled) I just rebased to master to re-trigger.

@jcarrano jcarrano merged commit a38cb6c into RIOT-OS:master Dec 6, 2018
@cladmi cladmi deleted the pr/make/clean_all_docker_container branch December 6, 2018 15:05
@aabadie aabadie added this to the Release 2019.01 milestone Dec 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: build system Area: Build system Area: toolchain Area: toolchains; everything related to compilation, libc, linking, … CI: ready for build If set, CI server will compile all applications for all available boards for the labeled PR Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants