Flip incompatible_strict_action_env#26587
Conversation
|
I guess now I need someone from the bazel team to trigger https://github.com/bazelbuild/continuous-integration/blob/337638d2fcff6d86b66035901c09d37c9199b38a/buildkite/bazelci.py#L99 so that we can see all downstream projects. |
hofbi
force-pushed
the
flip-incompatible_strict_action_env
branch
2 times, most recently
from
05a780e to
fe4238b
Compare
|
I started a downstream run at https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4824 |
|
I also sent a PR to add the flag to the BCR pipeline: bazelbuild/bazel-central-registry#5319 |
Thanks @fmeum. Do we actually have a baseline for the downstream job? I was looking at a few of the failures showing up and a significant number fails because of |
|
I don't see any failures that I would attribute to the flip, but it's true that downstream is in bad shape. That makes the signal worse. Ideally we would be able to run a downstream test with Bazel 8 and the flag flipped instead, which should avoid most of the preexisting failures. @fweikert Is that possible? |
|
@fweikert ping |
|
I re-opened #6648 and added But based on historical experience, this one is very hard to flip as many actions do depend on the value of |
|
Thanks. Let's see what we get tomorrow.
Depending how many fixed we would have to do, how about setting |
Yeah, this would work for leaf projects, but if a common dependency actually needs this flag, we'll still need to fix it and migrate the ecosystem first. Let's see the result, I started a manual run at https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/492 |
|
Looks like only rules_python might be affected on Windows, can you please take a quick look? If it's not serious, I'm fine with moving this forward and fix new breakages as we discover them. |
|
Created bazel-contrib/rules_python#3160 to reproduce this on a PR |
|
@meteorcloudy I tried to reproduce the windows error in bazel-contrib/rules_python#3160, but on rules_python CI this job passes. See https://buildkite.com/bazel/rules-python-python/builds/12739/steps/canvas?sid=0198f0e1-0e1a-4f05-8f7e-2769e21a7640#0198f0e1-0f47-4ee1-9940-61d1a48af7db/278-292 where the flag is set on the windows CI. Do you have any idea why the windows build of https://buildkite.com/bazel/bcr-bazel-compatibility-test/builds/492 could have failed? Are they running on different windows environments? |
|
I think the only difference is that in the rules_python pipeline, it's running again Bazel 7.6.1, however in the BCR pipeline, we are testing against 8.3.1. Can you try upgrade Bazel and see if it's reproducible? |
hofbi
force-pushed
the
flip-incompatible_strict_action_env
branch
from
fe4238b to
453399e
Compare
github-actions
bot
added
team-Remote-Exec
|
bazel-contrib/rules_python#3160 is a fix for rules_python |
|
@bazel-io fork 9.0.0 |
github-actions
bot
removed
the
awaiting-review
Work towards bazel-contrib/SIG-rules-authors#42. Fixes bazelbuild#7026 Closes bazelbuild#26587. PiperOrigin-RevId: 831944371 Change-Id: I397faa463a709b2367d5a6eebc678077c4029bbb
Work towards bazel-contrib/SIG-rules-authors#42. Fixes #7026 Closes #26587. PiperOrigin-RevId: 831944371 Change-Id: I397faa463a709b2367d5a6eebc678077c4029bbb Commit e697ff7 Co-authored-by: Markus Hofbauer <markus.hofbauer@flyzipline.com>
…103) We only need that for less than Bazel 9 due to bazelbuild/bazel#26587
Bazel 9 flipped `--incompatible_strict_action_env` to `true` by default (bazelbuild/bazel#26587), which means tests no longer inherit `PATH` from the host environment. This breaks subprocess calls that rely on `PATH` lookup: ``` ==================== Test output for //distro:packaging_test: E. ====================================================================== ERROR: testBuild (__main__.PackagingTest.testBuild) ---------------------------------------------------------------------- Traceback (most recent call last): [...] build_result = subprocess.check_output(['bazel', 'build', '--enable_workspace', ':dummy_tar']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] FileNotFoundError: [Errno 2] No such file or directory: 'bazel' ``` This change therefore adds `env_inherit = ["PATH"]` to that very test, which matches the behavior of earlier Bazel versions and is consistent with the presence of a "noci" tag. Additionally, Bazel 9 removed `WORKSPACE` support entirely (bazelbuild/bazel#26131), requiring `bzlmod` with `MODULE.bazel`: ``` ==================== Test output for //distro:packaging_test: [...] WARNING: --enable_bzlmod is set, but no MODULE.bazel file was found at the workspace root. Bazel will create an empty MODULE.bazel file. Please consider migrating your external dependencies from WORKSPACE to MODULE.bazel. For more details, please refer to bazelbuild/bazel#18958. [...] ERROR: error loading package '': Unable to find package for @@[unknown repo 'not_named_rules_pkg' requested from @@]//pkg:tar.bzl: The repository '@@[unknown repo 'not_named_rules_pkg' requested from @@]' could not be resolved: No repository visible as '@not_named_rules_pkg' from main repository. [...] ERROR: Build did NOT complete successfully E. ====================================================================== ERROR: testBuild (__main__.PackagingTest.testBuild) ---------------------------------------------------------------------- [...] build_result = subprocess.check_output(['bazel', 'build'] + bazel_flags + [':dummy_tar']) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [...] subprocess.CalledProcessError: Command '['bazel', 'build', '--enable_workspace', ':dummy_tar']' returned non-zero exit status 1. ``` The test now detects the Bazel version at runtime and generates the appropriate setup: - Bazel 9+: `MODULE.bazel`, with `module` and `archive_override`, - earlier versions, as before: `WORKSPACE`, with `http_archive` and `--enable_workspace` flag. For good measure, the change also addresses a leftover TODO by replacing the `tar` subprocess invocation with pure Python code.
Bumping bazel_worker_api to v0.0.10 resolves bazelbuild/bazel-worker-api#21 (see also bazelbuild/bazel#28692): ```txt ERROR: external/bazel_tools/tools/build_defs/repo/http.bzl:200:45: An error occurred during the fetch of repository 'bazel_worker_api+': Traceback (most recent call last): File "external/bazel_tools/tools/build_defs/repo/http.bzl", line 200, column 45, in _http_archive_impl download_info = ctx.download_and_extract( Error in download_and_extract: java.io.IOException: Error extracting external/bazel_worker_api+/temp685257769505461943/bazel-worker-api-v0.0.6.tar.gz to external/bazel_worker_api+/temp685257769505461943: Tar entries cannot refer to files outside of their directory: external/bazel_worker_api+/temp685257769505461943/bazel-worker-api-v0.0.6.tar.gz has a link bazel-worker-api-0.0.6/proto/.bazelversion pointing to ../.bazelversion ``` Regarding the `test_framework.sh` breakage on macOS, Bazel 9.0.0 flipped `--incompatible_strict_action_env` to `true`. - https://bazel.build/versions/8.5.0/reference/command-line-reference#flag--incompatible_strict_action_env - https://bazel.build/versions/9.0.0/reference/command-line-reference#flag--incompatible_strict_action_env - bazelbuild/bazel#26587 - bazelbuild/bazel@60c5a03 This caused `test_framework.sh` to use `/bin/bash`, instead of any newer Bash on the user's `PATH`. On macOS, `/bin/bash` is version 3.2.57(1)-release, and it failed with an unbound variable error when the `SKIPPED` array was empty: ```txt ../rules_magic+/test/test_framework.sh: line 176: SKIPPED[@]: unbound variable ``` Bash 4.4 fixes the empty array behavior, which is why the error didn't appear earlier while testing with Bash 5 in the `PATH`: - https://cgit.git.savannah.gnu.org/cgit/bash.git/tree/CHANGES?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n878 - https://stackoverflow.com/a/7577209 The workaround was to check the length of `SKIPPED` before expanding it as part of an argument list to a function call.
Bumping bazel_worker_api to v0.0.10 resolves bazelbuild/bazel-worker-api#21 (see also bazelbuild/bazel#28692): ```txt ERROR: external/bazel_tools/tools/build_defs/repo/http.bzl:200:45: An error occurred during the fetch of repository 'bazel_worker_api+': Traceback (most recent call last): File "external/bazel_tools/tools/build_defs/repo/http.bzl", line 200, column 45, in _http_archive_impl download_info = ctx.download_and_extract( Error in download_and_extract: java.io.IOException: Error extracting external/bazel_worker_api+/temp685257769505461943/bazel-worker-api-v0.0.6.tar.gz to external/bazel_worker_api+/temp685257769505461943: Tar entries cannot refer to files outside of their directory: external/bazel_worker_api+/temp685257769505461943/bazel-worker-api-v0.0.6.tar.gz has a link bazel-worker-api-0.0.6/proto/.bazelversion pointing to ../.bazelversion ``` Regarding the `test_framework.sh` breakage on macOS, Bazel 9.0.0 flipped `--incompatible_strict_action_env` to `true`. - https://github.com/bazelbuild/bazel/releases/tag/9.0.0 - bazelbuild/bazel#26587 - bazelbuild/bazel@60c5a03 - https://bazel.build/versions/9.0.0/reference/command-line-reference#flag--incompatible_strict_action_env - https://bazel.build/versions/8.5.0/reference/command-line-reference#flag--incompatible_strict_action_env This caused `test_framework.sh` to use `/bin/bash`, instead of any newer Bash on the user's `PATH`. On macOS, `/bin/bash` is version 3.2.57(1)-release, and it failed with an unbound variable error when the `SKIPPED` array was empty: ```txt ../rules_magic+/test/test_framework.sh: line 176: SKIPPED[@]: unbound variable ``` Bash 4.4 fixes the empty array behavior, which is why the error didn't appear earlier while testing with Bash 5 in the `PATH`: - https://cgit.git.savannah.gnu.org/cgit/bash.git/tree/CHANGES?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n878 - https://stackoverflow.com/a/7577209 Temporarily setting `--noincompatible_strict_action_env` in `.bazelrc` indeed resolved the error. The permanent workaround is to check the length of `SKIPPED` before expanding it as part of an argument list to a function call.
`test_framework.sh` now checks the length of the `SKIPPED` array before expanding it as part of an argument list to a function call. This fixes the following unbound variable error for an empty array access on macOS, whereby `/bin/bash` is version 3.2.57(1)-release. ```txt ../rules_magic+/test/test_framework.sh: line 176: SKIPPED[@]: unbound variable ``` Bazel 9.0.0 flips `--incompatible_strict_action_env` to `true`, causing `test_framework.sh` to use `/bin/bash`, instead of Bash from the user's `PATH`. - https://github.com/bazelbuild/bazel/releases/tag/9.0.0 - bazelbuild/bazel#26587 - bazelbuild/bazel@60c5a03 - https://bazel.build/versions/9.0.0/reference/command-line-reference#flag--incompatible_strict_action_env - https://bazel.build/versions/8.5.0/reference/command-line-reference#flag--incompatible_strict_action_env Bash 4.4 fixes the empty array behavior, which is why the error didn't appear earlier while testing with Bash 5 in the `PATH`: - https://cgit.git.savannah.gnu.org/cgit/bash.git/tree/CHANGES?id=3ba697465bc74fab513a26dea700cc82e9f4724e#n878 - https://stackoverflow.com/a/7577209 Setting `--noincompatible_strict_action_env` in `.bazelrc` indeed resolved the error, but the fix in this commit is a better permanent solution.
3 participants
Work towards bazel-contrib/SIG-rules-authors#42.
Fixes #7026