Add some (opt in) mechanism to share `--repository_cache` and `--repo_contents_cache`

[malt3]

I'm using rules_bazel_integration_test quite happily in rules_img, but one issue I'm seeing a lot is rate limiting with various APIs that are accessed in repo rules while running integration tests.

The reason this is happening is the use of TEST_TMPDIR, which silos many of Bazel's default paths:

https://github.com/bazelbuild/bazel/blob/5b14d04f6d74f5a7238110b38e1d338879e4a545/src/main/cpp/startup_options.cc#L645-L656

You can also see this in Bazel's logs:

$TEST_TMPDIR defined, some defaults will be overridden
...
repository_cache: /home/malte/.cache/bazel/_bazel_malte/cd535aca85f70194fa75f44025138530/sandbox/linux-sandbox/3126/execroot/_main/_tmp/4efa138a8530aba334adcf98db25fa9f/_bazel_malte/cache/repos/v1
...

What I would like to do is to add some configuration that allows users to share the repository cache and repo contents cache across integration tests if they like.
This obviously makes tests less hermetic, as some repository rules may use cached results instead of running from scratch, but if that's acceptable for your test case, it can greatly improve test speeds and avoid accidentally hammering public APIs.

[cgrindel]

What do you have in mind? The Bazel code that you linked seems to be focused on the user-specific output directory. Could you create a machine-specific repository cache and point to that in your .bazelrc files?

[malt3]

I can't find the exact place in the Bazel codebase where the repository cache is set, but I'm quite sure it is based on TEST_TMPDIR if set. I'll post it here.

The API I have in mind is either an environment variable that can be set via --test_env (or the env inherit mechanism) or a string_flag.

If the configuration is present, I simply want to set --repository_cache (and maybe also --repo_contents_cache) in the nested Bazel to the configured value(s).
It's quite clear that this may require extra setup (when sandboxing is used) and may not work at all under remote execution, but I think that this could be a feature that is useful in many setups.
Also note that the repo cache (and repo contents cache) were designed with shared access across multiple Bazel repos in mind.

[cgrindel]

I am open to adding a way to support this use case. Let me know when you have a PR, and I will be happy to review.

[layus]

I have modified caching_test_runner.sh to add caching:

# Configure caching
extra_args=()
bit_cache="${BIT_CACHE:-${BASH_SOURCE[0]%%/sandbox/*}/bit_cache}"
mkdir -p "${bit_cache}"

[[ -w "${bit_cache}" ]] || exit_with_msg "Cannot create bazel integration tests cache
    BIT_CACHE: $BIT_CACHE
    bit_cache: $bit_cache
    BASH_SOURCE: ${BASH_SOURCE[0]}
"

# BAZELISK_HOME
bazelisk_home="${bit_cache}/bazelisk"
mkdir -p "$bazelisk_home"
export BAZELISK_HOME="$bazelisk_home"
# --repository_cache
repository_cache="${bit_cache}/repository_cache"
mkdir -p "$repository_cache"
extra_args+=("--repository_cache" "$repository_cache")
# --disk_cache
disk_cache="${bit_cache}/disk_cache"
mkdir -p "$disk_cache"
extra_args+=("--disk_cache" "$disk_cache")
# -- repo_contents_cache
bazel_version=$("${bazel}" --version | cut -d ' ' -f 2)
if [[ -n "${bazel_version}" ]] && "${bazel}" info --repo_contents_cache= &>/dev/null ; then
  repo_contents_cache="${bit_cache}/repo_contents_cache/$bazel_version"
  mkdir -p "$repo_contents_cache"
  extra_args+=("--repo_contents_cache" "$repo_contents_cache")
fi

#...

  "${bazel}" "${cmd_parts[@]}" "${extra_args[@]}"

The code works well with processwrapper-sandbox, but not other sandbox kinds. Most probably because the directory is not writable by default in stricter sandboxes. Also, splitting on /sandbox/ can catch other sandboxes from outer containers.

Do you know of any way to find the real output_base from within tests ?

To be viable this script would also need a conversion to .bat.

Another thing I discovered is that --repository_cache sets --repo_contents_cache implicitly. But here I tried to have a different --repo_contents_cache per bazel version. Not sure if it is needed though.

As a bonus, these caches could be cached through https://github.com/bazel-contrib/setup-bazel or https://github.com/bazel-contrib/setup-bazelisk for CI runs. Although it may be argued that CI should exercise the whole pipeline.

But o be fair, this whole setup may be overkill. It's fairly trivial to define a custom runner by copy-pasting the default one and adding the flags you want there, as in agluszak/rules_kotlin@c8274ee#diff-e36db2dedb6510006222994dccc93ff4d62974b6a632ba44f0b9cd97f43fc025R34-R40.

So, the real benefit of such an upstream integration would be to automatically derive a good location for this cache (the output_base I think) and ensure that caches are aptly named to avoid invalidation churn.

[Vooblin]

If you want to use the default repository cache for integration tests and keep using default test runner, you can add this flag to your bazel commands:

DEFAULT_REPO_CACHE_FLAG = "--repository_cache=$(env -u TEST_TMPDIR ${BIT_BAZEL_BINARY} info repository_cache)"

default_test_runner(
    name = "simple_test_runner",
    bazel_cmds = [
        "info " + DEFAULT_REPO_CACHE_FLAG,
    ],
)