Unable to use ccache anymore

[jbleonesio]

Since migrating from BuildStream 1.6.8 to 2.0.1 files created during a "bst build" step are not kept, even using a workspace.

As a result builds using ccache with a CCACHE_DIR contained inside the workspace directory are silently slowed down because they keep caching compilation objects without being able to take advantage of them in a later compilation.

I created a fully reproducible example of this behavior difference between bst 1.6.8 and bst 2.0.1 here : https://github.com/jbleonesio/bst_build_in_workspace_tests

flatpak-builder has a --ccache option flag dedicated to this purpose. Is there any way to achieve a similar goal with BuildStream and it is documented ?

For information, the CI/CD pipeline duration of an open-source project that I maintain went from ~8min to ~20min for that reason.

[jbleonesio]

I ended up using recc by following @gtristan POC (see implementation here).

[nanonyme]

@gtristan this now feels not great if downstreams have already started patching BuildStream to get casd socket in. Please resume #1772 and let's formalize sandbox configuration that allows the socket in.

[nanonyme]

@jbleonesio do you happen to have any performance metrics after switching to recc?

[jbleonesio]

@nanonyme After switching to recc I managed to get the pipeline to complete in ~4min30 (but in the meantime I migrated from BuildStream 1.6.8 to 2.0.1, from Freedesktop SDK 22.08 to 23.08 and I also simplified some build scripts).

[nanonyme]

I am expecting we will start providing recc as a freedesktop-sdk element as soon as BuildStream lands support for this socket handling properly.

[nanonyme]

Well, we did land recc in freedesktop-sdk 24.08 (not yet released). The BuildStream-side change is still disappointingly absent.

[jbleonesio]

I'm now using recc from freedesktop-sdk 24.08 instead of building it on my own.

Since BuildStream 2.3.0, the patching needs to be updated to :

sed '135 i \            buildbox_command.append("--bind-mount={}:/tmp/casd.sock".format(casd._socket_path))\n' -i /path/to/_sandboxbuildboxrun.py

Still looking forward to have all of this officially supported.

[nanonyme]

We had some discussion about this and @juergbi was hesitant towards the approach since it may expose entire LocalCAS to sandbox.

[jbleonesio]

I do not know enough BuildStream internals to suggest nor appreciate a better design but there should be a supported and documented way to cache previously compiled objects IMHO.

Having a compilation time being 4x longer than what it should be (and that's just my own figures) could sadly turn users away from this really powerful and flexible project.

[nanonyme]

Some relevant discussion here #1945 (comment)

[nanonyme]

I do not know enough BuildStream internals to suggest nor appreciate a better design but there should be a supported and documented way to cache previously compiled objects IMHO.

Having a compilation time being 4x longer than what it should be (and that's just my own figures) could sadly turn users away from this really powerful and flexible project.

I hear you. Just thought to mention it looks like it's not as simple as just merging the proposed change.

[juergbi]

With BuildStream 2.6, recc can now be used and this use case is also covered by integration tests. See #1751 and linked issues and PRs.