Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Hi!

Recently I checked Profile-Guided Optimization (PGO) improvements on multiple projects. The results are here. Since PGO shows improvements for many workloads, I think trying to optimize PuzzleFS with PGO can be a good idea. E.g. compression routines can be optimized with PGO since the official Zstd implementation supports PGO build.

I can suggest the following action points:

• Perform PGO benchmarks on PuzzleFS. If it shows improvements - add a note about possible improvements in PuzzleFS's performance with PGO to the documentation (README file, I guess).
• Providing an easier way (e.g. a build option) to build scripts with PGO can be useful for the end-users and maintainers since they will be able to optimize PuzzleFS according to their own workloads.

Here you can find examples of how PGO (and sometimes LLVM BOLT) is already integrated into different projects. Here are documentation examples with PGO in different projects.

Maybe testing Post-Link Optimization techniques (like LLVM BOLT) would be interesting too but I recommend starting from the usual PGO.

For the Rust projects, I suggest PGO optimizing with cargo-pgo (it supports LLVM BOLT too).

[ariel-miculas]

I'm wondering whether the zstd bindings should also support PGO or if this is not a necessary step for optimizing the Zstd compression.

For the Rust projects, I suggest PGO optimizing with cargo-pgo (it supports LLVM BOLT too).

Do you have an example of a Rust project that has support for PGO? I'm referring to the build option you've mentioned when you suggested to provide an easier way for end-users to optimize PuzzleFS.

[zamazan4ik]

I'm wondering whether the zstd bindings should also support PGO or if this is not a necessary step for optimizing the Zstd compression.

Since it's the binding library - yes, it needs special support. More discussions about that topic can be found here: Kobzol/cargo-pgo#38 . But it's true only for non-Rust dependencies - Rust dependencies and PuzzleFS own code can be optimized with PGO.

Do you have an example of a Rust project that has support for PGO? I'm referring to the build option you've mentioned when you suggested providing an easier way for end-users to optimize PuzzleFS.

Sure, I have some examples:

• Rustc: a CI script for the multi-stage build
• Pydantic-core: GitHub PR

Maybe examples from the non-Rust projects would be helpful too (since there is no huge difference in PGO implementation details between Rust and C/C++ projects):

• GCC:
  • Official docs, section "Building with profile feedback" (even AutoFDO build is supported)
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag