Adding a Python API

[fpedd]

Hi @mmoffitt,

Thanks for your excellent work on this allocator. Reading your paper was a lot of fun!

I wanted to experiment with your allocator, particularly to embed it in a larger compiler flow and benchmark it against other allocators I've been using. For this, I needed a Python interface (as many ML compiler workflows are Python-based), so I went ahead and created one using pybind11. You can find the initial version here. While it's still somewhat rough, I've ported all the tests to pytest and added examples and a Python CLI.

With this, one can now:

pip install minimalloc

and use it like this:

import minimalloc as mm
problem = mm.from_csv_file("benchmarks/examples/input.12.csv")
problem.capacity = 12
solver = mm.Solver()
solution = solver.solve(problem)
print(solution)  # Solution(offsets=[8, 8, 4, 4, 0])

Since the Python interface is just a wrapper, the performance is identical to the C++ interface. I've added a small script that uses both the C++ and Python CLIs to verify identical results and comparable performance:

(venv) minimalloc$ bash scripts/run-challenging-compare.sh 
benchmarks/challenging/A.1048576.csv 	[Python] Elapsed time: 4.011s 	[C++] Elapsed time: 4.046s 	PASS
benchmarks/challenging/B.1048576.csv 	[Python] Elapsed time: 2.791s 	[C++] Elapsed time: 2.776s 	PASS
benchmarks/challenging/C.1048576.csv 	[Python] Elapsed time: 0.528s 	[C++] Elapsed time: 0.540s 	PASS
benchmarks/challenging/D.1048576.csv 	[Python] Elapsed time: 2.769s 	[C++] Elapsed time: 2.757s 	PASS
benchmarks/challenging/E.1048576.csv 	[Python] Elapsed time: 2.849s 	[C++] Elapsed time: 2.859s 	PASS
benchmarks/challenging/F.1048576.csv 	[Python] Elapsed time: 4.609s 	[C++] Elapsed time: 4.641s 	PASS
benchmarks/challenging/G.1048576.csv 	[Python] Elapsed time: 0.977s 	[C++] Elapsed time: 0.975s 	PASS
benchmarks/challenging/H.1048576.csv 	[Python] Elapsed time: 1.407s 	[C++] Elapsed time: 1.421s 	PASS
benchmarks/challenging/I.1048576.csv 	[Python] Elapsed time: 3.026s 	[C++] Elapsed time: 3.031s 	PASS
benchmarks/challenging/J.1048576.csv 	[Python] Elapsed time: 1.087s 	[C++] Elapsed time: 1.093s 	PASS
benchmarks/challenging/K.1048576.csv 	[Python] Elapsed time: 0.425s 	[C++] Elapsed time: 0.430s 	PASS
All tests passed - Python and C++ APIs produced identical results

I wanted to ask whether you'd be interested in upstreaming this to your project. If so, I'd be happy to clean up the PR, split my changes into smaller, more reviewable PRs, and help maintain the Python bindings going forward.

This would enable many opportunities down the line, such as publishing to PyPI, making your work easily accessible to a much wider audience for integration into their compilers and experiments. For example, an ETH Zurich project called Deeploy has integrated your allocator in their Python code. But they currently have to go through CSV files.

Thanks again for your work and for making it available to the community.

Best,
Fabian

[mmoffitt]

This sounds fantastic, I love it. Let me know if there's anything I can do to help get this idea off the ground!

[fpedd]

Hey @mmoffitt,

Thanks a lot for merging the changes and for being so welcoming to contributions!

In regard to your comments here #16 (comment): With these changes in, we can now build and use the Python package locally. This enables people to pip install minimalloc/, but only if they have previously downloaded/cloned the repo git clone --recursive git@github.com:google/minimalloc.git. If we wanted to make this a true 'hands off' experience, i.e. enable people to "just" pip install minimalloc, we'd need to publish the package on PyPI. If you are okay with that, I am happy to look in to it. There are some minor additions needed to the pyproject.toml, but the rest should be pretty straightforward.

I also wanted to ask you if you'd be interested in setting up some github actions, such that the tests and some formatting checks can be run automatically on new PRs. If we set this up, we could also automate the PyPI publishing, similar to how they have done here https://github.com/google/tunix/blob/main/.github/workflows/pypi_release.yml, for example.

[fpedd]

I went ahead and did a test upload to TestPyPI https://test.pypi.org/project/minimalloc/0.1.0/. Let me know what you think. With this, you can now do a truly "hands-off" pip install without having to clone this repo. Just do pip install -i https://test.pypi.org/simple/ minimalloc (the -i ... is still needed because we have only published to the test index, not the real one). If you are okay with this, then we can do the same to the official PyPI. If not, I will delete the TestPyPI entry again.

[mmoffitt]

Sure -- as long as you don't mind doing this, I think it sounds great!