Persistent Topological Features in Large Language Models - ICML 2025

Authors:

Yuri Gardinazzi, Karthik Viswanathan, Giada Panerai, Alessio Ansuini, Alberto Cazzaniga and Matteo Biagetti.

paper: link

With the following code you can reproduce the results of the paper and it is subdivided in four main parts inside the folder src

• representation: folder for the extraction of the hidden representations.
• zigzag :folder for the execution fo the ZigZag algorithm over the representations.
• benchmark: folder where it is possible to run the benchmarks with the different prunig methods.
  • Benchmark with lm-evaluation-harness.
  • Extraction of the blocks to cut with the Angular distance and Bi-Score are done with a modified version of short-transformers (to make it run also with Pyhia 6.9B).
• plots: folder where it is possible to reproduce the plots of the paper.

Environment setup

To install the library for FastZigZag refer to their paper and their github folder. Their code is not present in this repository due to their license which does not allow the redistribution of the software.

To install the rest of the environment conda env create -f environment.yml

LICENSE

This project is licensed under the MIT License - see the LICENSE file for details.