Previously, The Role of Attention for TinyZero Countdown found verification heads with a crude heuristic (attention heads that attend "non-trivially" to previous occurrence of correct token).
Can we localize this further? ie, what is the minimum number of attention heads we can prune out and get the same effect?
One way we can do this is to sort these attention heads based on how much they affect the activation of our MLP value vectors.
Then, we iteratively start pruning out attention heads based on this metric.
Resulting plot:
x-axis: # of attention heads pruned out
y-axis: intervention success rate.
Currently we have one data point -- pruning out 27 heads leads to perfect intervention success rate.
Relevant code: https://github.com/ajyl/verify_circuit/blob/main/notebooks/eval_interv.sync.ipynb
Also, how similar are the "O-vector" of these attention heads? What are their cosine similarity scores?
Previously, The Role of Attention for TinyZero Countdown found verification heads with a crude heuristic (attention heads that attend "non-trivially" to previous occurrence of correct token).
Can we localize this further? ie, what is the minimum number of attention heads we can prune out and get the same effect?
One way we can do this is to sort these attention heads based on how much they affect the activation of our MLP value vectors.
Then, we iteratively start pruning out attention heads based on this metric.
Resulting plot:
x-axis: # of attention heads pruned out
y-axis: intervention success rate.
Currently we have one data point -- pruning out 27 heads leads to perfect intervention success rate.
Relevant code: https://github.com/ajyl/verify_circuit/blob/main/notebooks/eval_interv.sync.ipynb
Also, how similar are the "O-vector" of these attention heads? What are their cosine similarity scores?