A Question About Influence Score

[youhanmo]

Hi Teams,

I'm reaching out to express my sincere appreciation for this wonderful library. Your efforts in creating and maintaining it are truly commendable. As someone not from the AI field, I've found it incredibly user-friendly and was able to successfully run the provided examples. This has been a great help to me, especially since in my domain (which is outside of AI), we use AI as a tool.

I have a couple of relatively basic questions, and I'd be extremely grateful if you could spare some time to address them.
First, I'm curious about the influence scores. Does a positive score imply that the corresponding training sample (let's call it A for simplicity) has a negative impact (or is harmful) on the prediction of given test sample (let's call it B for simplicity)?

Secondly, what if I apply this tool to data that wasn't part of the training set, say data C? Would the influence scores still hold meaning? Specifically, would the influence score accurately reflect whether data C is helpful in predicting the data sample B from the the test set?
Thank you so much for your time and consideration. I look forward to your insights.

Best regards

[sevendaystoglory]

For your first question. The answer is in how the influence score (the pairwise score) is computed in the library.
The library lets us define Task.compute_train_loss (on your point A) & Task.compute_measurement (on B).

They compute $\nabla f^T H^{-1} \nabla L$ by default (w/o the negative sign). This calculates the effect produced on the measurement of B, by down-weighting the sample A. In case of a high +ve score and if your measurement function (performance metric) $f$ is loss on B - this means that down-weighting A increases loss. Hence A itself is a helpful example.

Refer to #9.