Notes On Feature Explainability

Shap - https://github.com/slundberg/shap

Shap supports model level feature importance, and single prediction level importance. (This can be a little confusing depending on how you call the explainer, it may print out a different display.) Generally the plot function is shap.plot.force.

This is a good explanation for how Shap works under the hood: https://medium.com/@gabrieltseng/interpreting-complex-models-with-shap-values-1c187db6ec83

As for features of the library, this notebook from the doc is a pretty good showcase as well: https://slundberg.github.io/shap/notebooks/tree_explainer/Census%20income%20classification%20with%20LightGBM.html

From what I can tell in the library, you'll want to implement shap at the model level here: https://github.com/unmtransinfo/ProteinGraphML/blob/master/ProteinGraphML/MLTools/Models/models.py, since you need access to the model object itself. (via the xgboost package)

For XGBoost model level features: I'd definitely compare features that come up in these measures to ensure you have consistent model level features, these would be https://xgboost.readthedocs.io/en/latest/python/python_api.html#xgboost.Booster.get_score