Feature Importance

The Shapley Axioms (S1–S4)

Shapley values are the unique solution that satisfies four core game-theoretic axioms, which describe how to fairly distribute the "total gain" of a model's prediction among the participating features.

1. Efficiency (Local Accuracy): The sum of the importance scores ${\displaystyle \phi _{j}}$ for all features must equal the difference between the model's prediction for an instance ${\displaystyle f(x)}$ and the expected prediction ${\displaystyle E[f(X)]}$.
2. Symmetry: If two features ${\displaystyle i}$ and ${\displaystyle j}$ contribute exactly the same value to every possible subset of other features, they must receive the same importance score: ${\displaystyle \phi _{i}=\phi _{j}}$.
3. Dummy (Null Player): If a feature contributes nothing to the value function for any subset of features, its importance score must be zero. This is crucial for identifying irrelevant features.
4. Additivity (Linearity): If the value function is the sum of two functions ${\displaystyle v=v_{1}+v_{2}}$, then the importance scores must be the sum of the scores calculated for each function: ${\displaystyle \phi _{j}(v)=\phi _{j}(v_{1})+\phi _{j}(v_{2})}$.

Axioms for Explaining the Data

While the Dummy (Null Player) axiom is desirable for obtaining feature importance scores that can be used to optimize a model, this is not the case for explaining the data, since variables may be highly correlated (and hence offer little value to the existing model), yet remain causally linked to the response variable. Alternative axioms have been proposed for feature importance methods that intend to explain the data.

• Blood Relation (Causal Graph Association): A feature should have non-zero importance if and only if the feature is blood related (associated) with the response in the ground-truth causal graph.
• Marginal Contribution: The importance of a feature must be at least as high as the gain it provides when added to the set of all other features.
• Elimination: Removing other features from the feature set can only decrease (or leave unchanged) the importance of a remaining feature. It cannot increase it.
• Invariance under Redundant Information and Symmetry under Duplication: Adding a redundant feature should not change the importance of preexisting features, and identical features should receive equal importance.

By Purpose

The choice of method is categorized based on the user's objective.

• Model Explanation: The goal is to understand the "logic" of a black-box model to ensure safety, fairness, and reliability. Methods like SHAP, LIME, and Accumulated local effects are standard here.
• Data Explanation (Scientific Inference): The goal is to learn about the real world. Researchers prioritize methods that handle redundancy and correlation in a way that reflects the true underlying relationships (e.g., MCI, UMFI).
• Model Optimization (Feature Selection): The goal is to improve the model's performance by removing irrelevant or redundant features. Techniques like Recursive Feature Elimination (RFE) use importance scores as a selection criterion.

By Scope

Feature importance methods have been classified into four distinct settings based on two axes: Global vs. Local and Data vs. Model.

Methods like SHAP and LIME are local-model. Global-model importance metrics include permutation importance and SAGE. Global-data methods include MCI and UMFI.

By Treatment of Correlated Features

Feature importance methods often differ in how they treat the correlation between features. In the presence of correlated features, predictive credit can be distributed in a variety of ways. Methods can therefore be classified based on how they assign credit to correlated features, where one end of the spectrum is given by Marginal feature importance, and the other is given by Conditional feature importance.

• Conditional feature importance: Evaluates a feature by conditioning on the values of all other features. This approach respects the dependence structure of the data and measures the unique information a feature provides that is not already captured by other variables. Methods in this category include conditional permutation importance, Leave-One-Covariate-Out, and partial correlation.

• Marginal feature importance: Relies on associations between the response and predictors, regardless of multicollinearity. Purely marginal methods assign the same high importance to all features. Examples include correlation, marginal contribution feature importance, and Ultra-marginal feature importance.

Methods such as SHAP and permutation importance are somewhere in between the two extremes, as importance is shared among correlated features.

Gradient vs. Non-Gradient

The technical implementation of the importance measure often dictates its applicability to different model architectures.

Other Classifications

Achen (1982) introduced a classification of linear regression-based feature importance methods: "dispersion importance" (explained variance), "level importance" (impact on the mean), and "theoretical importance" (the change in response for a given change in regressor).