Stop Asking if a Model Is Interpretable

Source: Towards Data Science

Interpretability in AI: Asking the Right Question

Researchers, practitioners, and even regulators often ask whether a model is interpretable. This framing assumes interpretability is a property a model either possesses or lacks – but it isn’t.

A model is not interpretable or uninterpretable in the abstract. Here we are not talking about inherently transparent models such as linear regression or decision trees, whose reasoning can be inspected directly. Instead, we are concerned with complex models whose decision processes are not immediately accessible.

Interpretability is therefore not a checkbox, a visualization, or a specific algorithm. It is better understood as a set of methods that allow humans to analyze models in order to answer particular questions. Change the question, and the usefulness of the explanation changes with it. The real issue, then, is not whether a model is interpretable, but what we need an explanation for.

Once we see interpretability this way, a clearer structure emerges. In practice, explanations consistently serve three distinct scientific functions:

1. Diagnosing failures
2. Validating learning
3. Extracting knowledge

These roles are conceptually different, even when they rely on similar techniques. Understanding that distinction helps clarify both when interpretability is necessary and what kind of explanation we actually need.

Interpretability as Diagnosis

The first role appears during model development, when models are still experimental objects. At this stage they are unstable, imperfect, and often wrong in ways that aggregate metrics cannot reveal. Accuracy tells us whether a model succeeds, but not why it fails. Two models can achieve identical performance while relying on entirely different decision rules—one may be learning real structure; another may be exploiting accidental correlations.

Interpretability methods let us look inside a model’s decision process and identify hidden failure modes. In this sense they play a role similar to debugging tools in software engineering. Without them, improving a model becomes largely guesswork; with them, we can formulate testable hypotheses about what the model is actually doing.

Example: Handwritten Digit Classification

The MNIST dataset is deliberately simple, making it ideal for checking whether a model’s reasoning aligns with our expectations.

Source: Towards Interaction Detection Using Topological Analysis on Neural Networks

When we visualize which pixels influenced a prediction, we can immediately see whether the network is focusing on the digit strokes or on irrelevant background regions. The difference tells us whether the model learned a meaningful signal or a shortcut. In this diagnostic role, explanations are not meant for end users or stakeholders; they are instruments for developers trying to understand model behavior.

Interpretability as Validation

Once a model performs well, the question changes. We are no longer primarily concerned with why it fails; instead, we want to know whether it succeeds for the right reasons.

This distinction is subtle but crucial. A system can achieve high accuracy and still be scientifically misleading if it relies on spurious correlations. For example, a classifier trained to detect animals might appear to work perfectly while actually relying on background cues rather than the animals themselves. From a predictive standpoint the model looks successful; from a scientific standpoint it has learned the wrong concept.

Interpretability allows us to inspect internal representations and verify whether they align with domain expectations. In deep neural networks, intermediate layers encode learned features, and analyzing those representations can reveal whether the system discovered meaningful structure or merely memorized superficial patterns.

Example: ImageNet Classification

ImageNet images contain cluttered scenes, diverse contexts, and high intra‑class variability, so successful models must learn hierarchical representations rather than rely on shallow visual cues.

Source: Grad‑CAM for image classification (PyTorch)

When we visualize internal filters or activation maps, we can check whether early layers detect edges, middle layers capture textures, and deeper layers respond to shapes. The presence of this structure suggests that the network has learned something meaningful about the data; its absence suggests that performance metrics may be hiding conceptual failure.

In this second role, interpretability is not debugging a broken model but validating a successful one.

Interpretability as Knowledge

The third role emerges when models are applied in domains where prediction alone is insufficient. Here, machine‑learning systems are used not just to produce outputs but to generate insights. Interpretability becomes a tool for discovery.

Modern models can detect statistical regularities across datasets far larger than any human could analyze manually. When we can inspect their reasoning, they may reveal patterns that suggest new hypotheses or previously unnoticed relationships. In scientific applications, this capability is often the most valuable outcome of interpretability research.

More valuable than predictive accuracy itself.

Medical Imaging Example

Consider a neural network trained to detect lung cancer from CT scans.

Source: “Secure and interpretable lung‑cancer prediction model using MapReduce, private blockchain, federated learning and XAI” – Nature article

If such a model predicts malignancy, clinicians need to understand which regions influenced that decision.

• If the highlighted regions correspond to a tumor boundary, the explanation aligns with medical reasoning.
• If they do not, the prediction cannot be trusted regardless of its accuracy.
• A third possibility: explanations may reveal subtle structures clinicians had not previously considered diagnostically relevant. In such cases, interpretability does more than justify a prediction—it contributes to knowledge.

Here, explanations are not just tools for understanding models; they are tools for extending human understanding.

One Concept, Three Functions

What these examples illustrate is that interpretability is not a single objective but a multi‑functional framework. The same technique can help:

1. Debug a model
2. Validate its reasoning
3. Extract insight

depending on the question being asked. Confusion about interpretability often arises because discussions fail to distinguish between these goals.

The more useful question is not whether a model is interpretable, but whether it is interpretable enough for the task we care about. That requirement always depends on context: development, research, or deployment.

Seen this way, interpretability is best understood not as a constraint on machine learning but as an interface between humans and models. It allows us to diagnose, validate, and learn. Without it, predictions remain opaque outputs; with it, they become objects of scientific analysis.

What exactly do we want the explanation to explain?

Once that question is clear, interpretability stops being a vague requirement and becomes a scientific tool.

---

You’re welcome to contact me if you have questions, want to share feedback, or simply feel like showcasing your own projects.