Causal explanations

We humans reason and understand in complementary and complex ways, and we think both fast and slow. When thinking slow, we reference previous evidence in our memory, and remember similar cases to the one we are dealing with; we reason logically, sometimes by first principles, trying to build a theory of the world, or directly applying said principles to the world; we often infer causal relationships between objects, allowing us to act purposefully to achieve our goals. These inherently human skills are usually lost both on the black box, which rarely relies on deliberate slow reasoning in favor of fast pattern matching, and on the explanations themselves, which rarely present the user with the tools to properly understand. Conversely, in fast thinking, we leverage heuristics, stereotypes, and approximations, and aim to get a “good enough” result while thinking the least amount possible. Unsurprisingly, fast thinking is a primary source of undesired outcomes and biased results. In XAI, we find this dichotomy in the different languages expressing explanations: fast thinking is predicated upon simple languages for fast and approximate understanding, such as feature importance, and prototypes, while slow thinking involves more complex languages and structures, such as logical theories, knowledge bases, and causal models. In this research line, we aim to integrate slow thinking along three different and possibly complementary directions, namely causality, knowledge injection, and logical reasoning. Orthogonally to these directions, we aim to target slow thinking both internally to the explanation algorithm, that is, to have the explanation algorithm itself think slowly, and by design, that is, to have the black box itself thinking slowly.

2.1 Causality

Inferring a causal model is central to properly understanding a phenomenon. Following [6], a synergy paper with the HumaneAI project, where we survey the causal literature, we identify two families of causal models: graphical models, in which we infer the causal relationships and induced distributions of observed variables, and potential outcome models, in which we assume observed variables to be the outcome of a causal model, and we look to infer the counterfactual outcome of an intervention in the model. Graphical models encode variables and their conditional dependency relations, allowing us to understand what variables influence others. Pearl’s do-calculus introduced a formal calculus for intervention on causal models, allowing their users to purposefully act on the data knowing what each action will result into. Inferring a causal model is of benefit both for the user, who can test interventional actions, and to the black box, that can leverage it to perform better predictions. Explanation algorithms can leverage causal models for better feature importance computation, as it is the case for our proposal CALIME [CG21], in which we learn a causal model to infer feature importance in a principled way. We detail our work in Attachment A.1.3

2.2 Knowledge integration

Modern black boxes tend to rely on neural and subsymbolic approaches that are in stark contrast with human knowledge, which is usually symbolic in nature. The XAI community has shown an increasing interest both in symbolic knowledge injection in subsymbolic models [7], and more generally in neuro-symbolic integration. This trend is of great interest for domains with large knowledge bases, such as healthcare and Natural Language Processing (NLP) [8]. Several NLP tasks can leverage external structured and unstructured knowledge in the form of structured knowledge bases [9], e.g., Wikipedia, or free-form text [10]. This allows the models to leverage a set of relevant facts in the knowledge base, and provide them to the user to explain its reasoning. Some recent approaches go as far as using the whole live and raw web as a knowledge base, and search through it for useful facts to aid the prediction. Aside from injection, background knowledge can also be used post-hoc to align the black box learned concepts with given concepts. Besides a review of the literature, in this stream of research we have proposed two works: Doctor XAI [PPP2020], already presented in Section 1.3, and TriplEx [SMM2022].

2.2.1 TriplEx

In [SMM2022] we have developed TriplEx, an algorithm for explanation of Transformer-based models. TriplEx aims to locally explain text classification models on a plethora of tasks: natural language inference, semantic text similarity, and text classification. Given some text x to classify, TriplEx extracts a set of factual triples T, which form the basis of the explanation. Then, TriplEx looks for perturbations of T along given semantic dimensions, which vary according to the task at hand, to look for edge cases in which the black box’s prediction is preserved. In other words, TriplEx looks to generate a semi-factual explanation. The search for perturbations is guided by an external knowledge base, specifically WordNet, that allows TriplEx to perturb text along different semantic dimensions. Keeping with our running example, TriplEx may perturb “mice” and replace it with “rodent” to verify whether the model has learned to apply the same reasoning with all rodents, and not just mice. Finally, TriplEx ranks the label-preserving perturbations according to their semantic distance: the larger the semantic perturbation, the better. Additionally, for Transformer models, TriplEx also provides an alignment score of each triple, indicating what triple is more relevant for the black box, allowing the user to have a finer granularity of explanation. TriplEx extracts explanations which are correct by construction, and semantic perturbations tend to be realistic and plausible, as measured by perplexity, an automatic evaluation of the plausibility of some text. Semantic perturbations retain realistic text, indicating that leveraging semantic perturbations does indeed generate realistic explanations.

2.3 Logic reasoning

Logic is one of the most powerful languages to express slow thinking, as it enjoys several desirable properties. Logic programming allows us to induce discrete, noise-resistant, and explainable/declarative by design “programs as rules” with high levels of abstraction that mimics human reasoning. Derivations in logic yield deterministic proof trees that a user can inspect. Furthermore, logic programming lends itself to background knowledge injection, allowing the user to guide the model, even if partially, with concepts and theories that they already know to be true. These properties make it a perfect candidate language for slow thinking explanations. Statistical Relational Learning (STAR) aims to integrate logics, and by and large relational learning, and statistical learning. Some models, for instance, integrate a subsymbolic component, given by a black box, and a symbolic one, given by a logical theory, in an explainable by-design pipeline in which the black box is only tasked with learning a mapping from data to logical entities, and the logical theory is tasked with reasoning on top of the entities. An even tighter integration is presented by models directly encoding logic theories and predicates in subsymbolic structures, which often map logic connectives and quantifiers to predefined norms. Other works aim to constrain black box models with given knowledge in the form of First-Order Rules, or to extract a set of logical constraints learned by the black box. Our core approaches (LORE, GLocalX) are essentially logic-based, since they produce explanations in the form of rules (either directly inferred or as the result of abstracting sets of rules), and therefore it is natural to consider the surveyed logic-based approaches as candidates for extending the expressiveness of the explanation language of LORE and of the rule reasoning approach of GLocalX

Research line people

Franco
Turini

Full Professor
University of Pisa

R.LINE 1 ▪ 2 ▪ 5

Salvatore
Ruggieri

Full Professor
University of Pisa

R.LINE 1 ▪ 2

Mattia
Setzu

Phd Student
University of Pisa

R.LINE 1 ▪ 2

Carlo
Metta

Researcher
ISTI - CNR Pisa

R.LINE 1 ▪ 2 ▪ 3 ▪4

Isacco
Beretta

Phd Student
University of Pisa

R.LINE 2

Marta
Marchiori Manerba

Phd Student
University of Pisa

R.LINE 1 ▪ 2 ▪ 5

Michele
Fontana

Phd Student
University of Pisa

R.LINE 2

Martina
Cinquini

Phd Student
University of Pisa

R.LINE 1 ▪ 2

Chandana
Sree Mala

Phd Student
Scuola Normale

R.LINE 2

Stefano
Marmi

Professor
Scuola Normale

R.LINE 1 ▪ 2

Gabriele
Barlacchi

Phd Student
Scuola Normale

R.LINE 1 ▪ 2

Iacopo
Colombini

Phd Student
Scuola Normale

R.LINE 2 ▪ 4

Paolo Maria
Mancarella

Full Professor
University of Pisa

R.LINE 2

Line 2 - Publications

2025

A Practical Approach to Causal Inference over Time

Martina Cinquini, Isacco Beretta, Salvatore Ruggieri, and Isabel Valera

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

RESEARCH LINE
2

Abs Bib HTML

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting causal VAR framework allows us to perform causal inference over time from observational time series data. Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.
@article{CBR2025, author = {Cinquini, Martina and Beretta, Isacco and Ruggieri, Salvatore and Valera, Isabel}, doi = {10.1609/aaai.v39i14.33626}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {2}, month = apr, number = {14}, open_access = {Gold}, pages = {14832–14839}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {A Practical Approach to Causal Inference over Time}, visible_on_website = {YES}, volume = {39}, year = {2025} }
A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods

Martina Cinquini, Karima Makhlouf, Sami Zhioua, Catuscia Palamidessi, and Riccardo Guidotti

IEEE Access, Apr 2025

RESEARCH LINE
2

5

Abs Bib HTML

Causal discovery (CD) algorithms are increasingly applied to socially and ethically sensitive domains. However, their evaluation under realistic conditions remains challenging due to the scarcity of real-world datasets annotated with ground-truth causal structures. Whereas synthetic data generators support controlled benchmarking, they often overlook forms of bias, such as dependencies involving sensitive attributes, which may significantly affect the observed distribution and compromise the trustworthiness of downstream analysis. This paper introduces a novel synthetic data generation framework that enables controlled bias injection while preserving the causal relationships specified in a ground-truth causal graph. The framework aims to evaluate the reliability of CD methods by examining the impact of varying bias levels and outcome binarization thresholds. Experimental results show that even moderate bias levels can lead CD approaches to fail to correctly infer causal links, particularly those connecting sensitive attributes to decision outcomes. These findings underscore the need for expert validation and highlight the limitations of current CD methods in fairness-critical applications. Our proposal thus provides an essential tool for benchmarking and improving CD algorithms in biased, real-world data settings.
@article{CMZ2025, author = {Cinquini, Martina and Makhlouf, Karima and Zhioua, Sami and Palamidessi, Catuscia and Guidotti, Riccardo}, doi = {10.1109/access.2025.3573201}, issn = {2169-3536}, journal = {IEEE Access}, line = {2,5}, open_access = {Gold}, pages = {97376–97391}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods}, visible_on_website = {YES}, volume = {13}, year = {2025} }
Categorical Explaining Functors: Ensuring Coherence in Logical Explanations

Stefano Fioravanti, Francesco Giannini, Pietro Barbiero, Paolo Frazzetto, Roberto Confalonieri, and 2 more authors

In Proceedings of the TwentySecond International Conference on Principles of Knowledge Representation and Reasoning , Nov 2025

RESEARCH LINE
1

2

Abs Bib HTML

The most common methods in explainable artificial intelligence are post-hoc techniques which identify the most relevant features used by pretrained opaque models. Some of the most advanced post-hoc methods can generate explanations that account for the mutual interactions of input features in the form of logic rules. However, these methods frequently fail to guarantee the consistency of the extracted explanations with the model’s underlying reasoning. To bridge this gap, we propose a theoretically grounded approach to ensure coherence and fidelity of the extracted explanations, moving beyond the limitations of current heuristic-based approaches. To this end, drawing from category theory, we introduce an explaining functor which structurally preserves logical entailment between the explanation and the opaque model’s reasoning. As a proof of concept, we validate the proposed theoretical constructions on a synthetic benchmark verifying how the proposed approach significantly mitigates the generation of contradictory or unfaithful explanations. (arXiv)
@inproceedings{FGB2025, address = {California}, author = {Fioravanti, Stefano and Giannini, Francesco and Barbiero, Pietro and Frazzetto, Paolo and Confalonieri, Roberto and Zanasi, Fabio and Navarin, Nicolò}, booktitle = {Proceedings of the TwentySecond International Conference on Principles of Knowledge Representation and Reasoning}, collection = {KR-2025}, doi = {10.24963/kr.2025/30}, issn = {978-1-956792-08-9}, line = {1,2}, month = nov, open_access = {Gold}, pages = {306–315}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, series = {KR-2025}, title = {Categorical Explaining Functors: Ensuring Coherence in Logical Explanations}, visible_on_website = {YES}, year = {2025} }
Group Explainability Through Local Approximation

Mattia Setzu, Riccardo Guidotti, Dino Pedreschi, and Fosca Giannotti

Oct 2025

RESEARCH LINE
2

Abs Bib HTML

Machine learning models are becoming increasingly complex and widely adopted. Interpretable machine learning allows us to not only make predictions but also understand the rationale behind automated decisions through explanations. Explanations are typically characterized by their scope: local explanations are generated by local surrogate models for specific instances, while global explanations aim to approximate the behavior of the entire black-box model. In this paper, we break this dichotomy of locality to explore an underexamined area that lies between these two extremes: meso-level explanations. The goal of meso-level explainability is to provide explanations using a set of meso-level interpretable models, which capture patterns at an intermediate level of abstraction. To this end, we propose GrouX, an explainable-by-design algorithm that generates meso-level explanations in the form of feature importance scores. Our approach includes a partitioning phase that identifies meso groups, followed by the training of interpretable models within each group. We evaluate GrouX on a collection of tabular datasets, reporting both the accuracy and complexity of the resulting meso models, and compare it against other meso-level explainability algorithms. Additionally, we analyze the algorithm’s sensitivity to its hyperparameters to better understand its behavior and robustness.
@inbook{SGP2025, address = {ECAI 2025}, author = {Setzu, Mattia and Guidotti, Riccardo and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {ECAI 2025}, doi = {10.3233/faia250902}, isbn = {9781643686318}, issn = {1879-8314}, line = {2}, month = oct, open_access = {Gold}, pages = {952 - 958}, publisher = {IOS Press}, title = {Group Explainability Through Local Approximation}, visible_on_website = {YES}, year = {2025} }
Human-AI coevolution

Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-László Barabási, and 12 more authors

Artificial Intelligence, Feb 2025

RESEARCH LINE
2

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of black box models in classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. Moreover, diversity of counter-exemplars is modeled as an optimization problem, solved by a greedy algorithm with theoretical guarantee. We report experiments on three datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, diversity, and usefulness, and that is comparable to it in terms of stability.
@article{Pedreschi_2025, address = {Netherlands}, author = {Pedreschi, Dino and Pappalardo, Luca and Ferragina, Emanuele and Baeza-Yates, Ricardo and Barabási, Albert-László and Dignum, Frank and Dignum, Virginia and Eliassi-Rad, Tina and Giannotti, Fosca and Kertész, János and Knott, Alistair and Ioannidis, Yannis and Lukowicz, Paul and Passarella, Andrea and Pentland, Alex Sandy and Shawe-Taylor, John and Vespignani, Alessandro}, doi = {10.1016/j.artint.2024.104244}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {2}, month = feb, open_access = {Gold}, pages = {104244}, publisher = {Elsevier BV}, title = {Human-AI coevolution}, visible_on_website = {YES}, volume = {339}, year = {2025} }

Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts

Andrea Pugnana, Riccardo Massidda, Francesco Giannini, Pietro Barbiero, Mateo Espinosa Zarlenga, and 4 more authors

Dec 2025

RESEARCH LINE

Bib

@misc{PMG2025,
  author = {Pugnana, Andrea and Massidda, Riccardo and Giannini, Francesco and Barbiero, Pietro and Zarlenga, Mateo Espinosa and Pellungrini, Roberto and Dominici, Gabriele and Giannotti, Fosca and Bacciu, Davide},
  line = {1,2},
  month = dec,
  title = {Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts},
  year = {2025}
}

DeepProofLog: Efficient Proving in Deep Stochastic Logic Programs

Ying Jiao, Rodrigo Castellano Ontiveros, Luc De Raedt, Marco Gori, Francesco Giannini, and 2 more authors

Dec 2025

RESEARCH LINE

Bib

@misc{JCD2025,
  author = {Jiao, Ying and Ontiveros, Rodrigo Castellano and Raedt, Luc De and Gori, Marco and Giannini, Francesco and Diligenti, Michelangelo and Marra, Giuseppe},
  line = {2},
  month = dec,
  title = {DeepProofLog: Efficient Proving in Deep Stochastic Logic Programs},
  year = {2025}
}

2024

DINE: Dimensional Interpretability of Node Embeddings

Simone Piaggesi, Megha Khosla, André Panisson, and Avishek Anand

IEEE Transactions on Knowledge and Data Engineering, Dec 2024

RESEARCH LINE
1

2

Abs Bib HTML

Graphs are ubiquitous due to their flexibility in representing social and technological systems as networks of interacting elements. Graph representation learning methods, such as node embeddings, are powerful approaches to map nodes into a latent vector space, allowing their use for various graph tasks. Despite their success, only few studies have focused on explaining node embeddings locally. Moreover, global explanations of node embeddings remain unexplored, limiting interpretability and debugging potentials. We address this gap by developing human-understandable explanations for dimensions in node embeddings. Towards that, we first develop new metrics that measure the global interpretability of embedding vectors based on the marginal contribution of the embedding dimensions to predicting graph structure. We say that an embedding dimension is more interpretable if it can faithfully map to an understandable sub-structure in the input graph – like community structure. Having observed that standard node embeddings have low interpretability, we then introduce DINE (Dimension-based Interpretable Node Embedding), a novel approach that can retrofit existing node embeddings by making them more interpretable without sacrificing their task performance. We conduct extensive experiments on synthetic and real-world graphs and show that we can simultaneously learn highly interpretable node embeddings with effective performance in link prediction.
@article{PKP2024, author = {Piaggesi, Simone and Khosla, Megha and Panisson, André and Anand, Avishek}, doi = {10.1109/tkde.2024.3425460}, issn = {2326-3865}, journal = {IEEE Transactions on Knowledge and Data Engineering}, line = {1,2}, month = dec, number = {12}, open_access = {Gold}, pages = {7986–7997}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {DINE: Dimensional Interpretability of Node Embeddings}, visible_on_website = {YES}, volume = {36}, year = {2024} }
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers

Guillermo Fernandez, Riccardo Guidotti, Fosca Giannotti, Mattia Setzu, Juan A. Aledo, and 2 more authors

Dec 2024

RESEARCH LINE
1

2

Abs Bib HTML

The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inbook{FGG2024, address = {Cham, Switzerland}, author = {Fernandez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and Gámez, Jose A. and Puerta, Jose M.}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_16}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {197–209}, publisher = {Springer Nature Switzerland}, title = {FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers}, visible_on_website = {YES}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Causality-Aware Local Interpretable Model-Agnostic Explanations

Martina Cinquini, and Riccardo Guidotti

Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A main drawback of eXplainable Artificial Intelligence (XAI) approaches is the feature independence assumption, hindering the study of potential variable dependencies. This leads to approximating black box behaviors by analysing the effects on randomly generated feature values that may rarely occur in the original samples. This paper addresses this issue by integrating causal knowledge in an XAI method to enhance transparency and enable users to assess the quality of the generated explanations. Specifically, we propose a novel extension to a widely used local and model-agnostic explainer, which encodes explicit causal relationships within the data surrounding the instance being explained. Extensive experiments show that our approach overcomes the original method in terms of faithfully replicating the black-box model’s mechanism and the consistency and reliability of the generated explanations. (arXiv)
@inbook{CG2024, author = {Cinquini, Martina and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-63800-8_6}, isbn = {9783031638008}, issn = {1865-0937}, line = {1,2}, open_access = {NO}, pages = {108–124}, publisher = {Springer Nature Switzerland}, title = {Causality-Aware Local Interpretable Model-Agnostic Explanations}, visible_on_website = {YES}, year = {2024} }
GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning

Rami Haffar, Francesca Naretto, David Sánchez, Anna Monreale, and Josep Domingo-Ferrer

In 2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) , Jun 2024

RESEARCH LINE
1

2

Abs Bib HTML

The increasing spread of artificial intelligence applications has led to decentralized frameworks that foster collaborative model training among multiple entities. One of such frameworks is federated learning, which ensures data availability in client nodes without requiring the central server to retain any data. Nevertheless, similar to centralized neural networks, interpretability remains a challenge in understanding the predictions of these decentralized frameworks. The limited access to data on the server side further complicates the applicability of explainers in such frameworks. To address this challenge, we propose GLOR-FLEX, a framework designed to generate rule-based global explanations from local explainers. GLOR-FLEX ensures client privacy by preventing the sharing of actual data between the clients and the server. The proposed framework initiates the process by constructing local decision trees on each client’s side to produce local explanations. Subsequently, by using rule extraction from these trees and strategically sorting and merging those rules, the server obtains a merged set of rules suitable to be used as a global explainer. We empirically evaluate the performance of GLOR-FLEX on three distinct tabular data sets, showing high fidelity scores between the explainers and both the local and global models. Our results support the effectiveness of GLOR-FLEX in generating accurate explanations that efficiently detect and explain the behavior of both local and global models.
@inproceedings{HNS2024, author = {Haffar, Rami and Naretto, Francesca and Sánchez, David and Monreale, Anna and Domingo-Ferrer, Josep}, booktitle = {2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)}, doi = {10.1109/fuzz-ieee60900.2024.10611878}, line = {1,2}, month = jun, open_access = {NO}, pages = {1–9}, publisher = {IEEE}, title = {GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning}, visible_on_website = {YES}, year = {2024} }

2023

Topics in Selective Classification

Andrea Pugnana

Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2023

RESEARCH LINE
1

2

Abs Bib HTML

Selective classification (also known as classification with a reject option, or learning to defer) extends a classifier with a selection function (reject option/strategy) to determine whether or not a prediction should be accepted. This mechanism allows the AI system to abstain in those instances where the classifier is more uncertain about the class to predict, introducing a trade-off between performance and coverage (the percentage of cases where the classifier does not abstain). The reject option has been extensively studied from a theoretical standpoint. However, state-of-the-art practical approaches and tools are model-specific, e.g., they are tailored to Deep Neural Networks and focused/experimented mainly on image datasets. In this work I developed a model-agnostic heuristics that is able to lift any (probabilistic) classifier into a selective classifier. The approach exploits both a cross-fitting strategy and results from quantile estimation to build the selective function. The algorithm is tested on several real-world datasets, showing improvements compared to existing methodologies.
@article{P2024, author = {Pugnana, Andrea}, doi = {10.1609/aaai.v37i13.26925}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,2}, month = jun, number = {13}, open_access = {Gold}, pages = {16129–16130}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Topics in Selective Classification}, visible_on_website = {YES}, volume = {37}, year = {2023} }
Declarative Reasoning on Explanations Using Constraint Logic Programming

Laura State, Salvatore Ruggieri, and Franco Turini

Jun 2023

RESEARCH LINE
2

Abs Bib HTML

Explaining opaque Machine Learning models is an increasingly relevant problem. Current explanation in AI methods suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of abstraction and interactivity with the user. We propose REASONX, an explanation method based on Constraint Logic Programming. REASONX can provide declarative, interactive explanations for decision trees, which can be the machine learning models under analysis or global or local surrogate models of any black-box model. Users can express background or common sense knowledge using linear constraints and MILP optimization over features of factual and contrastive instances, and interact with the answer constraints at different levels of abstraction through constraint projection. We present here the architecture of REASONX, which consists of a Python layer, closer to the user, and a CLP layer.
@inbook{SRT2023b, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, booktitle = {Logics in Artificial Intelligence}, doi = {10.1007/978-3-031-43619-2_10}, isbn = {9783031436192}, issn = {1611-3349}, line = {2}, open_access = {NO}, pages = {132–141}, publisher = {Springer Nature Switzerland}, title = {Declarative Reasoning on Explanations Using Constraint Logic Programming}, visible_on_website = {YES}, year = {2023} }
The Importance of Time in Causal Algorithmic Recourse

Isacco Beretta, and Martina Cinquini

Jun 2023

RESEARCH LINE
2

Abs Bib HTML

The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inbook{BC2023, author = {Beretta, Isacco and Cinquini, Martina}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_16}, isbn = {9783031440649}, issn = {1865-0937}, line = {2}, open_access = {NO}, pages = {283–298}, publisher = {Springer Nature Switzerland}, title = {The Importance of Time in Causal Algorithmic Recourse}, visible_on_website = {YES}, year = {2023} }
Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations

Eleonora Cappuccio, Daniele Fadda, Rosa Lanzilotti, and Salvatore Rinzivillo

In Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter , Sep 2023

RESEARCH LINE
1

2

3

Abs Bib HTML

The Human-Computer Interaction (HCI) community has long stressed the need for a more user-centered approach to Explainable Artificial Intelligence (XAI), a research area that aims at defining algorithms and tools to illustrate the predictions of the so-called black-box models. This approach can benefit from the fields of user-interface, user experience, and visual analytics. In this demo, we propose a visual-based tool, "F.I.P.E.R.", that shows interactive explanations combining rules and feature importance.
@inproceedings{CFR2023, author = {Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore}, booktitle = {Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter}, collection = {CHItaly 2023}, doi = {10.1145/3605390.3610811}, line = {1,2,3}, month = sep, open_access = {NO}, pages = {1–4}, publisher = {ACM}, series = {CHItaly 2023}, title = {Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations}, visible_on_website = {YES}, year = {2023} }
AUC-based Selective Classification

Pugnana Andrea, and Ruggieri Salvatore

Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
@misc{PR2023, author = {Andrea, Pugnana and Salvatore, Ruggieri}, line = {2}, month = dec, pages = {2494--2514}, title = {AUC-based Selective Classification}, year = {2023} }
Position Paper: On the Role of Abductive Reasoning in Semantic Image Segmentation.

Andrea Rafanelli; Stefania Costantini; Andrea Omicini

Dec 2023

RESEARCH LINE
2

Abs Bib

This position paper provides insights aiming at resolving the most pressing needs and issues of computer vision algorithms. Specifically, these problemsrelatetothescarcityofdata, theinabilityofsuchalgorithms to adapt to never-seen-before conditions, and the challenge of developing explainable and trustworthy algorithms. This work proposes the incorporation of reasoning systems, and in particular of abductive reasoning, into image segmentation algorithms as a potential solution to the aforementioned issues.
@misc{RCO2023, author = {Omicini, Andrea Rafanelli; Stefania Costantini; Andrea}, line = {2}, month = dec, title = {Position Paper: On the Role of Abductive Reasoning in Semantic Image Segmentation.}, year = {2023} }

2022

Methods and tools for causal discovery and causal inference

Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama

WIREs Data Mining and Knowledge Discovery, Jan 2022

RESEARCH LINE
2

Abs Bib HTML

Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{NPR2022, address = {Danvers, MS, USA}, author = {Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João}, doi = {10.1002/widm.1449}, issn = {1942-4795}, journal = {WIREs Data Mining and Knowledge Discovery}, line = {2}, month = jan, number = {2}, open_access = {Gold}, publisher = {Wiley}, title = {Methods and tools for causal discovery and causal inference}, visible_on_website = {YES}, volume = {12}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }

2021

TRIPLEx: Triple Extraction for Explanation

Mattia Setzu, Anna Monreale, and Pasquale Minervini

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
1

2

Abs Bib HTML

Transformer-based models are used to solve a variety of Natural Language Processing tasks. Still, these models are opaque and poorly understandable for their users. Current approaches to explainability focus on token importance, in which the explanation consists of a set of tokens relevant to the prediction, and natural language explanations, in which the explanation is a generated piece of text. The latter are usually learned by design with models trained end-to-end to provide a prediction and an explanation, or rely on powerful external text generators to do the heavy lifting for them. In this paper we present TriplEX, an explainability algorithm for Transformer-based models fine-tuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks. TriplEX explains Transformers-based models by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation.
@inproceedings{SMM2022, author = {Setzu, Mattia and Monreale, Anna and Minervini, Pasquale}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00015}, line = {1,2}, month = dec, open_access = {NO}, pages = {44–53}, publisher = {IEEE}, title = {TRIPLEx: Triple Extraction for Explanation}, visible_on_website = {YES}, year = {2021} }
Deriving a Single Interpretable Model by Merging Tree-Based Classifiers

Valerio Bonsignori, Riccardo Guidotti, and Anna Monreale

Dec 2021

RESEARCH LINE
1

2

Abs Bib HTML

Decision tree classifiers have been proved to be among the most interpretable models due to their intuitive structure that illustrates decision processes in form of logical rules. Unfortunately, more complex tree-based classifiers such as oblique trees and random forests overcome the accuracy of decision trees at the cost of becoming non interpretable. In this paper, we propose a method that takes as input any tree-based classifier and returns a single decision tree able to approximate its behavior. Our proposal merges tree-based classifiers by an intensional and extensional approach and applies a post-hoc explanation strategy. Our experiments shows that the retrieved single decision tree is at least as accurate as the original tree-based model, faithful, and more interpretable.
@inbook{BGM2021, author = {Bonsignori, Valerio and Guidotti, Riccardo and Monreale, Anna}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-88942-5_27}, isbn = {9783030889425}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {347–357}, publisher = {Springer International Publishing}, title = {Deriving a Single Interpretable Model by Merging Tree-Based Classifiers}, visible_on_website = {YES}, year = {2021} }
Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery

Martina Cinquini, Fosca Giannotti, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
2

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, artificial intelligence explanation, etc. In all such contexts, it is important to generate plausible data samples. A common assumption of approaches widely used for data generation is the independence of the features. However, typically, the variables of a dataset de-pend on one another, and these dependencies are not considered in data generation leading to the creation of implausible records. The main problem is that dependencies among variables are typically unknown. In this paper, we design a synthetic dataset generator for tabular data that is able to discover nonlinear causalities among the variables and use them at generation time. State-of-the-art methods for nonlinear causal discovery are typically inefficient. We boost them by restricting the causal discovery among the features appearing in the frequent patterns efficiently retrieved by a pattern mining algorithm. To validate our proposal, we design a framework for generating synthetic datasets with known causalities. Wide experimentation on many synthetic datasets and real datasets with known causalities shows the effectiveness of the proposed method.
@inproceedings{CGG2021, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00016}, line = {2}, month = dec, open_access = {NO}, pages = {54–63}, publisher = {IEEE}, title = {Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery}, visible_on_website = {YES}, year = {2021} }

2019

The AI black box explanation problem

Guidotti Riccardo, Monreale Anna, and Pedreschi Dino

Dec 2019

RESEARCH LINE
1

2

3

Abs Bib HTML

The use of machine learning in decision-making has triggered an intense debate about “fair algorithms”. Given that fairness intuitions differ and can led to conflicting technical requirements, there is a pressing need to integrate ethical thinking into research and design of machine learning. We outline a framework showing how this can be done.
@misc{GMP2019, author = {Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi}, line = {1,2,3}, month = dec, publisher = {ERCIM – the European Research Consortium for Informatics and Mathematics}, title = {The AI black box explanation problem}, year = {2019} }