People

Bib

@misc{PMG2025,
  author = {Pugnana, Andrea and Massidda, Riccardo and Giannini, Francesco and Barbiero, Pietro and Zarlenga, Mateo Espinosa and Pellungrini, Roberto and Dominici, Gabriele and Giannotti, Fosca and Bacciu, Davide},
  line = {1,2},
  month = dec,
  title = {Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts},
  year = {2025}
}

A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems

G. Barlacchi, M. Lalli, E. Ferragina, F. Giannotti, and L. Pappalardo

Dec 2025

RESEARCH LINE
4

Abs Bib HTML

Recommender systems continuously interact with users, creating feedback loops that shape both individual behavior and collective market dynamics. This paper introduces a simulation framework to model these loops in online retail environments, where recommenders are periodically retrained on evolving user–item interactions. Using the Amazon e-Commerce dataset, we analyze how different recommendation algorithms influence diversity, purchase concentration, and user homogenization over time. Results reveal a systematic trade-off: while the feedback loop increases individual diversity, it simultaneously reduces collective diversity and concentrates demand on a few popular items. Moreover, for some recommender systems, the feedback loop increases user homogenization over time, making user purchase profiles increasingly similar. These findings underscore the need for recommender designs that balance personalization with long-term diversity.
@misc{BLF2025, author = {Barlacchi, G. and Lalli, M. and Ferragina, E. and Giannotti, F. and Pappalardo, L.}, doi = {10.48550/arXiv.2510.14857}, line = {4}, month = dec, title = {A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems}, year = {2025} }

MAINLE: a Multi-Agent, Interactive, Natural Language Local Explainer of Classification Tasks

Paulo Bruno Serafim, Romula Ferrer Filho, STENIO Freitas, Gizem Gezici, Fosca Giannotti, and 2 more authors

Dec 2025

RESEARCH LINE

3

Bib

@misc{SFF2025,
  author = {Serafim, Paulo Bruno and Filho, Romula Ferrer and Freitas, STENIO and Gezici, Gizem and Giannotti, Fosca and Raimondi, Franco and Santos, Alexandre},
  line = {1,3},
  month = dec,
  title = {MAINLE: a Multi-Agent, Interactive, Natural Language Local Explainer of Classification Tasks},
  year = {2025}
}

Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding

Simone Piaggesi, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Dec 2025

RESEARCH LINE
1

Bib HTML
@misc{PGG2025, author = {Piaggesi, Simone and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, line = {1}, month = dec, title = {Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding}, year = {2025} }
Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Dec 2025

RESEARCH LINE
1

5

Abs Bib HTML

During the last few years, the abundance of data has significantly boosted the performance of Machine Learning models, integrating them into several aspects of daily life. However, the rise of powerful Artificial Intelligence tools has introduced ethical and legal complexities. This paper proposes a computational framework to analyze the ethical and legal dimensions of Machine Learning models, focusing specifically on privacy concerns and interpretability. In fact, recently, the research community proposed privacy attacks able to reveal whether a record was part of the black-box training set or inferring variable values by accessing and querying a Machine Learning model. These attacks highlight privacy vulnerabilities and prove that GDPR regulation might be violated by making data or Machine Learning models accessible. At the same time, the complexity of these models, often labelled as “black-boxes”, has made the development of explanation methods indispensable to enhance trust and facilitate their acceptance and adoption in high-stake scenarios.
@misc{NMG2025, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, line = {1,5}, month = dec, publisher = {Trans. Data Priv. 18 (2), 67-93}, title = {Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.}, year = {2025} }
"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy

Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, and 1 more author

Dec 2025

RESEARCH LINE
1

Bib
@misc{GGG2025, author = {Gambetta, Daniele and Gezici, Gizem and Giannotti, Fosca and Pedreschi, Dino and Knott, Alistair and Pappalardo, Luca}, line = {1}, month = dec, title = {"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy}, year = {2025} }

2024

A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges

Mario Alfonso Prado-Romero, Bardh Prenkaj, Giovanni Stilo, and Fosca Giannotti

ACM Computing Surveys, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Graph Neural Networks (GNNs) perform well in community detection and molecule classification. Counterfactual Explanations (CE) provide counter-examples to overcome the transparency limitations of black-box models. Due to the growing attention in graph learning, we focus on the concepts of CE for GNNs. We analysed the SoA to provide a taxonomy, a uniform notation, and the benchmarking datasets and evaluation metrics. We discuss fourteen methods, their evaluation protocols, twenty-two datasets, and nineteen metrics. We integrated the majority of methods into the GRETEL library to conduct an empirical evaluation to understand their strengths and pitfalls. We highlight open challenges and future work.
@article{PPS2024, author = {Prado-Romero, Mario Alfonso and Prenkaj, Bardh and Stilo, Giovanni and Giannotti, Fosca}, doi = {10.1145/3618105}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1}, month = apr, number = {7}, open_access = {Gold}, pages = {1–37}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey on Graph Counterfactual Explanations: Definitions, Methods, Evaluation, and Research Challenges}, visible_on_website = {YES}, volume = {56}, year = {2024} }
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers

Guillermo Fernandez, Riccardo Guidotti, Fosca Giannotti, Mattia Setzu, Juan A. Aledo, and 2 more authors

Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inbook{FGG2024, address = {Cham, Switzerland}, author = {Fernandez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and Gámez, Jose A. and Puerta, Jose M.}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_16}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {197–209}, publisher = {Springer Nature Switzerland}, title = {FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers}, visible_on_website = {YES}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Exploring Large Language Models Capabilities to Explain Decision Trees

Paulo Bruno Serafim, Pierluigi Crescenzi, Gizem Gezici, Eleonora Cappuccio, Salvatore Rinzivillo, and 1 more author

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are widely adopted in Machine Learning tasks due to their operation simplicity and interpretability aspects. However, following the decision process path taken by trees can be difficult in a complex scenario or in a case where a user has no familiarity with them. Prior research showed that converting outcomes to natural language is an accessible way to facilitate understanding for non-expert users in several tasks. More recently, there has been a growing effort to use Large Language Models (LLMs) as a tool for providing natural language texts. In this paper, we examine the proficiency of LLMs to explain decision tree predictions in simple terms through the generation of natural language explanations. By exploring different textual representations and prompt engineering strategies, we identify capabilities that strengthen LLMs as a competent explainer as well as highlight potential challenges and limitations, opening further research possibilities on natural language explanations for decision trees.
@inbook{SGC2024, author = {Serafim, Paulo Bruno and Crescenzi, Pierluigi and Gezici, Gizem and Cappuccio, Eleonora and Rinzivillo, Salvatore and Giannotti, Fosca}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240183}, isbn = {9781643685229}, issn = {1879-8314}, line = {1}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {Exploring Large Language Models Capabilities to Explain Decision Trees}, visible_on_website = {YES}, year = {2024} }
Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space

Simone Piaggesi, Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

IEEE Access, Jun 2024

RESEARCH LINE
1

Abs Bib HTML

We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities similarly to well-known dimensionality reduction techniques. Our approach introduces a transparent latent space optimized for interpretability of both counterfactual and prototypical explanations for tabular data. The approach enables the easy extraction of local and global explanations and ensures that the latent space preserves similarity relations, enabling meaningful prototypical and counterfactual examples for any classifier.
@article{PBG2024, author = {Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1109/access.2024.3496114}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {168983–169000}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Multi-Perspective Stance Detection

Benedetta Muscato, Praveen Bushipaka, Gizem Gezici, Lucia Passaro, and Fosca Giannotti

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspectiveaware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.
@misc{MBG2024bb, address = {Aachen, Germany}, author = {Muscato, Benedetta and Bushipaka, Praveen and Gezici, Gizem and Passaro, Lucia and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Multi-Perspective Stance Detection}, year = {2024} }
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers

Eleonora Cappuccio, Benedetta Muscato, Laura Pollacci, Marta Marchiori Manerba, Clara Punzi, and 5 more authors

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
@misc{CMP2024, address = {Aachen, Germany}, author = {Cappuccio, Eleonora and Muscato, Benedetta and Pollacci, Laura and Manerba, Marta Marchiori and Punzi, Clara and Mala, Chandana Sree and Lalli, Margherita and Gezici, Gizem and Natilli, Michela and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers}, year = {2024} }
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }
An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives

Benedetta Muscato, Chandana Sree Mala, Marta Marchiori Manerba, Gizem Gezici, and Fosca Giannotti

Dec 2024

RESEARCH LINE
3

Abs Bib HTML

The varied backgrounds and experiences of human annotators inject different opinions and potential biases into the data, inevitably leading to disagreements. Yet, traditional aggregation methods fail to capture individual judgments since they rely on the notion of a single ground truth. Our aim is to review prior contributions to pinpoint the shortcomings that might cause stereotypical content generation. As a preliminary study, our purpose is to investigate state-of-the-art approaches, primarily focusing on the following two research directions. First, we investigate how adding subjectivity aspects to LLMs might guarantee diversity. We then look into the alignment between humans and LLMs and discuss how to measure it. Considering existing gaps, our review explores possible methods to mitigate the perpetuation of biases targeting specific communities. However, we recognize the potential risk of disseminating sensitive information due to the utilization of socio-demographic data in the training process. These considerations underscore the inclusion of diverse perspectives while taking into account the critical importance of implementing robust safeguards to protect individuals’ privacy and prevent the inadvertent propagation of sensitive information.
@misc{MMM2024, address = {Torino, Italia}, author = {Muscato, Benedetta and Mala, Chandana Sree and Manerba, Marta Marchiori and Gezici, Gizem and Giannotti, Fosca}, line = {3}, month = dec, pages = {49--55}, publisher = {ELRA and ICCL}, title = {An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives}, year = {2024} }
Interpretable and Fair Mechanisms for Abstaining Classifiers

Daphne Lenders, Andrea Pugnana, Roberto Pellungrini, Toon Calders, Dino Pedreschi, and 1 more author

Dec 2024

RESEARCH LINE
1

5

Abs Bib HTML

Abstaining classifiers have the option to refrain from providing a prediction for instances that are difficult to classify. The abstention mechanism is designed to trade off the classifier‚Äôs performance on the accepted data while ensuring a minimum number of predictions. In this setting, fairness concerns often arise when the abstention mechanism reduces errors only for majority groups, increasing disparities across demographic groups. We introduce Interpretable and Fair Abstaining Classifier (IFAC), an algorithm that can reject predictions based on uncertainty and unfairness. By rejecting potentially unfair predictions, our method reduces disparities across groups of the non-rejected data. The unfairness-based rejections rely on interpretable rule-based fairness checks and situation testing, enabling transparent review and decision-making.
@misc{LPP2024, author = {Lenders, Daphne and Pugnana, Andrea and Pellungrini, Roberto and Calders, Toon and Pedreschi, Dino and Giannotti, Fosca}, doi = {[75,46,72,75,50,73,78]. }, line = {1,5}, month = dec, title = {Interpretable and Fair Mechanisms for Abstaining Classifiers}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }
Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }
Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic

Clara Punzi, Aleksandra Maslennikova, Gizem Gezici, Roberto Pellungrini, and Fosca Giannotti

Jun 2023

RESEARCH LINE
1

4

Abs Bib HTML

Pandemic vaccination campaigns must account for vaccine skepticism as an obstacle to overcome. Using machine learning to identify behavioral and psychological patterns in public survey datasets can provide valuable insights and inform vaccination campaigns based on empirical evidence. However, we argue that the adoption of local and global explanation methodologies can provide additional support to health practitioners by suggesting personalized communication strategies and revealing potential demographic, social, or structural barriers to vaccination requiring systemic changes. In this paper, we first implement a chain classification model for the adoption of the vaccine during the H1N1 influenza outbreak taking seasonal vaccination information into account, and then compare it with a binary classifier for vaccination to better understand the overall patterns in the data. Following that, we derive and compare global explanations using post-hoc methodologies and interpretable-by-design models. Our findings indicate that socio-demographic factors play a distinct role in the H1N1 vaccination as compared to the general vaccination. Nevertheless, medical recommendation and health insurance remain significant factors for both vaccinations. Then, we concentrated on the subpopulation of individuals who did not receive an H1N1 vaccination despite being at risk of developing severe symptoms. In an effort to assist practitioners in providing effective recommendations to patients, we present rules and counterfactuals for the selected instances based on local explanations. Finally, we raise concerns regarding gender and racial disparities in healthcare access by analysing the interaction effects of sensitive attributes on the model’s output.
@inbook{PMG2023, author = {Punzi, Clara and Maslennikova, Aleksandra and Gezici, Gizem and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_31}, isbn = {9783031440670}, issn = {1865-0937}, line = {1,4}, open_access = {Gold}, pages = {621–635}, publisher = {Springer Nature Switzerland}, title = {Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic}, visible_on_website = {YES}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }
Handling Missing Values in Local Post-hoc Explainability

Martina Cinquini, Fosca Giannotti, Riccardo Guidotti, and Andrea Mattei

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

Missing data are quite common in real scenarios when using Artificial Intelligence (AI) systems for decision-making with tabular data and effectively handling them poses a significant challenge for such systems. While some machine learning models used by AI systems can tackle this problem, the existing literature lacks post-hoc explainability approaches able to deal with predictors that encounter missing data. In this paper, we extend a widely used local model-agnostic post-hoc explanation approach that enables explainability in the presence of missing values by incorporating state-of-the-art imputation methods within the explanation process. Since our proposal returns explanations in the form of feature importance, the user will be aware also of the importance of a missing value in a given record for a particular prediction. Extensive experiments show the effectiveness of the proposed method with respect to some baseline solutions relying on traditional data imputation.
@inbook{CGG2023, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo and Mattei, Andrea}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_14}, isbn = {9783031440670}, issn = {1865-0937}, line = {1}, open_access = {Gold}, pages = {256–278}, publisher = {Springer Nature Switzerland}, title = {Handling Missing Values in Local Post-hoc Explainability}, visible_on_website = {YES}, year = {2023} }

2022

Privacy Risk of Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Sep 2022

RESEARCH LINE
5

Abs Bib HTML

In this paper we propose to study a methodology that enables the evaluation of the privacy risk exposure of global explainers based on an interpretable classifier that imitates the global reasoning of a black-box classifier. The idea is to verify if the layer of interpretability added by the interpretable model can jeopardize the privacy protection of the training data used for learning the black-box classifier. In order to address this problem, we exploit a well-known attack model called membership inference attack (MIA). We then compute the privacy risk change ΔR due to the introduction of the global explainer c. The preliminary experimental results suggest that global explainers based on decision trees introduce a higher risk of privacy, increasing the percentage of records identified as members of the training dataset used to train the original black-box classifiers. These results suggest that in order to provide Trustworthy AI, it becomes fundamental to consider the relationship between different ethical values to identify possible values like transparency and privacy that may be in contrast, and studying solutions that enable the simultaneous satisfaction of more than one value.
@inbook{NMG2022, address = {Amsterdam, the Netherlands}, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220206}, issn = {1879-8314}, line = {5}, month = sep, open_access = {Gold}, pages = {249 - 251}, publisher = {IOS Press}, title = {Privacy Risk of Global Explainers}, visible_on_website = {YES}, year = {2022} }
Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{PBP2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {4}, month = apr, open_access = {Gold}, pages = {1–9}, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, visible_on_website = {YES}, year = {2022} }
Interpretable Latent Space to Enable Counterfactual Explanations

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Apr 2022

RESEARCH LINE
1

Abs Bib HTML

Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{BGG2023c, address = {Montpellier, France}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_37}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {525–540}, publisher = {Springer Nature Switzerland}, title = {Interpretable Latent Space to Enable Counterfactual Explanations}, visible_on_website = {YES}, year = {2022} }
Evaluating the Privacy Exposure of Interpretable Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

In 2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2022

RESEARCH LINE
5

Abs Bib HTML

In recent years we are witnessing the diffusion of AI systems based on powerful Machine Learning models which find application in many critical contexts such as medicine, financial market and credit scoring. In such a context it is particularly important to design Trustworthy AI systems while guaranteeing transparency, with respect to their decision reasoning and privacy protection. Although many works in the literature addressed the lack of transparency and the risk of privacy exposure of Machine Learning models, the privacy risks of explainers have not been appropriately studied. This paper presents a methodology for evaluating the privacy exposure raised by interpretable global explainers able to imitate the original black-box classifier. Our methodology exploits the well-known Membership Inference Attack. The experimental results highlight that global explainers based on interpretable trees lead to an increase in privacy exposure.
@inproceedings{NMG2022b, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi56440.2022.00012}, line = {5}, month = dec, open_access = {NO}, pages = {13–19}, publisher = {IEEE}, title = {Evaluating the Privacy Exposure of Interpretable Global Explainers}, visible_on_website = {YES}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }
Transparent Latent Space Counterfactual Explanations for Tabular Data

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{BGG2023b, address = {Shenzhen, China}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)}, doi = {10.1109/dsaa54385.2022.10032407}, line = {1}, month = oct, open_access = {NO}, pages = {1–10}, publisher = {IEEE}, title = {Transparent Latent Space Counterfactual Explanations for Tabular Data}, visible_on_website = {YES}, year = {2022} }
Understanding peace through the world news

Vasiliki Voukelatou, Ioanna Miliou, Fosca Giannotti, and Luca Pappalardo

EPJ Data Science, Jan 2022

RESEARCH LINE
4

Abs Bib HTML

Peace is a principal dimension of well-being and is the way out of inequity and violence. Thus, its measurement has drawn the attention of researchers, policymakers, and peacekeepers. During the last years, novel digital data streams have drastically changed the research in this field. The current study exploits information extracted from a new digital database called Global Data on Events, Location, and Tone (GDELT) to capture peace through the Global Peace Index (GPI). Applying predictive machine learning models, we demonstrate that news media attention from GDELT can be used as a proxy for measuring GPI at a monthly level. Additionally, we use explainable AI techniques to obtain the most important variables that drive the predictions. This analysis highlights each country’s profile and provides explanations for the predictions, and particularly for the errors and the events that drive these errors. We believe that digital data exploited by researchers, policymakers, and peacekeepers, with data science tools as powerful as machine learning, could contribute to maximizing the societal benefits and minimizing the risks to peace.
@article{VMG2022, author = {Voukelatou, Vasiliki and Miliou, Ioanna and Giannotti, Fosca and Pappalardo, Luca}, doi = {10.1140/epjds/s13688-022-00315-z}, issn = {2193-1127}, journal = {EPJ Data Science}, line = {4}, month = jan, number = {1}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Understanding peace through the world news}, visible_on_website = {YES}, volume = {11}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2021

Trustworthy AI

Raja Chatila, Virginia Dignum, Michael Fisher, Fosca Giannotti, Katharina Morik, and 2 more authors

Dec 2021

RESEARCH LINE
5

Abs Bib HTML

Modern AI systems have become of widespread use in almost all sectors with a strong impact on our society. However, the very methods on which they rely, based on Machine Learning techniques for processing data to predict outcomes and to make decisions, are opaque, prone to bias and may produce wrong answers. Objective functions optimized in learning systems are not guaranteed to align with the values that motivated their definition. Properties such as transparency, verifiability, explainability, security, technical robustness and safety, are key to build operational governance frameworks, so that to make AI systems justifiably trustworthy and to align their development and use with human rights and values.
@inbook{CDF2021, author = {Chatila, Raja and Dignum, Virginia and Fisher, Michael and Giannotti, Fosca and Morik, Katharina and Russell, Stuart and Yeung, Karen}, booktitle = {Reflections on Artificial Intelligence for Humanity}, doi = {10.1007/978-3-030-69128-8_2}, isbn = {9783030691288}, issn = {1611-3349}, line = {5}, open_access = {NO}, pages = {13–39}, publisher = {Springer International Publishing}, title = {Trustworthy AI}, visible_on_website = {YES}, year = {2021} }
GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }
Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery

Martina Cinquini, Fosca Giannotti, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
2

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, artificial intelligence explanation, etc. In all such contexts, it is important to generate plausible data samples. A common assumption of approaches widely used for data generation is the independence of the features. However, typically, the variables of a dataset de-pend on one another, and these dependencies are not considered in data generation leading to the creation of implausible records. The main problem is that dependencies among variables are typically unknown. In this paper, we design a synthetic dataset generator for tabular data that is able to discover nonlinear causalities among the variables and use them at generation time. State-of-the-art methods for nonlinear causal discovery are typically inefficient. We boost them by restricting the causal discovery among the features appearing in the frequent patterns efficiently retrieved by a pattern mining algorithm. To validate our proposal, we design a framework for generating synthetic datasets with known causalities. Wide experimentation on many synthetic datasets and real datasets with known causalities shows the effectiveness of the proposed method.
@inproceedings{CGG2021, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00016}, line = {2}, month = dec, open_access = {NO}, pages = {54–63}, publisher = {IEEE}, title = {Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery}, visible_on_website = {YES}, year = {2021} }

2020

Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks

Francesca Naretto, Roberto Pellungrini, Franco Maria Nardini, and Fosca Giannotti

Dec 2020

RESEARCH LINE
4

5

Abs Bib HTML

The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.
@inbook{NPN2020, author = {Naretto, Francesca and Pellungrini, Roberto and Nardini, Franco Maria and Giannotti, Fosca}, booktitle = {ECML PKDD 2020 Workshops}, doi = {10.1007/978-3-030-65965-3_34}, isbn = {9783030659653}, issn = {1865-0937}, line = {4,5}, open_access = {NO}, pages = {501–516}, publisher = {Springer International Publishing}, title = {Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks}, visible_on_website = {YES}, year = {2020} }
Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{GMS2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, open_access = {NO}, pages = {167–176}, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, visible_on_website = {YES}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2025

Human-AI coevolution

Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-László Barabási, and 12 more authors

Artificial Intelligence, Feb 2025

RESEARCH LINE
1

Abs Bib HTML

Human-AI coevolution, defined as a process in which humans and AI algorithms continuously influence each other, increasingly characterises our society, but is understudied in artificial intelligence and complexity science literature. Recommender systems and assistants play a prominent role in human-AI coevolution, as they permeate many facets of daily life and influence human choices through online platforms. The interaction between users and AI results in a potentially endless feedback loop, wherein users’ choices generate data to train AI models, which, in turn, shape subsequent user preferences. This human-AI feedback loop has peculiar characteristics compared to traditional human-machine interaction and gives rise to complex and often “unintended” systemic outcomes. This paper introduces human-AI coevolution as the cornerstone for a new field of study at the intersection between AI and complexity science focused on the theoretical, empirical, and mathematical investigation of the human-AI feedback loop. In doing so, we: (i) outline the pros and cons of existing methodologies and highlight shortcomings and potential ways for capturing feedback loop mechanisms; (ii) propose a reflection at the intersection between complexity science, AI and society; (iii) provide real-world examples for different human-AI ecosystems; and (iv) illustrate challenges to the creation of such a field of study, conceptualising them at increasing levels of abstraction, i.e., scientific, legal and socio-political.
@article{PPF2025, address = {Netherlands}, author = {Pedreschi, Dino and Pappalardo, Luca and Ferragina, Emanuele and Baeza-Yates, Ricardo and Barabási, Albert-László and Dignum, Frank and Dignum, Virginia and Eliassi-Rad, Tina and Giannotti, Fosca and Kertész, János and Knott, Alistair and Ioannidis, Yannis and Lukowicz, Paul and Passarella, Andrea and Pentland, Alex Sandy and Shawe-Taylor, John and Vespignani, Alessandro}, doi = {10.1016/j.artint.2024.104244}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1}, month = feb, open_access = {Gold}, pages = {104244}, publisher = {Elsevier BV}, title = {Human-AI coevolution}, visible_on_website = {YES}, volume = {339}, year = {2025} }
Group Explainability Through Local Approximation

Mattia Setzu, Riccardo Guidotti, Dino Pedreschi, and Fosca Giannotti

Oct 2025

RESEARCH LINE
2

Abs Bib HTML

Machine learning models are becoming increasingly complex and widely adopted. Interpretable machine learning allows us to not only make predictions but also understand the rationale behind automated decisions through explanations. Explanations are typically characterized by their scope: local explanations are generated by local surrogate models for specific instances, while global explanations aim to approximate the behavior of the entire black-box model. In this paper, we break this dichotomy of locality to explore an underexamined area that lies between these two extremes: meso-level explanations. The goal of meso-level explainability is to provide explanations using a set of meso-level interpretable models, which capture patterns at an intermediate level of abstraction. To this end, we propose GrouX, an explainable-by-design algorithm that generates meso-level explanations in the form of feature importance scores. Our approach includes a partitioning phase that identifies meso groups, followed by the training of interpretable models within each group. We evaluate GrouX on a collection of tabular datasets, reporting both the accuracy and complexity of the resulting meso models, and compare it against other meso-level explainability algorithms. Additionally, we analyze the algorithm’s sensitivity to its hyperparameters to better understand its behavior and robustness.
@inbook{SGP2025, address = {ECAI 2025}, author = {Setzu, Mattia and Guidotti, Riccardo and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {ECAI 2025}, doi = {10.3233/faia250902}, isbn = {9781643686318}, issn = {1879-8314}, line = {2}, month = oct, open_access = {Gold}, pages = {952 - 958}, publisher = {IOS Press}, title = {Group Explainability Through Local Approximation}, visible_on_website = {YES}, year = {2025} }
Human-AI coevolution

Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-László Barabási, and 12 more authors

Artificial Intelligence, Feb 2025

RESEARCH LINE
2

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of black box models in classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. Moreover, diversity of counter-exemplars is modeled as an optimization problem, solved by a greedy algorithm with theoretical guarantee. We report experiments on three datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, diversity, and usefulness, and that is comparable to it in terms of stability.
@article{Pedreschi_2025, address = {Netherlands}, author = {Pedreschi, Dino and Pappalardo, Luca and Ferragina, Emanuele and Baeza-Yates, Ricardo and Barabási, Albert-László and Dignum, Frank and Dignum, Virginia and Eliassi-Rad, Tina and Giannotti, Fosca and Kertész, János and Knott, Alistair and Ioannidis, Yannis and Lukowicz, Paul and Passarella, Andrea and Pentland, Alex Sandy and Shawe-Taylor, John and Vespignani, Alessandro}, doi = {10.1016/j.artint.2024.104244}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {2}, month = feb, open_access = {Gold}, pages = {104244}, publisher = {Elsevier BV}, title = {Human-AI coevolution}, visible_on_website = {YES}, volume = {339}, year = {2025} }
Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding

Simone Piaggesi, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Dec 2025

RESEARCH LINE
1

Bib HTML
@misc{PGG2025, author = {Piaggesi, Simone and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, line = {1}, month = dec, title = {Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding}, year = {2025} }
"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy

Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, and 1 more author

Dec 2025

RESEARCH LINE
1

Bib
@misc{GGG2025, author = {Gambetta, Daniele and Gezici, Gizem and Giannotti, Fosca and Pedreschi, Dino and Knott, Alistair and Pappalardo, Luca}, line = {1}, month = dec, title = {"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy}, year = {2025} }

2024

Explaining Siamese networks in few-shot learning

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Machine Learning, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Siamese neural networks are widely used in few-shot learning tasks thanks to their ability to compare pairs of samples and generalize from very limited labeled data. However, their internal decision-making process remains opaque, since similarity-based representations do not provide intuitive explanations for end users. In this work, we investigate how to explain Siamese networks by attributing contribution scores to both input samples involved in the comparison. We introduce an explanation method specifically tailored to pairwise architectures, producing two synchronized saliency maps that highlight which regions of the support and query examples drive the similarity judgment. We evaluate the approach on image-based few-shot classification benchmarks, showing that the explanations highlight semantically meaningful structures and remain consistent across different evaluation episodes.
@article{FGP2024, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, doi = {10.1007/s10994-024-06529-8}, issn = {1573-0565}, journal = {Machine Learning}, line = {1}, month = apr, number = {10}, open_access = {Gold}, pages = {7723–7760}, publisher = {Springer Science and Business Media LLC}, title = {Explaining Siamese networks in few-shot learning}, visible_on_website = {YES}, volume = {113}, year = {2024} }
Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space

Simone Piaggesi, Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

IEEE Access, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities similarly to well-known dimensionality reduction techniques. Our approach introduces a transparent latent space optimized for interpretability of both counterfactual and prototypical explanations for tabular data. The approach enables the easy extraction of local and global explanations and ensures that the latent space preserves similarity relations, enabling meaningful prototypical and counterfactual examples for any classifier.
@article{PBG2024, author = {Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1109/access.2024.3496114}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {168983–169000}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space}, visible_on_website = {YES}, volume = {12}, year = {2024} }
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }
Interpretable and Fair Mechanisms for Abstaining Classifiers

Daphne Lenders, Andrea Pugnana, Roberto Pellungrini, Toon Calders, Dino Pedreschi, and 1 more author

Dec 2024

RESEARCH LINE
1

5

Abs Bib HTML

Abstaining classifiers have the option to refrain from providing a prediction for instances that are difficult to classify. The abstention mechanism is designed to trade off the classifier‚Äôs performance on the accepted data while ensuring a minimum number of predictions. In this setting, fairness concerns often arise when the abstention mechanism reduces errors only for majority groups, increasing disparities across demographic groups. We introduce Interpretable and Fair Abstaining Classifier (IFAC), an algorithm that can reject predictions based on uncertainty and unfairness. By rejecting potentially unfair predictions, our method reduces disparities across groups of the non-rejected data. The unfairness-based rejections rely on interpretable rule-based fairness checks and situation testing, enabling transparent review and decision-making.
@misc{LPP2024, author = {Lenders, Daphne and Pugnana, Andrea and Pellungrini, Roberto and Calders, Toon and Pedreschi, Dino and Giannotti, Fosca}, doi = {[75,46,72,75,50,73,78]. }, line = {1,5}, month = dec, title = {Interpretable and Fair Mechanisms for Abstaining Classifiers}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }
Effects of Route Randomization on Urban Emissions

Giuliano Cornacchia, Mirco Nanni, Dino Pedreschi, and Luca Pappalardo

SUMO Conference Proceedings, Jun 2023

RESEARCH LINE
4

5

Abs Bib HTML

Routing algorithms typically suggest the fastest path or slight variation to reach a user’s desired destination. Although this suggestion at the individual level is undoubtedly advantageous for the user, from a collective point of view, the aggregation of all single suggested paths may result in an increasing impact (e.g., in terms of emissions). In this study, we use SUMO to simulate the effects of incorporating randomness into routing algorithms on emissions, their distribution, and travel time in the urban area of Milan (Italy). Our results reveal that, given the common practice of routing towards the fastest path, a certain level of randomness in routes reduces emissions and travel time. In other words, the stronger the random component in the routes, the more pronounced the benefits upon a certain threshold. Our research provides insight into the potential advantages of considering collective outcomes in routing decisions and highlights the need to explore further the relationship between route randomization and sustainability in urban transportation.
@article{CNP2023, author = {Cornacchia, Giuliano and Nanni, Mirco and Pedreschi, Dino and Pappalardo, Luca}, doi = {10.52825/scp.v4i.217}, issn = {2750-4425}, journal = {SUMO Conference Proceedings}, line = {4,5}, month = jun, open_access = {Gold}, pages = {75–87}, publisher = {TIB Open Publishing}, title = {Effects of Route Randomization on Urban Emissions}, visible_on_website = {YES}, volume = {4}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{PBP2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {4}, month = apr, open_access = {Gold}, pages = {1–9}, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, visible_on_website = {YES}, year = {2022} }
Interpretable Latent Space to Enable Counterfactual Explanations

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Apr 2022

RESEARCH LINE
1

Abs Bib HTML

Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{BGG2023c, address = {Montpellier, France}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_37}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {525–540}, publisher = {Springer Nature Switzerland}, title = {Interpretable Latent Space to Enable Counterfactual Explanations}, visible_on_website = {YES}, year = {2022} }
Explaining Siamese Networks in Few-Shot Learning for Audio Data

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Apr 2022

RESEARCH LINE
4

Abs Bib HTML

Machine learning models are not able to generalize correctly when queried on samples belonging to class distributions that were never seen during training. This is a critical issue, since real world applications might need to quickly adapt without the necessity of re-training. To overcome these limitations, few-shot learning frameworks have been proposed and their applicability has been studied widely for computer vision tasks. Siamese Networks learn pairs similarity in form of a metric that can be easily extended on new unseen classes. Unfortunately, the downside of such systems is the lack of explainability. We propose a method to explain the outcomes of Siamese Networks in the context of few-shot learning for audio data. This objective is pursued through a local perturbation-based approach that evaluates segments-weighted-average contributions to the final outcome considering the interplay between different areas of the audio spectrogram. Qualitative and quantitative results demonstrate that our method is able to show common intra-class characteristics and erroneous reliance on silent sections.
@inbook{FGP2022, address = {Cham, Switzerland}, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_36}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {509–524}, publisher = {Springer Nature Switzerland}, title = {Explaining Siamese Networks in Few-Shot Learning for Audio Data}, visible_on_website = {YES}, year = {2022} }
Methods and tools for causal discovery and causal inference

Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama

WIREs Data Mining and Knowledge Discovery, Jan 2022

RESEARCH LINE
2

Abs Bib HTML

Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{NPR2022, address = {Danvers, MS, USA}, author = {Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João}, doi = {10.1002/widm.1449}, issn = {1942-4795}, journal = {WIREs Data Mining and Knowledge Discovery}, line = {2}, month = jan, number = {2}, open_access = {Gold}, publisher = {Wiley}, title = {Methods and tools for causal discovery and causal inference}, visible_on_website = {YES}, volume = {12}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }
Transparent Latent Space Counterfactual Explanations for Tabular Data

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{BGG2023b, address = {Shenzhen, China}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)}, doi = {10.1109/dsaa54385.2022.10032407}, line = {1}, month = oct, open_access = {NO}, pages = {1–10}, publisher = {IEEE}, title = {Transparent Latent Space Counterfactual Explanations for Tabular Data}, visible_on_website = {YES}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2021

GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }
FairLens: Auditing black-box clinical decision support systems

Cecilia Panigutti, Alan Perotti, André Panisson, Paolo Bajardi, and Dino Pedreschi

Information Processing & Management, Sep 2021

RESEARCH LINE
1

4

Abs Bib HTML

Highlights: We present a pipeline to detect and explain potential fairness issues in Clinical DSS. We study and compare different multi-label classification disparity measures. We explore ICD9 bias in MIMIC-IV, an openly available ICU benchmark dataset
@article{PPB2021, author = {Panigutti, Cecilia and Perotti, Alan and Panisson, André and Bajardi, Paolo and Pedreschi, Dino}, doi = {10.1016/j.ipm.2021.102657}, issn = {0306-4573}, journal = {Information Processing & Management}, line = {1,4}, month = sep, number = {5}, open_access = {Gold}, pages = {102657}, publisher = {Elsevier BV}, title = {FairLens: Auditing black-box clinical decision support systems}, visible_on_website = {YES}, volume = {58}, year = {2021} }

2020

Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Sep 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by “morphing” into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
@inbook{GMM2019, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-46150-8_12}, isbn = {9783030461508}, issn = {1611-3349}, line = {1,4}, pages = {189–205}, publisher = {Springer International Publishing}, title = {Black Box Explanation by Learning Image Exemplars in the Latent Feature Space}, visible_on_website = {YES}, year = {2020} }
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box image classifiers through synthetic exemplar and counter-exemplar learnt in the latent feature space. Our explanation method exploits the latent representations learned through an adversarial autoencoder for generating a synthetic neighborhood of the image for which an explanation is required. A decision tree is trained on a set of images represented in the latent space, and its decision rules are used to generate exemplar images showing how the original image can be modified to stay within its class. Counterfactual rules are used to generate counter-exemplars showing how the original image can “morph” into another class. The explanation also comprehends a saliency map highlighting the areas that contribute to its classification, and areas that push it into another class. A wide and deep experimental evaluation proves that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability, besides providing the most useful and interpretable explanations.
@article{GMM2020, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, doi = {10.1609/aaai.v34i09.7116}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,4}, month = apr, number = {09}, open_access = {NO}, pages = {13665–13668}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations}, visible_on_website = {YES}, volume = {34}, year = {2020} }
Doctor XAI: an ontology-based approach to black-box sequential data classification explanations

Cecilia Panigutti, Alan Perotti, and Dino Pedreschi

In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , Jan 2020

RESEARCH LINE
1

3

4

Abs Bib HTML

Several recent advancements in Machine Learning involve blackbox models: algorithms that do not provide human-understandable explanations in support of their decisions. This limitation hampers the fairness, accountability and transparency of these models; the field of eXplainable Artificial Intelligence (XAI) tries to solve this problem providing human-understandable explanations for black-box models. However, healthcare datasets (and the related learning tasks) often present peculiar features, such as sequential data, multi-label predictions, and links to structured background knowledge. In this paper, we introduce Doctor XAI, a model-agnostic explainability technique able to deal with multi-labeled, sequential, ontology-linked data. We focus on explaining Doctor AI, a multilabel classifier which takes as input the clinical history of a patient in order to predict the next visit. Furthermore, we show how exploiting the temporal dimension in the data and the domain knowledge encoded in the medical ontology improves the quality of the mined explanations.
@inproceedings{PPP2020, author = {Panigutti, Cecilia and Perotti, Alan and Pedreschi, Dino}, booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency}, collection = {FAT* ’20}, doi = {10.1145/3351095.3372855}, line = {1,3,4}, month = jan, open_access = {NO}, pages = {629–639}, publisher = {ACM}, series = {FAT* ’20}, title = {Doctor XAI: an ontology-based approach to black-box sequential data classification explanations}, visible_on_website = {YES}, year = {2020} }
Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{GMS2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, open_access = {NO}, pages = {167–176}, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, visible_on_website = {YES}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Helping Your Docker Images to Spread Based on Explainable Models

Riccardo Guidotti, Jacopo Soldani, Davide Neri, Antonio Brogi, and Dino Pedreschi

Dec 2019

RESEARCH LINE
1

Abs Bib HTML

Docker is on the rise in today’s enterprise IT. It permits shipping applications inside portable containers, which run from so-called Docker images. Docker images are distributed in public registries, which also monitor their popularity. The popularity of an image impacts on its actual usage, and hence on the potential revenues for its developers. In this paper, we present a solution based on interpretable decision tree and regression trees for estimating the popularity of a given Docker image, and for understanding how to improve an image to increase its popularity. The results presented in this work can provide valuable insights to Docker developers, helping them in spreading their images. Code related to this paper is available at: https://github.com/di-unipi-socc/DockerImageMiner.
@inbook{GSN2021, author = {Guidotti, Riccardo and Soldani, Jacopo and Neri, Davide and Brogi, Antonio and Pedreschi, Dino}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-10997-4_13}, isbn = {9783030109974}, issn = {1611-3349}, line = {1}, pages = {205–221}, publisher = {Springer International Publishing}, title = {Helping Your Docker Images to Spread Based on Explainable Models}, visible_on_website = {YES}, year = {2019} }
Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }
Explaining Multi-label Black-Box Classifiers for Health Applications

Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi

Aug 2019

RESEARCH LINE
1

4

Abs Bib HTML

Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{PGM2019, author = {Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino}, booktitle = {Precision Health and Medicine}, doi = {10.1007/978-3-030-24409-5_9}, isbn = {9783030244095}, issn = {1860-9503}, line = {1,4}, month = aug, pages = {97–110}, publisher = {Springer International Publishing}, title = {Explaining Multi-label Black-Box Classifiers for Health Applications}, visible_on_website = {YES}, year = {2019} }
The AI black box explanation problem

Guidotti Riccardo, Monreale Anna, and Pedreschi Dino

Dec 2019

RESEARCH LINE
1

2

3

Abs Bib HTML

The use of machine learning in decision-making has triggered an intense debate about “fair algorithms”. Given that fairness intuitions differ and can led to conflicting technical requirements, there is a pressing need to integrate ethical thinking into research and design of machine learning. We outline a framework showing how this can be done.
@misc{GMP2019, author = {Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi}, line = {1,2,3}, month = dec, publisher = {ERCIM – the European Research Consortium for Informatics and Mathematics}, title = {The AI black box explanation problem}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2025

SafeGen: safeguarding privacy and fairness through a genetic method

Martina Cinquini, Marta Marchiori Manerba, Federico Mazzoni, Francesca Pratesi, and Riccardo Guidotti

Machine Learning, Sep 2025

RESEARCH LINE
5

Abs Bib HTML

To ensure that Machine Learning systems produce unharmful outcomes, pursuing a joint optimization of performance and ethical profiles such as privacy and fairness is crucial. However, jointly optimizing these two ethical dimensions while maintaining predictive accuracy remains a fundamental challenge. Indeed, privacy-preserving techniques may worsen fairness and restrain the model’s ability to learn accurate statistical patterns, while data mitigation techniques may inadvertently compromise privacy. Aiming to bridge this gap, we propose safeGen, a preprocessing fairness enhancing and privacy-preserving method for tabular data. SafeGen employs synthetic data generation through a genetic algorithm to ensure that sensitive attributes are protected while maintaining the necessary statistical properties. We assess our method across multiple datasets, comparing it against state-of-the-art privacy-preserving and fairness approaches through a threefold evaluation: privacy preservation, fairness enhancement, and generated data plausibility. Through extensive experiments, we demonstrate that SafeGen consistently achieves strong anonymization while preserving or improving dataset fairness across several benchmarks. Additionally, through hybrid privacy-fairness constraints and the use of a genetic synthesizer, SafeGen ensures the plausibility of synthetic records while minimizing discrimination. Our findings demonstrate that modeling fairness and privacy within a unified generative method yields significantly better outcomes than addressing these constraints separately, reinforcing the importance of integrated approaches when multiple ethical objectives must be simultaneously satisfied.
@article{CMM2025, author = {Cinquini, Martina and Marchiori Manerba, Marta and Mazzoni, Federico and Pratesi, Francesca and Guidotti, Riccardo}, doi = {10.1007/s10994-025-06835-9}, issn = {1573-0565}, journal = {Machine Learning}, line = {5}, month = sep, number = {10}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {SafeGen: safeguarding privacy and fairness through a genetic method}, visible_on_website = {YES}, volume = {114}, year = {2025} }
A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods

Martina Cinquini, Karima Makhlouf, Sami Zhioua, Catuscia Palamidessi, and Riccardo Guidotti

IEEE Access, Sep 2025

RESEARCH LINE
2

5

Abs Bib HTML

Causal discovery (CD) algorithms are increasingly applied to socially and ethically sensitive domains. However, their evaluation under realistic conditions remains challenging due to the scarcity of real-world datasets annotated with ground-truth causal structures. Whereas synthetic data generators support controlled benchmarking, they often overlook forms of bias, such as dependencies involving sensitive attributes, which may significantly affect the observed distribution and compromise the trustworthiness of downstream analysis. This paper introduces a novel synthetic data generation framework that enables controlled bias injection while preserving the causal relationships specified in a ground-truth causal graph. The framework aims to evaluate the reliability of CD methods by examining the impact of varying bias levels and outcome binarization thresholds. Experimental results show that even moderate bias levels can lead CD approaches to fail to correctly infer causal links, particularly those connecting sensitive attributes to decision outcomes. These findings underscore the need for expert validation and highlight the limitations of current CD methods in fairness-critical applications. Our proposal thus provides an essential tool for benchmarking and improving CD algorithms in biased, real-world data settings.
@article{CMZ2025, author = {Cinquini, Martina and Makhlouf, Karima and Zhioua, Sami and Palamidessi, Catuscia and Guidotti, Riccardo}, doi = {10.1109/access.2025.3573201}, issn = {2169-3536}, journal = {IEEE Access}, line = {2,5}, open_access = {Gold}, pages = {97376–97391}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods}, visible_on_website = {YES}, volume = {13}, year = {2025} }
Balancing Fairness and Interpretability in Clustering with FairParTree

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

The revolution involving Machine Learning has transformed data analytics, making algorithms important in decision-making processes across various domains, even in sensitive scenarios. Indeed, traditional clustering algorithms often lack interpretability and exhibit biases, leading to discriminatory practices and opaque decision-making. To overcome these limitations, we introduce FairParTree, a fair and interpretable clustering algorithm that integrates fairness constraints directly into the clustering process, ensuring that the resulting clusters do not disproportionately disadvantage any particular group. By leveraging the structure of decision trees, FairParTree enhances the interpretability of clustering results by providing clear and understandable motivations for cluster assignments through rule-based explanations. We evaluate FairParTree against state-of-the-art competitors. Through extensive experiments, we show that it maintains strong performances w.r.t. fairness, interpretability, and clustering quality across different dataset sizes, thus positioning itself as a competitive, fair, and interpretable clustering algorithm.
@inbook{LCM2025c, author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_5}, isbn = {9783032083241}, issn = {1865-0937}, line = {1}, month = oct, open_access = {NO}, pages = {104–127}, publisher = {Springer Nature Switzerland}, title = {Balancing Fairness and Interpretability in Clustering with FairParTree}, visible_on_website = {YES}, year = {2025} }
MASCOTS: Model-Agnostic Symbolic COunterfactual Explanations for Time Series

Dawid Płudowski, Francesco Spinnato, Piotr Wilczyński, Krzysztof Kotowski, Evridiki Vasileia Ntagiou, and 2 more authors

Sep 2025

RESEARCH LINE
1

Abs Bib HTML

Counterfactual explanations provide an intuitive way to understand model decisions by identifying minimal changes required to alter an outcome. However, applying counterfactual methods to time series models remains challenging due to temporal dependencies, high dimensionality, and the lack of an intuitive human-interpretable representation. We introduce MASCOTS, a method that leverages the Bag-of-Receptive-Fields representation alongside symbolic transformations inspired by Symbolic Aggregate Approximation. By operating in a symbolic feature space, it enhances interpretability while preserving fidelity to the original data and model. Unlike existing approaches that either depend on model structure or autoencoder-based sampling, MASCOTS directly generates meaningful and diverse counterfactual observations in a model-agnostic manner, operating on both univariate and multivariate data. We evaluate MASCOTS on univariate and multivariate benchmark datasets, demonstrating comparable validity, proximity, and plausibility to state-of-the-art methods, while significantly improving interpretability and sparsity. Its symbolic nature allows for explanations that can be expressed visually, in natural language, or through semantic representations, making counterfactual reasoning more accessible and actionable.
@inbook{PSW2025, author = {Płudowski, Dawid and Spinnato, Francesco and Wilczyński, Piotr and Kotowski, Krzysztof and Ntagiou, Evridiki Vasileia and Guidotti, Riccardo and Biecek, Przemysław}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-032-06078-5_6}, isbn = {9783032060785}, issn = {1611-3349}, line = {1}, month = sep, open_access = {Gold}, pages = {94–112}, publisher = {Springer Nature Switzerland}, title = {MASCOTS: Model-Agnostic Symbolic COunterfactual Explanations for Time Series}, visible_on_website = {YES}, year = {2025} }
Balancing Fairness and Interpretability in Clustering with FairParTree

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

5

Abs Bib HTML

The revolution involving Machine Learning has transformed data analytics, making algorithms important in decision-making processes across various domains, even in sensitive scenarios. Indeed, traditional clustering algorithms often lack interpretability and exhibit biases, leading to discriminatory practices and opaque decision-making. To overcome these limitations, we introduce FairParTree, a fair and interpretable clustering algorithm that integrates fairness constraints directly into the clustering process, ensuring that the resulting clusters do not disproportionately disadvantage any particular group. By leveraging the structure of decision trees, FairParTree enhances the interpretability of clustering results by providing clear and understandable motivations for cluster assignments through rule-based explanations. We evaluate FairParTree against state-of-the-art competitors. Through extensive experiments, we show that it maintains strong performances w.r.t. fairness, interpretability, and clustering quality across different dataset sizes, thus positioning itself as a competitive, fair, and interpretable clustering algorithm.
@inbook{LCM2025, author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_5}, isbn = {9783032083241}, issn = {1865-0937}, line = {1,5}, month = oct, open_access = {Gold}, pages = {104–127}, publisher = {Springer Nature Switzerland}, title = {Balancing Fairness and Interpretability in Clustering with FairParTree}, visible_on_website = {YES}, year = {2025} }
An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items

Luca Corbucci, Javier Alejandro Borges Legrottaglie, Francesco Spinnato, Anna Monreale, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

Accurately identifying items forgotten during a supermarket visit and providing clear, interpretable explanations for recommending them remains an underexplored problem within the Next Basket Prediction (NBP) domain. Existing NBP approaches typically only focus on forecasting future purchases, without explicitly addressing the detection of unintentionally omitted items. This gap is partly due to the scarcity of real-world datasets that allow for the reliable estimation of forgotten items. Furthermore, most current NBP methods rely on black-box models, which lack transparency and limit the ability to justify recommendations to end users. In this paper, we formally introduce the forgotten-item prediction task and propose two novel interpretable-by-design algorithms. These methods are tailored to identify forgotten items while offering intuitive, human-understandable explanations. Experiments on a real-world retail dataset show our algorithms outperform state-of-the-art NBP baselines by 10–15% across multiple evaluation metrics.
@inbook{CBS2025, author = {Corbucci, Luca and Borges Legrottaglie, Javier Alejandro and Spinnato, Francesco and Monreale, Anna and Guidotti, Riccardo}, booktitle = {ECAI 2025}, doi = {10.3233/faia250912}, isbn = {9781643686318}, issn = {1879-8314}, line = {1}, month = oct, open_access = {Gold}, publisher = {IOS Press}, title = {An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items}, visible_on_website = {YES}, year = {2025} }
Group Explainability Through Local Approximation

Mattia Setzu, Riccardo Guidotti, Dino Pedreschi, and Fosca Giannotti

Oct 2025

RESEARCH LINE
2

Abs Bib HTML

Machine learning models are becoming increasingly complex and widely adopted. Interpretable machine learning allows us to not only make predictions but also understand the rationale behind automated decisions through explanations. Explanations are typically characterized by their scope: local explanations are generated by local surrogate models for specific instances, while global explanations aim to approximate the behavior of the entire black-box model. In this paper, we break this dichotomy of locality to explore an underexamined area that lies between these two extremes: meso-level explanations. The goal of meso-level explainability is to provide explanations using a set of meso-level interpretable models, which capture patterns at an intermediate level of abstraction. To this end, we propose GrouX, an explainable-by-design algorithm that generates meso-level explanations in the form of feature importance scores. Our approach includes a partitioning phase that identifies meso groups, followed by the training of interpretable models within each group. We evaluate GrouX on a collection of tabular datasets, reporting both the accuracy and complexity of the resulting meso models, and compare it against other meso-level explainability algorithms. Additionally, we analyze the algorithm’s sensitivity to its hyperparameters to better understand its behavior and robustness.
@inbook{SGP2025, address = {ECAI 2025}, author = {Setzu, Mattia and Guidotti, Riccardo and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {ECAI 2025}, doi = {10.3233/faia250902}, isbn = {9781643686318}, issn = {1879-8314}, line = {2}, month = oct, open_access = {Gold}, pages = {952 - 958}, publisher = {IOS Press}, title = {Group Explainability Through Local Approximation}, visible_on_website = {YES}, year = {2025} }
Interpretable Instance-Based Learning Through Pairwise Distance Trees

Andrea Fedele, Alessio Cascione, Riccardo Guidotti, and Cristiano Landi

Sep 2025

RESEARCH LINE
1

Abs Bib HTML

Instance-based models offer natural interpretability by making decisions based on concrete examples. However, their transparency is often hindered by the use of complex similarity measures, which are difficult to interpret, especially in high-dimensional datasets. To address this issue, this paper presents a meta-learning framework that enhances the interpretability of instance-based models by replacing traditional, complex pairwise distance functions with interpretable pairwise distance trees. These trees are designed to prioritize simplicity and transparency while preserving the model’s effectiveness. By offering a clear decision-making process, the framework makes the instance selection more understandable. Also, the framework mitigates the computational burden of instance-based models, which typically require calculating all pairwise distances. Leveraging the generalization capabilities of pairwise distance trees and employing sampling strategies to select representative subsets, the method significantly reduces computational complexity. Our experiments demonstrate that the proposed approach improves computational efficiency with only a modest trade-off in accuracy while substantially enhancing the interpretability of the learned distance measure.
@inbook{FCG2025, author = {Fedele, Andrea and Cascione, Alessio and Guidotti, Riccardo and Landi, Cristiano}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-032-06078-5_1}, isbn = {9783032060785}, issn = {1611-3349}, line = {1}, month = sep, open_access = {Gold}, pages = {3–21}, publisher = {Springer Nature Switzerland}, title = {Interpretable Instance-Based Learning Through Pairwise Distance Trees}, visible_on_website = {YES}, year = {2025} }
Unsupervised and Interpretable Detection of User Personalities in Online Social Networks

Alessio Cascione, Laura Pollacci, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

Personalized moderation interventions in online social networks foster healthier interactions by adapting responses to both individual traits and contextual factors. However, implementing such interventions is challenging due to transparency concerns and the lack of ground-truth behavioral data from expert psychologists. Interpretability is crucial for addressing these challenges, as it enables platforms to tailor moderation strategies while ensuring fairness and user trust. In this paper, we present an unsupervised, data-driven framework to build an interpretable predictive model capable of distinguishing between toxic and non-toxic users with different personality traits. We leverage personality representations from an external resource to uncover behavioral profiles through clustering, utilizing embeddings of both toxic and non-toxic users. Then, we model users with features capturing linguistic and affective dimensions, training an interpretable personality detector capable of distinguishing between behavioral profiles in a transparent and explainable manner. A case study on Reddit demonstrates the effectiveness of our approach, highlighting how an interpretable model can achieve competitive performance comparable to a black-box alternative while offering meaningful insights into toxic and non-toxic users behavior.
@inbook{CPG2025, author = {Cascione, Alessio and Pollacci, Laura and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08327-2_8}, isbn = {9783032083272}, issn = {1865-0937}, line = {1}, month = oct, open_access = {Gold}, pages = {162–179}, publisher = {Springer Nature Switzerland}, title = {Unsupervised and Interpretable Detection of User Personalities in Online Social Networks}, visible_on_website = {YES}, year = {2025} }

A Note on Methods for Explainable Malware Analysis

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Dec 2025

RESEARCH LINE

Bib

@misc{LCM2025b,
  author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo},
  line = {1},
  month = dec,
  title = {A Note on Methods for Explainable Malware Analysis},
  year = {2025}
}

Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding

Simone Piaggesi, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Dec 2025

RESEARCH LINE
1

Bib HTML
@misc{PGG2025, author = {Piaggesi, Simone and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, line = {1}, month = dec, title = {Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding}, year = {2025} }

2024

Enhancing Echo State Networks with Gradient-based Explainability Methods

Francesco Spinnato, Andrea Cossu, Riccardo Guidotti, Andrea Ceni, Claudio Gallicchio, and 1 more author

In ESANN 2024 proceesdings , Dec 2024

RESEARCH LINE
1

Abs Bib HTML

In sequence classification problems, the readout typically receives only the final state of the reservoir. However, averaging all states can be beneficial. In this work, we assess whether a weighted average of hidden states can enhance Echo State Network performance. We propose a gradient-based explainable technique to guide the contribution of each hidden state toward the final prediction. Our approach outperforms the naive average and other baselines in time series classification, particularly on noisy data.
@inproceedings{SCG2024, author = {Spinnato, Francesco and Cossu, Andrea and Guidotti, Riccardo and Ceni, Andrea and Gallicchio, Claudio and Bacciu, Davide}, booktitle = {ESANN 2024 proceesdings}, collection = {ESANN 2024}, doi = {10.14428/esann/2024.es2024-78}, line = {1}, open_access = {Gold}, pages = {17–22}, publisher = {Ciaco - i6doc.com}, series = {ESANN 2024}, title = {Enhancing Echo State Networks with Gradient-based Explainability Methods}, visible_on_website = {YES}, year = {2024} }
Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

IEEE Access, Dec 2024

RESEARCH LINE
1

Abs Bib HTML

The current trend in time series classification is to develop highly accurate algorithms by combining multiple models in ensemble hybrids, representing time series in complex feature spaces, and extracting features from different representations. As a consequence, the best time series classifiers are black-box models, not understandable for humans. Even the approaches regarded as interpretable, such as shapelet-based ones, rely on randomization to maintain efficiency, which poses challenges for interpretability. We propose the Bag-Of-Receptive-Fields (BORF), a fast, interpretable, and deterministic time series transform. Building on the Bag-Of-Patterns, we bridge the gap between convolutional operators and discretization, enhancing the Symbolic Aggregate Approximation with dilation and stride to better capture temporal patterns at multiple scales. We propose an algorithmic speedup that reduces the time complexity associated with SAX-based classifiers, enabling the more flexible BORF, represented as a sparse multivariate tensor.
@article{SGM2024, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, doi = {10.1109/access.2024.3464743}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {137893–137912}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Shape-based Methods in Mobility Data Analysis: Effectiveness and Limitations

Cristiano Landi, and Riccardo Guidotti

Nov 2024

RESEARCH LINE
1

Abs Bib HTML

Although Mobility Data Analysis (MDA) has been explored for a long time, it still lags behind advancements in other fields.A common issue in MDA is the lack of methods’ standardization and reusability. On the other hand, for instance, in time series analysis, the existing methods are typically general-purpose, and it is possible to apply them across diverse datasets and applications without extensive customization. Still, in MDA, most contributions are ad-hoc and designed to address specific research questions, which limits their generalizability and reusability. Recently, some researchers explored the application of shapelet transform to trajectory data, i.e., extracting discriminatory sub-trajectories from training data to be used as classification features. Unlike current MDA methods, this line of research eliminates the need for feature engineering, greatly improving its ability to generalize. While shapelets on mobility data have shown state-of-the-art performance on public classification datasets, it is still not clear why they work. Are these subtrajectories merely proxies for geographic location, or do they also capture motion dynamics? We empirically show that shapelet-based approaches are a viable alternative to classical methods and flexible enough to solve MDA tasks related solely to trajectory shape, solely to movement dynamics, and those related to both. Additionally, we investigate the problem of Geographic Transferability, showing that such approaches offer a promising starting point for tackling this challenge.
@article{LG2024, author = {Landi, Cristiano and Guidotti, Riccardo}, doi = {10.21203/rs.3.rs-5369626/v1}, line = {1}, month = nov, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Shape-based Methods in Mobility Data Analysis: Effectiveness and Limitations}, visible_on_website = {YES}, year = {2024} }
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers

Guillermo Fernandez, Riccardo Guidotti, Fosca Giannotti, Mattia Setzu, Juan A. Aledo, and 2 more authors

Nov 2024

RESEARCH LINE
1

2

Abs Bib HTML

The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inbook{FGG2024, address = {Cham, Switzerland}, author = {Fernandez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and Gámez, Jose A. and Puerta, Jose M.}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_16}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {197–209}, publisher = {Springer Nature Switzerland}, title = {FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers}, visible_on_website = {YES}, year = {2024} }
Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions

Luca Longo, Mario Brcic, Federico Cabitza, Jaesik Choi, Roberto Confalonieri, and 14 more authors

Information Fusion, Jun 2024

RESEARCH LINE
1

Abs Bib HTML

As systems based on opaque Artificial Intelligence (AI) continue to flourish in diverse real-world applications, understanding these black box models has become paramount. In response, Explainable AI (XAI) has emerged as a field of research with practical and ethical benefits across various domains. This paper not only highlights the advancements in XAI and its application in real-world scenarios but also addresses the ongoing challenges within XAI, emphasizing the need for broader perspectives and collaborative efforts. We bring together experts from diverse fields to identify open problems, striving to synchronize research agendas and accelerate XAI in practical applications. By fostering collaborative discussion and interdisciplinary cooperation, we aim to propel XAI forward, contributing to its continued success. Our goal is to put forward a comprehensive proposal for advancing XAI. To achieve this goal, we present a manifesto of 27 open problems categorized into nine categories. These challenges encapsulate the complexities and nuances of XAI and offer a road map for future research. For each problem, we provide promising research directions in the hope of harnessing the collective intelligence of interested stakeholders.
@article{LBC2024, author = {Longo, Luca and Brcic, Mario and Cabitza, Federico and Choi, Jaesik and Confalonieri, Roberto and Ser, Javier Del and Guidotti, Riccardo and Hayashi, Yoichi and Herrera, Francisco and Holzinger, Andreas and Jiang, Richard and Khosravi, Hassan and Lecue, Freddy and Malgieri, Gianclaudio and Páez, Andrés and Samek, Wojciech and Schneider, Johannes and Speith, Timo and Stumpf, Simone}, doi = {10.1016/j.inffus.2024.102301}, issn = {1566-2535}, journal = {Information Fusion}, line = {1}, month = jun, open_access = {Gold}, pages = {102301}, publisher = {Elsevier BV}, title = {Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions}, visible_on_website = {YES}, volume = {106}, year = {2024} }
Data-Agnostic Pivotal Instances Selection for Decision-Making Models

Alessio Cascione, Mattia Setzu, and Riccardo Guidotti

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

As decision-making processes grow increasingly complex, machine learning tools have become essential in tackling business and societal challenges. However, many methodologies rely on complex models that are difficult for experts and users to interpret. We propose a hierarchical and interpretable pivot selection model inspired by Decision Trees, selecting representative pivotal instances based on similarity. The approach is data-agnostic and can leverage pretrained networks for data transformation. Experiments across tabular, text, image, and time-series datasets show superior performance to naive and state-of-the-art instance selectors, while minimizing the number of pivots and maintaining interpretability.
@inbook{CSG2024b, author = {Cascione, Alessio and Setzu, Mattia and Guidotti, Riccardo}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-031-70341-6_22}, isbn = {9783031703416}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {367–386}, publisher = {Springer Nature Switzerland}, title = {Data-Agnostic Pivotal Instances Selection for Decision-Making Models}, visible_on_website = {YES}, year = {2024} }
Drifting explanations in continual learning

Andrea Cossu, Francesco Spinnato, Riccardo Guidotti, and Davide Bacciu

Neurocomputing, Sep 2024

RESEARCH LINE
1

Abs Bib HTML

Continual Learning (CL) trains models on streams of data, with the aim of learning new information without forgetting previous knowledge. We study the behavior of different explanation methods in CL and propose CLEX (ContinuaL EXplanations), an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios, where forgetting is pronounced. We observed that models with similar predictive accuracy do not generate similar explanations. (openportal.isti.cnr.it)
@article{CSG2024, author = {Cossu, Andrea and Spinnato, Francesco and Guidotti, Riccardo and Bacciu, Davide}, doi = {10.1016/j.neucom.2024.127960}, issn = {0925-2312}, journal = {Neurocomputing}, line = {1}, month = sep, open_access = {Gold}, pages = {127960}, publisher = {Elsevier BV}, title = {Drifting explanations in continual learning}, visible_on_website = {YES}, volume = {597}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Social Bias Probing: Fairness Benchmarking for Language Models

Marta Marchiori Manerba, Karolina Stanczak, Riccardo Guidotti, and Isabelle Augenstein

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Apr 2024

RESEARCH LINE
5

Abs Bib HTML

While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limiting our understanding of bias complexities. This paper proposes a novel framework for probing language models for social biases by assessing disparate treatment, which involves treating individuals differently according to their affiliation with a sensitive demographic group. We curate SoFa, a large-scale benchmark designed to address the limitations of existing fairness collections. SoFa expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. Comparing our methodology with existing benchmarks, we reveal that biases within language models are more nuanced than acknowledged, indicating a broader scope of encoded biases than previously recognized. Benchmarking LMs on SoFa, we expose how identities expressing different religions lead to the most pronounced disparate treatments across all models. Finally, our findings indicate that real-life adversities faced by various groups such as women and people with disabilities are mirrored in the behavior of these models. (arXiv)
@inproceedings{MSG2024, author = {Marchiori Manerba, Marta and Stanczak, Karolina and Guidotti, Riccardo and Augenstein, Isabelle}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, doi = {10.18653/v1/2024.emnlp-main.812}, line = {5}, open_access = {Gold}, pages = {14653–14671}, publisher = {Association for Computational Linguistics}, title = {Social Bias Probing: Fairness Benchmarking for Language Models}, visible_on_website = {YES}, year = {2024} }
Causality-Aware Local Interpretable Model-Agnostic Explanations

Martina Cinquini, and Riccardo Guidotti

Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A main drawback of eXplainable Artificial Intelligence (XAI) approaches is the feature independence assumption, hindering the study of potential variable dependencies. This leads to approximating black box behaviors by analysing the effects on randomly generated feature values that may rarely occur in the original samples. This paper addresses this issue by integrating causal knowledge in an XAI method to enhance transparency and enable users to assess the quality of the generated explanations. Specifically, we propose a novel extension to a widely used local and model-agnostic explainer, which encodes explicit causal relationships within the data surrounding the instance being explained. Extensive experiments show that our approach overcomes the original method in terms of faithfully replicating the black-box model’s mechanism and the consistency and reliability of the generated explanations. (arXiv)
@inbook{CG2024, author = {Cinquini, Martina and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-63800-8_6}, isbn = {9783031638008}, issn = {1865-0937}, line = {1,2}, open_access = {NO}, pages = {108–124}, publisher = {Springer Nature Switzerland}, title = {Causality-Aware Local Interpretable Model-Agnostic Explanations}, visible_on_website = {YES}, year = {2024} }
A Frank System for Co-Evolutionary Hybrid Decision-Making

Federico Mazzoni, Riccardo Guidotti, and Alessio Malizia

Apr 2024

RESEARCH LINE
1

3

4

Abs Bib HTML

Hybrid decision-making systems combine human judgment with algorithmic recommendations, yet coordinating these two sources of information remains challenging. We present FRANK, a co-evolutionary framework enabling humans and AI agents to iteratively exchange feedback and refine decisions over time. FRANK integrates rule-based reasoning, preference modeling, and a learning module that adapts recommendations based on user interaction. Through simulated and real-user experiments, we show that the co-evolution process helps users converge toward more stable and accurate decisions while increasing perceived transparency. The system allows humans to override or modify machine suggestions while the AI agent reshapes its internal models in response to human rationale. FRANK thus promotes a collaborative decision environment where human expertise and machine learning strengthen each other.
@inbook{MBP2024, author = {Mazzoni, Federico and Guidotti, Riccardo and Malizia, Alessio}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_19}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,3,4}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {A Frank System for Co-Evolutionary Hybrid Decision-Making}, visible_on_website = {YES}, year = {2024} }
Explaining Siamese networks in few-shot learning

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Machine Learning, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Siamese neural networks are widely used in few-shot learning tasks thanks to their ability to compare pairs of samples and generalize from very limited labeled data. However, their internal decision-making process remains opaque, since similarity-based representations do not provide intuitive explanations for end users. In this work, we investigate how to explain Siamese networks by attributing contribution scores to both input samples involved in the comparison. We introduce an explanation method specifically tailored to pairwise architectures, producing two synchronized saliency maps that highlight which regions of the support and query examples drive the similarity judgment. We evaluate the approach on image-based few-shot classification benchmarks, showing that the explanations highlight semantically meaningful structures and remain consistent across different evaluation episodes.
@article{FGP2024, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, doi = {10.1007/s10994-024-06529-8}, issn = {1573-0565}, journal = {Machine Learning}, line = {1}, month = apr, number = {10}, open_access = {Gold}, pages = {7723–7760}, publisher = {Springer Science and Business Media LLC}, title = {Explaining Siamese networks in few-shot learning}, visible_on_website = {YES}, volume = {113}, year = {2024} }
Generative Model for Decision Trees

Riccardo Guidotti, Anna Monreale, Mattia Setzu, and Giulia Volpi

Proceedings of the AAAI Conference on Artificial Intelligence, Mar 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are among the most popular supervised models due to their interpretability and knowledge representation resembling human reasoning. Yet their design and tuning remain largely manual and analytic. In this work we place our proposal between discriminative-only design and full generative modeling: we design a generative model for decision trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. We then sample novel trees aimed at specific tasks, such as improving interpretability, compression, or fairness. Empirical results on synthetic and real data demonstrate that our generative model successfully produces new decision trees tailored to different desiderata while preserving predictive performance.
@article{GMS2024, author = {Guidotti, Riccardo and Monreale, Anna and Setzu, Mattia and Volpi, Giulia}, doi = {10.1609/aaai.v38i19.30104}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = mar, number = {19}, open_access = {Gold}, pages = {21116–21124}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Generative Model for Decision Trees}, visible_on_website = {YES}, volume = {38}, year = {2024} }
Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space

Simone Piaggesi, Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

IEEE Access, Mar 2024

RESEARCH LINE
1

Abs Bib HTML

We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities similarly to well-known dimensionality reduction techniques. Our approach introduces a transparent latent space optimized for interpretability of both counterfactual and prototypical explanations for tabular data. The approach enables the easy extraction of local and global explanations and ensures that the latent space preserves similarity relations, enabling meaningful prototypical and counterfactual examples for any classifier.
@article{PBG2024, author = {Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1109/access.2024.3496114}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {168983–169000}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Requirements of eXplainable AI in Algorithmic Hiring

A. Beretta, G. Ercoli, A. Ferraro, R. Guidotti, A. Iommi, and 4 more authors

Dec 2024

RESEARCH LINE
1

Abs Bib HTML

AI models for ranking candidates to a job position are increasingly adopted. They bring a new layer of opaqueness in the way candidates are evaluated. We present preliminary research on stakeholder analysis and requirement elicitation for designing an explainability component in AI models for ranking candidates to a job position. (CEUR-WS)
@misc{BEF2024, author = {Beretta, A. and Ercoli, G. and Ferraro, A. and Guidotti, R. and Iommi, A. and Mastropietro, A. and Monreale, A. and Rotelli, D. and Ruggieri, S.}, line = {1}, month = dec, title = {Requirements of eXplainable AI in Algorithmic Hiring}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }
Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }
Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
Text to Time Series Representations: Towards Interpretable Predictive Models

Mattia Poggioli, Francesco Spinnato, and Riccardo Guidotti

Jun 2023

RESEARCH LINE
1

Abs Bib HTML

In this paper, we investigate the impact of converting text observations into time series observations to solve interpretable text classification through time series representations. By considering temporal dependencies, TSA can be used for various purposes, such as descriptive analysis, clustering, classification, and forecasting. We propose using shapelets in NLP by turning texts into time series. To perform this transformation, we design and implement TOTS, a framework to turn text to time series. TOTS exploits a range of different conversion alternatives for tokenization, feature extraction and aggregation. We formalize our Text to Time Series framework as a feature extraction and aggregation process, proposing a set of different conversion alternatives for each step. We then exploit interpretable models originally developed for time series (e.g., shapelet-based models) as interpretable text classifiers. Our experiments show promising results for classification and interpretability.
@inbook{PSG2023, author = {Poggioli, Mattia and Spinnato, Francesco and Guidotti, Riccardo}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_16}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {230–245}, publisher = {Springer Nature Switzerland}, title = {Text to Time Series Representations: Towards Interpretable Predictive Models}, visible_on_website = {YES}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Handling Missing Values in Local Post-hoc Explainability

Martina Cinquini, Fosca Giannotti, Riccardo Guidotti, and Andrea Mattei

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

Missing data are quite common in real scenarios when using Artificial Intelligence (AI) systems for decision-making with tabular data and effectively handling them poses a significant challenge for such systems. While some machine learning models used by AI systems can tackle this problem, the existing literature lacks post-hoc explainability approaches able to deal with predictors that encounter missing data. In this paper, we extend a widely used local model-agnostic post-hoc explanation approach that enables explainability in the presence of missing values by incorporating state-of-the-art imputation methods within the explanation process. Since our proposal returns explanations in the form of feature importance, the user will be aware also of the importance of a missing value in a given record for a particular prediction. Extensive experiments show the effectiveness of the proposed method with respect to some baseline solutions relying on traditional data imputation.
@inbook{CGG2023, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo and Mattei, Andrea}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_14}, isbn = {9783031440670}, issn = {1865-0937}, line = {1}, open_access = {Gold}, pages = {256–278}, publisher = {Springer Nature Switzerland}, title = {Handling Missing Values in Local Post-hoc Explainability}, visible_on_website = {YES}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{LSG2023, address = {Cham, Switzerland}, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Advances in Intelligent Data Analysis XXI}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, visible_on_website = {YES}, year = {2023} }

2022

Counterfactual explanations and how to find them: literature review and benchmarking

Riccardo Guidotti

Data Mining and Knowledge Discovery, Apr 2022

RESEARCH LINE
1

Abs Bib HTML

Interpretable machine learning aims at unveiling the reasons behind predictions returned by uninterpretable classifiers. One of the most valuable types of explanation consists of counterfactuals. A counterfactual explanation reveals what should have been different in an instance to observe a diverse outcome. For instance, a bank customer asks for a loan that is rejected. The counterfactual explanation consists of what should have been different for the customer in order to have the loan accepted. Recently, there has been an explosion of proposals for counterfactual explainers. The aim of this work is to survey the most recent explainers returning counterfactual explanations. We categorize explainers based on the approach adopted to return the counterfactuals, and we label them according to characteristics of the method and properties of the counterfactuals returned. In addition, we visually compare the explanations, and we report quantitative benchmarking assessing minimality, actionability, stability, diversity, discriminative power, and running time. The results make evident that the current state of the art does not provide a counterfactual explainer able to guarantee all these properties simultaneously.
@article{G2022, address = {Netherlands}, author = {Guidotti, Riccardo}, doi = {10.1007/s10618-022-00831-6}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1}, month = apr, number = {5}, open_access = {Gold}, pages = {2770–2824}, publisher = {Springer Science and Business Media LLC}, title = {Counterfactual explanations and how to find them: literature review and benchmarking}, visible_on_website = {YES}, volume = {38}, year = {2022} }
Interpretable Latent Space to Enable Counterfactual Explanations

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Apr 2022

RESEARCH LINE
1

Abs Bib HTML

Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{BGG2023c, address = {Montpellier, France}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_37}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {525–540}, publisher = {Springer Nature Switzerland}, title = {Interpretable Latent Space to Enable Counterfactual Explanations}, visible_on_website = {YES}, year = {2022} }
Explaining Siamese Networks in Few-Shot Learning for Audio Data

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Apr 2022

RESEARCH LINE
4

Abs Bib HTML

Machine learning models are not able to generalize correctly when queried on samples belonging to class distributions that were never seen during training. This is a critical issue, since real world applications might need to quickly adapt without the necessity of re-training. To overcome these limitations, few-shot learning frameworks have been proposed and their applicability has been studied widely for computer vision tasks. Siamese Networks learn pairs similarity in form of a metric that can be easily extended on new unseen classes. Unfortunately, the downside of such systems is the lack of explainability. We propose a method to explain the outcomes of Siamese Networks in the context of few-shot learning for audio data. This objective is pursued through a local perturbation-based approach that evaluates segments-weighted-average contributions to the final outcome considering the interplay between different areas of the audio spectrogram. Qualitative and quantitative results demonstrate that our method is able to show common intra-class characteristics and erroneous reliance on silent sections.
@inbook{FGP2022, address = {Cham, Switzerland}, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_36}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {509–524}, publisher = {Springer Nature Switzerland}, title = {Explaining Siamese Networks in Few-Shot Learning for Audio Data}, visible_on_website = {YES}, year = {2022} }
Explaining Crash Predictions on Multivariate Time Series Data

Francesco Spinnato, Riccardo Guidotti, Mirco Nanni, Daniele Maccagnola, Giulia Paciello, and 1 more author

Apr 2022

RESEARCH LINE
4

Abs Bib HTML

In Assicurazioni Generali, an automatic decision-making model is used to check real-time multivariate time series and alert if a car crash happened. In such a way, a Generali operator can call the customer to provide first assistance. The high sensitivity of the model used, combined with the fact that the model is not interpretable, might cause the operator to call customers even though a car crash did not happen but only due to a harsh deviation or the fact that the road is bumpy. Our goal is to tackle the problem of interpretability for car crash prediction and propose an eXplainable Artificial Intelligence (XAI) workflow that allows gaining insights regarding the logic behind the deep learning predictive model adopted by Generali. We reach our goal by building an interpretable alternative to the current obscure model that also reduces the training data usage and the prediction time.
@inbook{SGN2022, address = {Cham, Switzerland}, author = {Spinnato, Francesco and Guidotti, Riccardo and Nanni, Mirco and Maccagnola, Daniele and Paciello, Giulia and Farina, Antonio Bencini}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_39}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {556–566}, publisher = {Springer Nature Switzerland}, title = {Explaining Crash Predictions on Multivariate Time Series Data}, visible_on_website = {YES}, year = {2022} }
Investigating Debiasing Effects on Classification and Explainability

Marta Marchiori Manerba, and Riccardo Guidotti

In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2022

RESEARCH LINE
1

5

Abs Bib HTML

During each stage of a dataset creation and development process, harmful biases can be accidentally introduced, leading to models that perpetuates marginalization and discrimination of minorities, as the role of the data used during the training is critical. We propose an evaluation framework that investigates the impact on classification and explainability of bias mitigation preprocessing techniques used to assess data imbalances concerning minorities’ representativeness and mitigate the skewed distributions discovered. Our evaluation focuses on assessing fairness, explainability and performance metrics. We analyze the behavior of local model-agnostic explainers on the original and mitigated datasets to examine whether the proxy models learned by the explainability techniques to mimic the black-boxes disproportionately rely on sensitive attributes, demonstrating biases rooted in the explainers. We conduct several experiments about known biased datasets to demonstrate our proposal’s novelty and effectiveness for evaluation and bias detection purposes.
@inproceedings{MG2022, author = {Marchiori Manerba, Marta and Guidotti, Riccardo}, booktitle = {Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’22}, doi = {10.1145/3514094.3534170}, line = {1,5}, month = jul, open_access = {NO}, pages = {468–478}, publisher = {ACM}, series = {AIES ’22}, title = {Investigating Debiasing Effects on Classification and Explainability}, visible_on_website = {YES}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions

Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti

IEEE Access, Nov 2022

RESEARCH LINE
1

Abs Bib HTML

Time series data is increasingly used in a wide range of fields, and it is often relied on in crucial applications and high-stakes decision-making. For instance, sensors generate time series data to recognize different types of anomalies through automatic decision-making systems. Typically, these systems are realized with machine learning models that achieve top-tier performance on time series classification tasks. Unfortunately, the logic behind their prediction is opaque and hard to understand from a human standpoint. Recently, we observed a consistent increase in the development of explanation methods for time series classification justifying the need to structure and review the field. In this work, we (a) present the first extensive literature review on Explainable AI (XAI) for time series classification, (b) categorize the research field through a taxonomy subdividing the methods into time points-based, subsequences-based and instance-based, and (c) identify open research directions regarding the type of explanations and the evaluation of explanations and interpretability.
@article{TSS2022, address = { New York, USA}, author = {Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo}, doi = {10.1109/access.2022.3207765}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {100700–100724}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions}, visible_on_website = {YES}, volume = {10}, year = {2022} }
Transparent Latent Space Counterfactual Explanations for Tabular Data

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{BGG2023b, address = {Shenzhen, China}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)}, doi = {10.1109/dsaa54385.2022.10032407}, line = {1}, month = oct, open_access = {NO}, pages = {1–10}, publisher = {IEEE}, title = {Transparent Latent Space Counterfactual Explanations for Tabular Data}, visible_on_website = {YES}, year = {2022} }
Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

Sep 2022

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing models allowing interaction between decision systems and humans by making the decisions understandable. We propose a case study for skin lesion diagnosis showing how it is possible to provide explanations of the decisions of deep neural network trained to label skin lesions.
@inbook{MGY2021b, address = {Amsterdam, the Netherlands}, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220209}, issn = {1879-8314}, line = {1}, month = sep, open_access = {NO}, pages = {258 - 260}, publisher = {IOS Press}, title = {Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers}, visible_on_website = {YES}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2021

Designing Shapelets for Interpretable Data-Agnostic Classification

Riccardo Guidotti, and Anna Monreale

In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2021

RESEARCH LINE
1

Abs Bib HTML

Time series shapelets are discriminatory subsequences which are representative of a class, and their similarity to a time series can be used for successfully tackling the time series classification problem. The literature shows that Artificial Intelligence (AI) systems adopting classification models based on time series shapelets can be interpretable, more accurate, and significantly fast. Thus, in order to design a data-agnostic and interpretable classification approach, in this paper we first extend the notion of shapelets to different types of data, i.e., images, tabular and textual data. Then, based on this extended notion of shapelets we propose an interpretable data-agnostic classification method. Since the shapelets discovery can be time consuming, especially for data types more complex than time series, we exploit a notion of prototypes for finding candidate shapelets, and reducing both the time required to find a solution and the variance of shapelets. A wide experimentation on datasets of different types shows that the data-agnostic prototype-based shapelets returned by the proposed method empower an interpretable classification which is also fast, accurate, and stable. In addition, we show and we prove that shapelets can be at the basis of explainable AI methods.
@inproceedings{GM2021, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’21}, doi = {10.1145/3461702.3462553}, line = {1}, month = jul, open_access = {NO}, pages = {532–542}, publisher = {ACM}, series = {AIES ’21}, title = {Designing Shapelets for Interpretable Data-Agnostic Classification}, visible_on_website = {YES}, year = {2021} }
GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }
Matrix Profile-Based Interpretable Time Series Classifier

Riccardo Guidotti, and Matteo D’Onofrio

Frontiers in Artificial Intelligence, Oct 2021

RESEARCH LINE
1

Abs Bib HTML

Time series classification (TSC) is a pervasive and transversal problem in various fields ranging from disease diagnosis to anomaly detection in finance. Unfortunately, the most effective models used by Artificial Intelligence (AI) systems for TSC are not interpretable and hide the logic of the decision process, making them unusable in sensitive domains. Recent research is focusing on explanation methods to pair with the obscure classifier to recover this weakness. However, a TSC approach that is transparent by design and is simultaneously efficient and effective is even more preferable. To this aim, we propose an interpretable TSC method based on the patterns, which is possible to extract from the Matrix Profile (MP) of the time series in the training set. A smart design of the classification procedure allows obtaining an efficient and effective transparent classifier modeled as a decision tree that expresses the reasons for the classification as the presence of discriminative subsequences. Quantitative and qualitative experimentation shows that the proposed method overcomes the state-of-the-art interpretable approaches.
@article{GD2021, author = {Guidotti, Riccardo and D’Onofrio, Matteo}, doi = {10.3389/frai.2021.699448}, issn = {2624-8212}, journal = {Frontiers in Artificial Intelligence}, line = {1}, month = oct, open_access = {Gold}, publisher = {Frontiers Media SA}, title = {Matrix Profile-Based Interpretable Time Series Classifier}, visible_on_website = {YES}, volume = {4}, year = {2021} }
Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

In 2021 IEEE Symposium on Computers and Communications (ISCC) , Sep 2021

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing mechanisms allowing for an interaction between decision systems and humans by making the decisions of the formers understandable. This is particularly important in sensitive contexts like in the medical domain. We propose a use case study, for skin lesion diagnosis, illustrating how it is possible to provide the practitioner with explanations on the decisions of a state of the art deep neural network classifier trained to characterize skin lesions from examples. Our framework consists of a trained classifier onto which an explanation module operates. The latter is able to offer the practitioner exemplars and counterexemplars for the classification diagnosis thus allowing the physician to interact with the automatic diagnosis system. The exemplars are generated via an adversarial autoencoder. We illustrate the behavior of the system on representative examples.
@inproceedings{MGY2021, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {2021 IEEE Symposium on Computers and Communications (ISCC)}, doi = {10.1109/iscc53001.2021.9631485}, line = {1}, month = sep, open_access = {NO}, pages = {1–7}, publisher = {IEEE}, title = {Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling}, visible_on_website = {YES}, year = {2021} }
Deriving a Single Interpretable Model by Merging Tree-Based Classifiers

Valerio Bonsignori, Riccardo Guidotti, and Anna Monreale

Sep 2021

RESEARCH LINE
1

2

Abs Bib HTML

Decision tree classifiers have been proved to be among the most interpretable models due to their intuitive structure that illustrates decision processes in form of logical rules. Unfortunately, more complex tree-based classifiers such as oblique trees and random forests overcome the accuracy of decision trees at the cost of becoming non interpretable. In this paper, we propose a method that takes as input any tree-based classifier and returns a single decision tree able to approximate its behavior. Our proposal merges tree-based classifiers by an intensional and extensional approach and applies a post-hoc explanation strategy. Our experiments shows that the retrieved single decision tree is at least as accurate as the original tree-based model, faithful, and more interpretable.
@inbook{BGM2021, author = {Bonsignori, Valerio and Guidotti, Riccardo and Monreale, Anna}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-88942-5_27}, isbn = {9783030889425}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {347–357}, publisher = {Springer International Publishing}, title = {Deriving a Single Interpretable Model by Merging Tree-Based Classifiers}, visible_on_website = {YES}, year = {2021} }
FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems

Marta Marchiori Manerba, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
1

5

Abs Bib HTML

At every stage of a supervised learning process, harmful biases can arise and be inadvertently introduced, ultimately leading to marginalization, discrimination, and abuse towards minorities. This phenomenon becomes particularly impactful in the sensitive real-world context of abusive language detection systems, where non-discrimination is difficult to assess. In addition, given the opaqueness of their internal behavior, the dynamics leading a model to a certain decision are often not clear nor accountable, and significant problems of trust could emerge. A robust value-oriented evaluation of models’ fairness is therefore necessary. In this paper, we present FairShades, a model-agnostic approach for auditing the outcomes of abusive language detection systems. Combining explainability and fairness evaluation, FairShades can identify unintended biases and sensitive categories towards which models are most discriminative. This objective is pursued through the auditing of meaningful counterfactuals generated within CheckList framework. We conduct several experiments on BERT-based models to demonstrate our proposal’s novelty and effectiveness for unmasking biases.
@inproceedings{MG2021, author = {Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00014}, line = {1,5}, month = dec, open_access = {NO}, pages = {34–43}, publisher = {IEEE}, title = {FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems}, visible_on_website = {YES}, year = {2021} }
Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery

Martina Cinquini, Fosca Giannotti, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
2

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, artificial intelligence explanation, etc. In all such contexts, it is important to generate plausible data samples. A common assumption of approaches widely used for data generation is the independence of the features. However, typically, the variables of a dataset de-pend on one another, and these dependencies are not considered in data generation leading to the creation of implausible records. The main problem is that dependencies among variables are typically unknown. In this paper, we design a synthetic dataset generator for tabular data that is able to discover nonlinear causalities among the variables and use them at generation time. State-of-the-art methods for nonlinear causal discovery are typically inefficient. We boost them by restricting the causal discovery among the features appearing in the frequent patterns efficiently retrieved by a pattern mining algorithm. To validate our proposal, we design a framework for generating synthetic datasets with known causalities. Wide experimentation on many synthetic datasets and real datasets with known causalities shows the effectiveness of the proposed method.
@inproceedings{CGG2021, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00016}, line = {2}, month = dec, open_access = {NO}, pages = {54–63}, publisher = {IEEE}, title = {Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery}, visible_on_website = {YES}, year = {2021} }
Evaluating local explanation methods on ground truth

Riccardo Guidotti

Artificial Intelligence, Feb 2021

RESEARCH LINE
1

Abs Bib HTML

Evaluating local explanation methods is a difficult task due to the lack of a shared and universally accepted definition of explanation. In the literature, one of the most common ways to assess the performance of an explanation method is to measure the fidelity of the explanation with respect to the classification of a black box model adopted by an Artificial Intelligent system for making a decision. However, this kind of evaluation only measures the degree of adherence of the local explainer in reproducing the behavior of the black box classifier with respect to the final decision. Therefore, the explanation provided by the local explainer could be different in the content even though it leads to the same decision of the AI system. In this paper, we propose an approach that allows to measure to which extent the explanations returned by local explanation methods are correct with respect to a synthetic ground truth explanation. Indeed, the proposed methodology enables the generation of synthetic transparent classifiers for which the reason for the decision taken, i.e., a synthetic ground truth explanation, is available by design. Experimental results show how the proposed approach allows to easily evaluate local explanations on the ground truth and to characterize the quality of local explanation methods.
@article{G2021, author = {Guidotti, Riccardo}, doi = {10.1016/j.artint.2020.103428}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1}, month = feb, open_access = {NO}, pages = {103428}, publisher = {Elsevier BV}, title = {Evaluating local explanation methods on ground truth}, visible_on_website = {YES}, volume = {291}, year = {2021} }
Ensemble of Counterfactual Explainers

Guidotti Riccardo, and Ruggieri Salvatore

Dec 2021

RESEARCH LINE
1

Abs Bib HTML

In eXplainable Artificial Intelligence (XAI), several counterfactual explainers have been proposed, each focusing on some desirable properties of counterfactual instances: minimality, actionability, stability, diversity, plausibility, discriminative power. We propose an ensemble of counterfactual explainers that boosts weak explainers, which provide only a subset of such properties, to a powerful method covering all of them. The ensemble runs weak explainers on a sample of instances and of features, and it combines their results by exploiting a diversity-driven selection function. The method is model-agnostic and, through a wrapping approach based on autoencoders, it is also data-agnostic
@misc{GR2021, author = {Riccardo, Guidotti and Salvatore, Ruggieri}, line = {1}, month = dec, title = {Ensemble of Counterfactual Explainers}, year = {2021} }

2020

Global Explanations with Local Scoring

Mattia Setzu, Riccardo Guidotti, Anna Monreale, and Franco Turini

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{SGM2019, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-43823-4_14}, isbn = {9783030438234}, issn = {1865-0937}, line = {1}, open_access = {NO}, pages = {159–171}, publisher = {Springer International Publishing}, title = {Global Explanations with Local Scoring}, visible_on_website = {YES}, year = {2020} }
Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Dec 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by “morphing” into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
@inbook{GMM2019, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-46150-8_12}, isbn = {9783030461508}, issn = {1611-3349}, line = {1,4}, pages = {189–205}, publisher = {Springer International Publishing}, title = {Black Box Explanation by Learning Image Exemplars in the Latent Feature Space}, visible_on_website = {YES}, year = {2020} }
Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars

Orestis Lampridis, Riccardo Guidotti, and Salvatore Ruggieri

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of a black box model for sentiment classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. We report experiments on two datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, and usefulness, and that is comparable to it in terms of stability.
@inbook{LGR2020, author = {Lampridis, Orestis and Guidotti, Riccardo and Ruggieri, Salvatore}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-61527-7_24}, isbn = {9783030615277}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {357–373}, publisher = {Springer International Publishing}, title = {Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars}, visible_on_website = {YES}, year = {2020} }
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box image classifiers through synthetic exemplar and counter-exemplar learnt in the latent feature space. Our explanation method exploits the latent representations learned through an adversarial autoencoder for generating a synthetic neighborhood of the image for which an explanation is required. A decision tree is trained on a set of images represented in the latent space, and its decision rules are used to generate exemplar images showing how the original image can be modified to stay within its class. Counterfactual rules are used to generate counter-exemplars showing how the original image can “morph” into another class. The explanation also comprehends a saliency map highlighting the areas that contribute to its classification, and areas that push it into another class. A wide and deep experimental evaluation proves that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability, besides providing the most useful and interpretable explanations.
@article{GMM2020, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, doi = {10.1609/aaai.v34i09.7116}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,4}, month = apr, number = {09}, open_access = {NO}, pages = {13665–13668}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations}, visible_on_website = {YES}, volume = {34}, year = {2020} }
Data-Agnostic Local Neighborhood Generation

Riccardo Guidotti, and Anna Monreale

In 2020 IEEE International Conference on Data Mining (ICDM) , Nov 2020

RESEARCH LINE
1

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, machine learning explanation, etc. In such contexts, it is important to generate data samples located within “local” areas surrounding specific instances. Local synthetic data can help the learning phase of predictive models, and it is fundamental for methods explaining the local behavior of obscure classifiers. The contribution of this paper is twofold. First, we introduce a method based on generative operators allowing the synthetic neighborhood generation by applying specific perturbations on a given input instance. The key factor consists in performing a data transformation that makes applicable to any type of data, i.e., data-agnostic. Second, we design a framework for evaluating the goodness of local synthetic neighborhoods exploiting both supervised and unsupervised methodologies. A deep experimentation shows the effectiveness of the proposed method.
@inproceedings{GM2020, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {2020 IEEE International Conference on Data Mining (ICDM)}, doi = {10.1109/icdm50108.2020.00122}, issn = {2374-8486}, line = {1}, month = nov, open_access = {NO}, pages = {1040–1045}, publisher = {IEEE}, title = {Data-Agnostic Local Neighborhood Generation}, visible_on_website = {YES}, year = {2020} }
Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{GMS2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, open_access = {NO}, pages = {167–176}, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, visible_on_website = {YES}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Helping Your Docker Images to Spread Based on Explainable Models

Riccardo Guidotti, Jacopo Soldani, Davide Neri, Antonio Brogi, and Dino Pedreschi

Dec 2019

RESEARCH LINE
1

Abs Bib HTML

Docker is on the rise in today’s enterprise IT. It permits shipping applications inside portable containers, which run from so-called Docker images. Docker images are distributed in public registries, which also monitor their popularity. The popularity of an image impacts on its actual usage, and hence on the potential revenues for its developers. In this paper, we present a solution based on interpretable decision tree and regression trees for estimating the popularity of a given Docker image, and for understanding how to improve an image to increase its popularity. The results presented in this work can provide valuable insights to Docker developers, helping them in spreading their images. Code related to this paper is available at: https://github.com/di-unipi-socc/DockerImageMiner.
@inbook{GSN2021, author = {Guidotti, Riccardo and Soldani, Jacopo and Neri, Davide and Brogi, Antonio and Pedreschi, Dino}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-10997-4_13}, isbn = {9783030109974}, issn = {1611-3349}, line = {1}, pages = {205–221}, publisher = {Springer International Publishing}, title = {Helping Your Docker Images to Spread Based on Explainable Models}, visible_on_website = {YES}, year = {2019} }
Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }
Explaining Multi-label Black-Box Classifiers for Health Applications

Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi

Aug 2019

RESEARCH LINE
1

4

Abs Bib HTML

Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{PGM2019, author = {Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino}, booktitle = {Precision Health and Medicine}, doi = {10.1007/978-3-030-24409-5_9}, isbn = {9783030244095}, issn = {1860-9503}, line = {1,4}, month = aug, pages = {97–110}, publisher = {Springer International Publishing}, title = {Explaining Multi-label Black-Box Classifiers for Health Applications}, visible_on_website = {YES}, year = {2019} }
The AI black box explanation problem

Guidotti Riccardo, Monreale Anna, and Pedreschi Dino

Dec 2019

RESEARCH LINE
1

2

3

Abs Bib HTML

The use of machine learning in decision-making has triggered an intense debate about “fair algorithms”. Given that fairness intuitions differ and can led to conflicting technical requirements, there is a pressing need to integrate ethical thinking into research and design of machine learning. We outline a framework showing how this can be done.
@misc{GMP2019, author = {Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi}, line = {1,2,3}, month = dec, publisher = {ERCIM – the European Research Consortium for Informatics and Mathematics}, title = {The AI black box explanation problem}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2025

The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR

Laura State, Alejandra Bringas Colmenarejo, Andrea Beretta, Salvatore Ruggieri, Franco Turini, and 1 more author

Artificial Intelligence and Law, Jan 2025

RESEARCH LINE
4

Abs Bib HTML

Explainable AI (XAI) provides methods to understand non-interpretable machine learning models. However, we have little knowledge about what legal experts expect from these explanations, including their legal compliance with, and value against European Union legislation. To close this gap, we present the Explanation Dialogues, an expert focus study to uncover the expectations, reasoning, and understanding of legal experts and practitioners towards XAI, with a specific focus on the European General Data Protection Regulation. The study consists of an online questionnaire and follow-up interviews, and is centered around a use-case in the credit domain. We extract both a set of hierarchical and interconnected codes using grounded theory, and present the standpoints of the participating experts towards XAI. We find that the presented explanations are hard to understand and lack information, and discuss issues that can arise from the different interests of the data controller and subject. Finally, we present a set of recommendations for developers of XAI methods, and indications of legal areas of discussion. Among others, recommendations address the presentation, choice, and content of an explanation, technical risks as well as the end-user, while we provide legal pointers to the contestability of explanations, transparency thresholds, intellectual property rights as well as the relationship between involved parties.
@article{SBB2025, author = {State, Laura and Bringas Colmenarejo, Alejandra and Beretta, Andrea and Ruggieri, Salvatore and Turini, Franco and Law, Stephanie}, doi = {10.1007/s10506-024-09430-w}, issn = {1572-8382}, journal = {Artificial Intelligence and Law}, line = {4}, month = jan, open_access = {Green}, publisher = {Springer Science and Business Media LLC}, title = {The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR}, visible_on_website = {YES}, year = {2025} }

2023

Reason to Explain: Interactive Contrastive Explanations (REASONX)

Laura State, Salvatore Ruggieri, and Franco Turini

Jan 2023

RESEARCH LINE
1

3

Abs Bib HTML

Many high-performing machine learning models are not interpretable. As they are increasingly used in decision scenarios that can critically affect individuals, it is necessary to develop tools to better understand their outputs. Popular explanation methods include contrastive explanations. However, they suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of interactivity. While (dialogue-like) interactivity is important to better communicate an explanation, background knowledge has the potential to significantly improve their quality, e.g., by adapting the explanation to the needs of the end-user. To close this gap, we present REASONX, an explanation tool based on Constraint Logic Programming (CLP). REASONX provides interactive contrastive explanations that can be augmented by background knowledge, and allows to operate under a setting of under-specified information, leading to increased flexibility in the provided explanations. REASONX computes factual and contrastive decision rules, as well as closest contrastive examples. It provides explanations for decision trees, which can be the ML models under analysis, or global/local surrogate models of any ML model. While the core part of REASONX is built on CLP, we also provide a program layer that allows to compute the explanations via Python, making the tool accessible to a wider audience. We illustrate the capability of REASONX on a synthetic data set, and on a well-developed example in the credit domain. In both cases, we can show how REASONX can be flexibly used and tailored to the needs of the user.
@inbook{SRT2023, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_22}, isbn = {9783031440649}, issn = {1865-0937}, line = {1,3}, open_access = {NO}, pages = {421–437}, publisher = {Springer Nature Switzerland}, title = {Reason to Explain: Interactive Contrastive Explanations (REASONX)}, visible_on_website = {YES}, year = {2023} }
Declarative Reasoning on Explanations Using Constraint Logic Programming

Laura State, Salvatore Ruggieri, and Franco Turini

Jan 2023

RESEARCH LINE
2

Abs Bib HTML

Explaining opaque Machine Learning models is an increasingly relevant problem. Current explanation in AI methods suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of abstraction and interactivity with the user. We propose REASONX, an explanation method based on Constraint Logic Programming. REASONX can provide declarative, interactive explanations for decision trees, which can be the machine learning models under analysis or global or local surrogate models of any black-box model. Users can express background or common sense knowledge using linear constraints and MILP optimization over features of factual and contrastive instances, and interact with the answer constraints at different levels of abstraction through constraint projection. We present here the architecture of REASONX, which consists of a Python layer, closer to the user, and a CLP layer.
@inbook{SRT2023b, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, booktitle = {Logics in Artificial Intelligence}, doi = {10.1007/978-3-031-43619-2_10}, isbn = {9783031436192}, issn = {1611-3349}, line = {2}, open_access = {NO}, pages = {132–141}, publisher = {Springer Nature Switzerland}, title = {Declarative Reasoning on Explanations Using Constraint Logic Programming}, visible_on_website = {YES}, year = {2023} }

2022

Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }

2021

GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }

2020

Global Explanations with Local Scoring

Mattia Setzu, Riccardo Guidotti, Anna Monreale, and Franco Turini

May 2020

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{SGM2019, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-43823-4_14}, isbn = {9783030438234}, issn = {1865-0937}, line = {1}, open_access = {NO}, pages = {159–171}, publisher = {Springer International Publishing}, title = {Global Explanations with Local Scoring}, visible_on_website = {YES}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2025

A Practical Approach to Causal Inference over Time

Martina Cinquini, Isacco Beretta, Salvatore Ruggieri, and Isabel Valera

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

RESEARCH LINE
2

Abs Bib HTML

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting causal VAR framework allows us to perform causal inference over time from observational time series data. Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.
@article{CBR2025, author = {Cinquini, Martina and Beretta, Isacco and Ruggieri, Salvatore and Valera, Isabel}, doi = {10.1609/aaai.v39i14.33626}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {2}, month = apr, number = {14}, open_access = {Gold}, pages = {14832–14839}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {A Practical Approach to Causal Inference over Time}, visible_on_website = {YES}, volume = {39}, year = {2025} }
Counterfactual Situation Testing: From Single to Multidimensional Discrimination

Jose M. Alvarez, and Salvatore Ruggieri

Journal of Artificial Intelligence Research, Apr 2025

RESEARCH LINE
1

Abs Bib HTML

As machine learning models enable decisions once performed only by humans, it is central to develop tools that assess the fairness of such models. Notably, within high-stake settings like hiring and lending, these tools must be able to detect potentially discriminatory models. We present counterfactual situation testing (CST), a causal data mining framework for detecting individual discrimination in a dataset of classifier decisions. CST answers the question “what would have been the model outcome had the individual, or complainant, been of a different protected status?” It extends the legally-grounded situation testing (ST) of Thanh et al. (2011) by operationalizing the notion of fairness given the difference via counterfactual reasoning. ST finds for each complainant similar protected and non-protected instances in the dataset; constructs, respectively, a control and test group; and compares the groups such that a difference in model outcomes implies a potential case of individual discrimination. CST, instead, avoids this idealized comparison by establishing the test group on the complainant’s generated counterfactual, which reflects how the protected attribute when changed influences other seemingly neutral attributes of the complainant. Under CST we test for discrimination for each complainant by comparing similar individuals within the control and test group but dissimilar individuals across these groups. We consider single (e.g., gender) and multidimensional (e.g., gender and race) discrimination testing. For multidimensional discrimination we study multiple and intersectional discrimination and, as feared by legal scholars, find evidence that the former fails to account for the latter kind. Using a k-nearest neighbor implementation, we showcase CST on synthetic and real data. Experimental results show that CST uncovers a higher number of cases than ST, even when the model is counterfactually fair. CST, in fact, extends counterfactual fairness (CF) of Kusner et al. (2017) by equipping CF with confidence intervals, which we report for all experiments.
@article{AR2025, author = {Alvarez, Jose M. and Ruggieri, Salvatore}, doi = {10.1613/jair.1.17935}, issn = {1076-9757}, journal = {Journal of Artificial Intelligence Research}, line = {1}, month = apr, open_access = {Gold}, pages = {2279–2323}, publisher = {AI Access Foundation}, title = {Counterfactual Situation Testing: From Single to Multidimensional Discrimination}, visible_on_website = {YES}, volume = {82}, year = {2025} }
The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR

Laura State, Alejandra Bringas Colmenarejo, Andrea Beretta, Salvatore Ruggieri, Franco Turini, and 1 more author

Artificial Intelligence and Law, Jan 2025

RESEARCH LINE
4

Abs Bib HTML

Explainable AI (XAI) provides methods to understand non-interpretable machine learning models. However, we have little knowledge about what legal experts expect from these explanations, including their legal compliance with, and value against European Union legislation. To close this gap, we present the Explanation Dialogues, an expert focus study to uncover the expectations, reasoning, and understanding of legal experts and practitioners towards XAI, with a specific focus on the European General Data Protection Regulation. The study consists of an online questionnaire and follow-up interviews, and is centered around a use-case in the credit domain. We extract both a set of hierarchical and interconnected codes using grounded theory, and present the standpoints of the participating experts towards XAI. We find that the presented explanations are hard to understand and lack information, and discuss issues that can arise from the different interests of the data controller and subject. Finally, we present a set of recommendations for developers of XAI methods, and indications of legal areas of discussion. Among others, recommendations address the presentation, choice, and content of an explanation, technical risks as well as the end-user, while we provide legal pointers to the contestability of explanations, transparency thresholds, intellectual property rights as well as the relationship between involved parties.
@article{SBB2025, author = {State, Laura and Bringas Colmenarejo, Alejandra and Beretta, Andrea and Ruggieri, Salvatore and Turini, Franco and Law, Stephanie}, doi = {10.1007/s10506-024-09430-w}, issn = {1572-8382}, journal = {Artificial Intelligence and Law}, line = {4}, month = jan, open_access = {Green}, publisher = {Springer Science and Business Media LLC}, title = {The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR}, visible_on_website = {YES}, year = {2025} }

2024

Requirements of eXplainable AI in Algorithmic Hiring

A. Beretta, G. Ercoli, A. Ferraro, R. Guidotti, A. Iommi, and 4 more authors

Dec 2024

RESEARCH LINE
1

Abs Bib HTML

AI models for ranking candidates to a job position are increasingly adopted. They bring a new layer of opaqueness in the way candidates are evaluated. We present preliminary research on stakeholder analysis and requirement elicitation for designing an explainability component in AI models for ranking candidates to a job position. (CEUR-WS)
@misc{BEF2024, author = {Beretta, A. and Ercoli, G. and Ferraro, A. and Guidotti, R. and Iommi, A. and Mastropietro, A. and Monreale, A. and Rotelli, D. and Ruggieri, S.}, line = {1}, month = dec, title = {Requirements of eXplainable AI in Algorithmic Hiring}, year = {2024} }

2023

Reason to Explain: Interactive Contrastive Explanations (REASONX)

Laura State, Salvatore Ruggieri, and Franco Turini

Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

Many high-performing machine learning models are not interpretable. As they are increasingly used in decision scenarios that can critically affect individuals, it is necessary to develop tools to better understand their outputs. Popular explanation methods include contrastive explanations. However, they suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of interactivity. While (dialogue-like) interactivity is important to better communicate an explanation, background knowledge has the potential to significantly improve their quality, e.g., by adapting the explanation to the needs of the end-user. To close this gap, we present REASONX, an explanation tool based on Constraint Logic Programming (CLP). REASONX provides interactive contrastive explanations that can be augmented by background knowledge, and allows to operate under a setting of under-specified information, leading to increased flexibility in the provided explanations. REASONX computes factual and contrastive decision rules, as well as closest contrastive examples. It provides explanations for decision trees, which can be the ML models under analysis, or global/local surrogate models of any ML model. While the core part of REASONX is built on CLP, we also provide a program layer that allows to compute the explanations via Python, making the tool accessible to a wider audience. We illustrate the capability of REASONX on a synthetic data set, and on a well-developed example in the credit domain. In both cases, we can show how REASONX can be flexibly used and tailored to the needs of the user.
@inbook{SRT2023, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_22}, isbn = {9783031440649}, issn = {1865-0937}, line = {1,3}, open_access = {NO}, pages = {421–437}, publisher = {Springer Nature Switzerland}, title = {Reason to Explain: Interactive Contrastive Explanations (REASONX)}, visible_on_website = {YES}, year = {2023} }
Declarative Reasoning on Explanations Using Constraint Logic Programming

Laura State, Salvatore Ruggieri, and Franco Turini

Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Explaining opaque Machine Learning models is an increasingly relevant problem. Current explanation in AI methods suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of abstraction and interactivity with the user. We propose REASONX, an explanation method based on Constraint Logic Programming. REASONX can provide declarative, interactive explanations for decision trees, which can be the machine learning models under analysis or global or local surrogate models of any black-box model. Users can express background or common sense knowledge using linear constraints and MILP optimization over features of factual and contrastive instances, and interact with the answer constraints at different levels of abstraction through constraint projection. We present here the architecture of REASONX, which consists of a Python layer, closer to the user, and a CLP layer.
@inbook{SRT2023b, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, booktitle = {Logics in Artificial Intelligence}, doi = {10.1007/978-3-031-43619-2_10}, isbn = {9783031436192}, issn = {1611-3349}, line = {2}, open_access = {NO}, pages = {132–141}, publisher = {Springer Nature Switzerland}, title = {Declarative Reasoning on Explanations Using Constraint Logic Programming}, visible_on_website = {YES}, year = {2023} }
AUC-based Selective Classification

Pugnana Andrea, and Ruggieri Salvatore

Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
@misc{PR2023, author = {Andrea, Pugnana and Salvatore, Ruggieri}, line = {2}, month = dec, pages = {2494--2514}, title = {AUC-based Selective Classification}, year = {2023} }

2022

Methods and tools for causal discovery and causal inference

Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama

WIREs Data Mining and Knowledge Discovery, Jan 2022

RESEARCH LINE
2

Abs Bib HTML

Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{NPR2022, address = {Danvers, MS, USA}, author = {Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João}, doi = {10.1002/widm.1449}, issn = {1942-4795}, journal = {WIREs Data Mining and Knowledge Discovery}, line = {2}, month = jan, number = {2}, open_access = {Gold}, publisher = {Wiley}, title = {Methods and tools for causal discovery and causal inference}, visible_on_website = {YES}, volume = {12}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }

2021

Ensemble of Counterfactual Explainers

Guidotti Riccardo, and Ruggieri Salvatore

Dec 2021

RESEARCH LINE
1

Abs Bib HTML

In eXplainable Artificial Intelligence (XAI), several counterfactual explainers have been proposed, each focusing on some desirable properties of counterfactual instances: minimality, actionability, stability, diversity, plausibility, discriminative power. We propose an ensemble of counterfactual explainers that boosts weak explainers, which provide only a subset of such properties, to a powerful method covering all of them. The ensemble runs weak explainers on a sample of instances and of features, and it combines their results by exploiting a diversity-driven selection function. The method is model-agnostic and, through a wrapping approach based on autoencoders, it is also data-agnostic
@misc{GR2021, author = {Riccardo, Guidotti and Salvatore, Ruggieri}, line = {1}, month = dec, title = {Ensemble of Counterfactual Explainers}, year = {2021} }

2020

Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars

Orestis Lampridis, Riccardo Guidotti, and Salvatore Ruggieri

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of a black box model for sentiment classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. We report experiments on two datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, and usefulness, and that is comparable to it in terms of stability.
@inbook{LGR2020, author = {Lampridis, Orestis and Guidotti, Riccardo and Ruggieri, Salvatore}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-61527-7_24}, isbn = {9783030615277}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {357–373}, publisher = {Springer International Publishing}, title = {Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars}, visible_on_website = {YES}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2024

Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

IEEE Access, Dec 2024

RESEARCH LINE
1

Abs Bib HTML

The current trend in time series classification is to develop highly accurate algorithms by combining multiple models in ensemble hybrids, representing time series in complex feature spaces, and extracting features from different representations. As a consequence, the best time series classifiers are black-box models, not understandable for humans. Even the approaches regarded as interpretable, such as shapelet-based ones, rely on randomization to maintain efficiency, which poses challenges for interpretability. We propose the Bag-Of-Receptive-Fields (BORF), a fast, interpretable, and deterministic time series transform. Building on the Bag-Of-Patterns, we bridge the gap between convolutional operators and discretization, enhancing the Symbolic Aggregate Approximation with dilation and stride to better capture temporal patterns at multiple scales. We propose an algorithmic speedup that reduces the time complexity associated with SAX-based classifiers, enabling the more flexible BORF, represented as a sparse multivariate tensor.
@article{SGM2024, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, doi = {10.1109/access.2024.3464743}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {137893–137912}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Mapping the landscape of ethical considerations in explainable AI research

Luca Nannini, Marta Marchiori Manerba, and Isacco Beretta

Ethics and Information Technology, Jun 2024

RESEARCH LINE
1

5

Abs Bib HTML

With its potential to contribute to the ethical governance of AI, explainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet the substantiation of these claims with rigorous ethical analysis and reflection remains largely unexamined. This contribution endeavours to scrutinise the relationship between XAI and ethical considerations. We conduct a systematic review of the XAI literature by applying a multi-stage filtering process and then developing a taxonomy for classifying the depth and quality of ethical engagement in the field. Our findings show that although a growing body of XAI research references ethical issues, the majority does so at a superficial level. We identify nine ethical themes (e.g., autonomy, justice, transparency) and show how they are engaged in the literature. We discuss trends, deficits, and future directions for integrating ethical considerations meaningfully into XAI research.
@article{NMB2024, author = {Nannini, Luca and Marchiori Manerba, Marta and Beretta, Isacco}, doi = {10.1007/s10676-024-09773-7}, issn = {1572-8439}, journal = {Ethics and Information Technology}, line = {1,5}, month = jun, number = {3}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Mapping the landscape of ethical considerations in explainable AI research}, visible_on_website = {YES}, volume = {26}, year = {2024} }
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }

2023

Effects of Route Randomization on Urban Emissions

Giuliano Cornacchia, Mirco Nanni, Dino Pedreschi, and Luca Pappalardo

SUMO Conference Proceedings, Jun 2023

RESEARCH LINE
4

5

Abs Bib HTML

Routing algorithms typically suggest the fastest path or slight variation to reach a user’s desired destination. Although this suggestion at the individual level is undoubtedly advantageous for the user, from a collective point of view, the aggregation of all single suggested paths may result in an increasing impact (e.g., in terms of emissions). In this study, we use SUMO to simulate the effects of incorporating randomness into routing algorithms on emissions, their distribution, and travel time in the urban area of Milan (Italy). Our results reveal that, given the common practice of routing towards the fastest path, a certain level of randomness in routes reduces emissions and travel time. In other words, the stronger the random component in the routes, the more pronounced the benefits upon a certain threshold. Our research provides insight into the potential advantages of considering collective outcomes in routing decisions and highlights the need to explore further the relationship between route randomization and sustainability in urban transportation.
@article{CNP2023, author = {Cornacchia, Giuliano and Nanni, Mirco and Pedreschi, Dino and Pappalardo, Luca}, doi = {10.52825/scp.v4i.217}, issn = {2750-4425}, journal = {SUMO Conference Proceedings}, line = {4,5}, month = jun, open_access = {Gold}, pages = {75–87}, publisher = {TIB Open Publishing}, title = {Effects of Route Randomization on Urban Emissions}, visible_on_website = {YES}, volume = {4}, year = {2023} }
Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{LSG2023, address = {Cham, Switzerland}, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Advances in Intelligent Data Analysis XXI}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, visible_on_website = {YES}, year = {2023} }
Modeling Events and Interactions through Temporal Processes – A Survey

Liguori Angelica, Caroprese Luciano, Minici Marco, Veloso Bruno, Spinnato Francesco, and 3 more authors

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

In real-world scenario, many phenomena produce a collection of events that occur in continuous time. Point Processes provide a natural mathematical framework for modeling these sequences of events. In this survey, we investigate probabilistic models for modeling event sequences through temporal processes. We revise the notion of event modeling and provide the mathematical foundations that characterize the literature on the topic. We define an ontology to categorize the existing approaches in terms of three families: simple, marked, and spatio-temporal point processes. For each family, we systematically review the existing approaches based based on deep learning. Finally, we analyze the scenarios where the proposed techniques can be used for addressing prediction and modeling aspects.
@misc{LCM2023, author = {Angelica, Liguori and Luciano, Caroprese and Marco, Minici and Bruno, Veloso and Francesco, Spinnato and Mirco, Nanni and Giuseppe, Manco and Joao, Gama}, doi = {10.48550/ARXIV.2303.06067}, line = {1}, month = dec, publisher = {Arxiv}, title = {Modeling Events and Interactions through Temporal Processes -- A Survey}, year = {2023} }

2022

Explaining Crash Predictions on Multivariate Time Series Data

Francesco Spinnato, Riccardo Guidotti, Mirco Nanni, Daniele Maccagnola, Giulia Paciello, and 1 more author

Dec 2022

RESEARCH LINE
4

Abs Bib HTML

In Assicurazioni Generali, an automatic decision-making model is used to check real-time multivariate time series and alert if a car crash happened. In such a way, a Generali operator can call the customer to provide first assistance. The high sensitivity of the model used, combined with the fact that the model is not interpretable, might cause the operator to call customers even though a car crash did not happen but only due to a harsh deviation or the fact that the road is bumpy. Our goal is to tackle the problem of interpretability for car crash prediction and propose an eXplainable Artificial Intelligence (XAI) workflow that allows gaining insights regarding the logic behind the deep learning predictive model adopted by Generali. We reach our goal by building an interpretable alternative to the current obscure model that also reduces the training data usage and the prediction time.
@inbook{SGN2022, address = {Cham, Switzerland}, author = {Spinnato, Francesco and Guidotti, Riccardo and Nanni, Mirco and Maccagnola, Daniele and Paciello, Giulia and Farina, Antonio Bencini}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_39}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {556–566}, publisher = {Springer Nature Switzerland}, title = {Explaining Crash Predictions on Multivariate Time Series Data}, visible_on_website = {YES}, year = {2022} }

2025

Human-AI coevolution

Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-László Barabási, and 12 more authors

Artificial Intelligence, Feb 2025

RESEARCH LINE
1

Abs Bib HTML

Human-AI coevolution, defined as a process in which humans and AI algorithms continuously influence each other, increasingly characterises our society, but is understudied in artificial intelligence and complexity science literature. Recommender systems and assistants play a prominent role in human-AI coevolution, as they permeate many facets of daily life and influence human choices through online platforms. The interaction between users and AI results in a potentially endless feedback loop, wherein users’ choices generate data to train AI models, which, in turn, shape subsequent user preferences. This human-AI feedback loop has peculiar characteristics compared to traditional human-machine interaction and gives rise to complex and often “unintended” systemic outcomes. This paper introduces human-AI coevolution as the cornerstone for a new field of study at the intersection between AI and complexity science focused on the theoretical, empirical, and mathematical investigation of the human-AI feedback loop. In doing so, we: (i) outline the pros and cons of existing methodologies and highlight shortcomings and potential ways for capturing feedback loop mechanisms; (ii) propose a reflection at the intersection between complexity science, AI and society; (iii) provide real-world examples for different human-AI ecosystems; and (iv) illustrate challenges to the creation of such a field of study, conceptualising them at increasing levels of abstraction, i.e., scientific, legal and socio-political.
@article{PPF2025, address = {Netherlands}, author = {Pedreschi, Dino and Pappalardo, Luca and Ferragina, Emanuele and Baeza-Yates, Ricardo and Barabási, Albert-László and Dignum, Frank and Dignum, Virginia and Eliassi-Rad, Tina and Giannotti, Fosca and Kertész, János and Knott, Alistair and Ioannidis, Yannis and Lukowicz, Paul and Passarella, Andrea and Pentland, Alex Sandy and Shawe-Taylor, John and Vespignani, Alessandro}, doi = {10.1016/j.artint.2024.104244}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1}, month = feb, open_access = {Gold}, pages = {104244}, publisher = {Elsevier BV}, title = {Human-AI coevolution}, visible_on_website = {YES}, volume = {339}, year = {2025} }
Human-AI coevolution

Dino Pedreschi, Luca Pappalardo, Emanuele Ferragina, Ricardo Baeza-Yates, Albert-László Barabási, and 12 more authors

Artificial Intelligence, Feb 2025

RESEARCH LINE
2

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of black box models in classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. Moreover, diversity of counter-exemplars is modeled as an optimization problem, solved by a greedy algorithm with theoretical guarantee. We report experiments on three datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, diversity, and usefulness, and that is comparable to it in terms of stability.
@article{Pedreschi_2025, address = {Netherlands}, author = {Pedreschi, Dino and Pappalardo, Luca and Ferragina, Emanuele and Baeza-Yates, Ricardo and Barabási, Albert-László and Dignum, Frank and Dignum, Virginia and Eliassi-Rad, Tina and Giannotti, Fosca and Kertész, János and Knott, Alistair and Ioannidis, Yannis and Lukowicz, Paul and Passarella, Andrea and Pentland, Alex Sandy and Shawe-Taylor, John and Vespignani, Alessandro}, doi = {10.1016/j.artint.2024.104244}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {2}, month = feb, open_access = {Gold}, pages = {104244}, publisher = {Elsevier BV}, title = {Human-AI coevolution}, visible_on_website = {YES}, volume = {339}, year = {2025} }
A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems

G. Barlacchi, M. Lalli, E. Ferragina, F. Giannotti, and L. Pappalardo

Dec 2025

RESEARCH LINE
4

Abs Bib HTML

Recommender systems continuously interact with users, creating feedback loops that shape both individual behavior and collective market dynamics. This paper introduces a simulation framework to model these loops in online retail environments, where recommenders are periodically retrained on evolving user–item interactions. Using the Amazon e-Commerce dataset, we analyze how different recommendation algorithms influence diversity, purchase concentration, and user homogenization over time. Results reveal a systematic trade-off: while the feedback loop increases individual diversity, it simultaneously reduces collective diversity and concentrates demand on a few popular items. Moreover, for some recommender systems, the feedback loop increases user homogenization over time, making user purchase profiles increasingly similar. These findings underscore the need for recommender designs that balance personalization with long-term diversity.
@misc{BLF2025, author = {Barlacchi, G. and Lalli, M. and Ferragina, E. and Giannotti, F. and Pappalardo, L.}, doi = {10.48550/arXiv.2510.14857}, line = {4}, month = dec, title = {A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems}, year = {2025} }
"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy

Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, and 1 more author

Dec 2025

RESEARCH LINE
1

Bib
@misc{GGG2025, author = {Gambetta, Daniele and Gezici, Gizem and Giannotti, Fosca and Pedreschi, Dino and Knott, Alistair and Pappalardo, Luca}, line = {1}, month = dec, title = {"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy}, year = {2025} }

2024

A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }

2023

Effects of Route Randomization on Urban Emissions

Giuliano Cornacchia, Mirco Nanni, Dino Pedreschi, and Luca Pappalardo

SUMO Conference Proceedings, Jun 2023

RESEARCH LINE
4

5

Abs Bib HTML

Routing algorithms typically suggest the fastest path or slight variation to reach a user’s desired destination. Although this suggestion at the individual level is undoubtedly advantageous for the user, from a collective point of view, the aggregation of all single suggested paths may result in an increasing impact (e.g., in terms of emissions). In this study, we use SUMO to simulate the effects of incorporating randomness into routing algorithms on emissions, their distribution, and travel time in the urban area of Milan (Italy). Our results reveal that, given the common practice of routing towards the fastest path, a certain level of randomness in routes reduces emissions and travel time. In other words, the stronger the random component in the routes, the more pronounced the benefits upon a certain threshold. Our research provides insight into the potential advantages of considering collective outcomes in routing decisions and highlights the need to explore further the relationship between route randomization and sustainability in urban transportation.
@article{CNP2023, author = {Cornacchia, Giuliano and Nanni, Mirco and Pedreschi, Dino and Pappalardo, Luca}, doi = {10.52825/scp.v4i.217}, issn = {2750-4425}, journal = {SUMO Conference Proceedings}, line = {4,5}, month = jun, open_access = {Gold}, pages = {75–87}, publisher = {TIB Open Publishing}, title = {Effects of Route Randomization on Urban Emissions}, visible_on_website = {YES}, volume = {4}, year = {2023} }

2022

Understanding peace through the world news

Vasiliki Voukelatou, Ioanna Miliou, Fosca Giannotti, and Luca Pappalardo

EPJ Data Science, Jan 2022

RESEARCH LINE
4

Abs Bib HTML

Peace is a principal dimension of well-being and is the way out of inequity and violence. Thus, its measurement has drawn the attention of researchers, policymakers, and peacekeepers. During the last years, novel digital data streams have drastically changed the research in this field. The current study exploits information extracted from a new digital database called Global Data on Events, Location, and Tone (GDELT) to capture peace through the Global Peace Index (GPI). Applying predictive machine learning models, we demonstrate that news media attention from GDELT can be used as a proxy for measuring GPI at a monthly level. Additionally, we use explainable AI techniques to obtain the most important variables that drive the predictions. This analysis highlights each country’s profile and provides explanations for the predictions, and particularly for the errors and the events that drive these errors. We believe that digital data exploited by researchers, policymakers, and peacekeepers, with data science tools as powerful as machine learning, could contribute to maximizing the societal benefits and minimizing the risks to peace.
@article{VMG2022, author = {Voukelatou, Vasiliki and Miliou, Ioanna and Giannotti, Fosca and Pappalardo, Luca}, doi = {10.1140/epjds/s13688-022-00315-z}, issn = {2193-1127}, journal = {EPJ Data Science}, line = {4}, month = jan, number = {1}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Understanding peace through the world news}, visible_on_website = {YES}, volume = {11}, year = {2022} }

2018

Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }

2025

Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model

Fabio Michele Russo, Carlo Metta, Anna Monreale, Salvatore Rinzivillo, and Fabio Pinelli

Dec 2025

RESEARCH LINE
1

Abs Bib HTML

As predictive machine learning models become increasingly adopted and advanced, their role has evolved from merely predicting outcomes to actively shaping them. This evolution has underscored the importance of Trustworthy AI, highlighting the necessity to extend our focus beyond mere accuracy and toward a comprehensive understanding of these models’ behaviors within the specific contexts of their applications. To further progress in explainability, we introduce poem, Prefetched Offline Explanation Model, a model-agnostic, local explainability algorithm for image data. The algorithm generates exemplars, counterexemplars and saliency maps to provide quick and effective explanations suitable for time-sensitive scenarios. Leveraging an existing local algorithm, poem infers factual and counterfactual rules from data to create illustrative examples and opposite scenarios with an enhanced stability by design. A novel mechanism then matches incoming test points with an explanation base and produces diverse exemplars, informative saliency maps and believable counterexemplars. Experimental results indicate that poem outperforms its predecessor abele in speed and ability to generate more nuanced and varied exemplars alongside more insightful saliency maps and valuable counterexemplars.
@inbook{RMM2025, author = {Russo, Fabio Michele and Metta, Carlo and Monreale, Anna and Rinzivillo, Salvatore and Pinelli, Fabio}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_11}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {167–182}, publisher = {Springer Nature Switzerland}, title = {Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model}, visible_on_website = {YES}, year = {2025} }

2024

One-Shot Clustering for Federated Learning

Maciej Krzysztof Zuziak, Roberto Pellungrini, and Salvatore Rinzivillo

In 2024 IEEE International Conference on Big Data (BigData) , Dec 2024

RESEARCH LINE
1

Abs Bib HTML

Federated Learning (FL) is a widespread and well-adopted paradigm of decentralized learning that allows training one model from multiple sources without the need to directly transfer data between participating clients. Since its inception in 2015, it has been divided into numerous sub-fields that deal with application-specific issues, be it data heterogeneity or resource allocation. One such sub-field, Clustered Federated Learning (CFL), is dealing with the problem of clustering the population of clients into separate cohorts to deliver personalized models. Although few remarkable works have been published in this domain, the problem is still largely unexplored, as its basic assumption and settings are slightly different from standard FL. In this work, we present One-Shot Clustered Federated Learning (OCFL), a clustering-agnostic algorithm that can automatically detect the earliest suitable moment for clustering. Our algorithm is based on the computation of cosine similarity between gradients of the clients and a temperature measure that detects when the federated model starts to converge. We empirically evaluate our methodology by testing various one-shot clustering algorithms for over thirty different tasks on three benchmark datasets. Our experiments showcase the good performance of our approach when used to perform CFL in an automated manner without the need to adjust hyperparameters.
@inproceedings{ZPR2024, address = {Washington, DC, USA}, author = {Zuziak, Maciej Krzysztof and Pellungrini, Roberto and Rinzivillo, Salvatore}, booktitle = {2024 IEEE International Conference on Big Data (BigData)}, doi = {10.1109/bigdata62323.2024.10825763}, line = {1}, month = dec, open_access = {NO}, pages = {8108–8117}, publisher = {IEEE}, title = {One-Shot Clustering for Federated Learning}, visible_on_website = {YES}, year = {2024} }
An Interactive Interface for Feature Space Navigation

Eleonora Cappuccio, Isacco Beretta, Marta Marchiori Manerba, and Salvatore Rinzivillo

Jun 2024

RESEARCH LINE
3

Abs Bib HTML

In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to provide users with an intuitive and direct way to navigate through the feature space, inspect model behavior, and perform what-if analyses via feature manipulations and visual feedback. We integrate multiple views including projections of high-dimensional data, decision boundary surfaces, and sensitivity indicators. The interface also supports real-time adjustments of feature values to observe the corresponding changes in the model predictions. Our experiments show that the system can help both novice and expert users to detect regions of uncertainty, identify influential features, and generate hypotheses for model improvement.
@inbook{CBM2024, author = {Cappuccio, Eleonora and Beretta, Isacco and Marchiori Manerba, Marta and Rinzivillo, Salvatore}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240184}, isbn = {9781643685229}, issn = {1879-8314}, line = {3}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {An Interactive Interface for Feature Space Navigation}, visible_on_website = {YES}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Exploring Large Language Models Capabilities to Explain Decision Trees

Paulo Bruno Serafim, Pierluigi Crescenzi, Gizem Gezici, Eleonora Cappuccio, Salvatore Rinzivillo, and 1 more author

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are widely adopted in Machine Learning tasks due to their operation simplicity and interpretability aspects. However, following the decision process path taken by trees can be difficult in a complex scenario or in a case where a user has no familiarity with them. Prior research showed that converting outcomes to natural language is an accessible way to facilitate understanding for non-expert users in several tasks. More recently, there has been a growing effort to use Large Language Models (LLMs) as a tool for providing natural language texts. In this paper, we examine the proficiency of LLMs to explain decision tree predictions in simple terms through the generation of natural language explanations. By exploring different textual representations and prompt engineering strategies, we identify capabilities that strengthen LLMs as a competent explainer as well as highlight potential challenges and limitations, opening further research possibilities on natural language explanations for decision trees.
@inbook{SGC2024, author = {Serafim, Paulo Bruno and Crescenzi, Pierluigi and Gezici, Gizem and Cappuccio, Eleonora and Rinzivillo, Salvatore and Giannotti, Fosca}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240183}, isbn = {9781643685229}, issn = {1879-8314}, line = {1}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {Exploring Large Language Models Capabilities to Explain Decision Trees}, visible_on_website = {YES}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }
Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories

Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda

Jun 2023

RESEARCH LINE
1

3

5

Abs Bib HTML

Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, identifying privacy risks is essential before deciding to release it publicly. Recent work has proposed using machine learning models for predicting privacy risk on raw mobility trajectories and using SHAP for risk explanation. However, applying SHAP to mobility data results in explanations of limited use both for privacy experts and end-users. In this work, we present EXPHLOT, a novel version of the Expert privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification to improve risk prediction while reducing computation time. We also devise an entropy-based mask to efficiently compute SHAP values and develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of privacy risk.
@inbook{NPR2023, author = {Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_22}, isbn = {9783031452758}, issn = {1611-3349}, line = {1,3,5}, open_access = {Gold}, pages = {325–340}, publisher = {Springer Nature Switzerland}, title = {EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories}, visible_on_website = {YES}, year = {2023} }
Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations

Eleonora Cappuccio, Daniele Fadda, Rosa Lanzilotti, and Salvatore Rinzivillo

In Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter , Sep 2023

RESEARCH LINE
1

2

3

Abs Bib HTML

The Human-Computer Interaction (HCI) community has long stressed the need for a more user-centered approach to Explainable Artificial Intelligence (XAI), a research area that aims at defining algorithms and tools to illustrate the predictions of the so-called black-box models. This approach can benefit from the fields of user-interface, user experience, and visual analytics. In this demo, we propose a visual-based tool, "F.I.P.E.R.", that shows interactive explanations combining rules and feature importance.
@inproceedings{CFR2023, author = {Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore}, booktitle = {Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter}, collection = {CHItaly 2023}, doi = {10.1145/3605390.3610811}, line = {1,2,3}, month = sep, open_access = {NO}, pages = {1–4}, publisher = {ACM}, series = {CHItaly 2023}, title = {Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations}, visible_on_website = {YES}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

Sep 2022

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing models allowing interaction between decision systems and humans by making the decisions understandable. We propose a case study for skin lesion diagnosis showing how it is possible to provide explanations of the decisions of deep neural network trained to label skin lesions.
@inbook{MGY2021b, address = {Amsterdam, the Netherlands}, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220209}, issn = {1879-8314}, line = {1}, month = sep, open_access = {NO}, pages = {258 - 260}, publisher = {IOS Press}, title = {Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers}, visible_on_website = {YES}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2021

Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

In 2021 IEEE Symposium on Computers and Communications (ISCC) , Sep 2021

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing mechanisms allowing for an interaction between decision systems and humans by making the decisions of the formers understandable. This is particularly important in sensitive contexts like in the medical domain. We propose a use case study, for skin lesion diagnosis, illustrating how it is possible to provide the practitioner with explanations on the decisions of a state of the art deep neural network classifier trained to characterize skin lesions from examples. Our framework consists of a trained classifier onto which an explanation module operates. The latter is able to offer the practitioner exemplars and counterexemplars for the classification diagnosis thus allowing the physician to interact with the automatic diagnosis system. The exemplars are generated via an adversarial autoencoder. We illustrate the behavior of the system on representative examples.
@inproceedings{MGY2021, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {2021 IEEE Symposium on Computers and Communications (ISCC)}, doi = {10.1109/iscc53001.2021.9631485}, line = {1}, month = sep, open_access = {NO}, pages = {1–7}, publisher = {IEEE}, title = {Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling}, visible_on_website = {YES}, year = {2021} }

2025

A Practical Approach to Causal Inference over Time

Martina Cinquini, Isacco Beretta, Salvatore Ruggieri, and Isabel Valera

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

RESEARCH LINE
2

Abs Bib HTML

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting causal VAR framework allows us to perform causal inference over time from observational time series data. Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.
@article{CBR2025, author = {Cinquini, Martina and Beretta, Isacco and Ruggieri, Salvatore and Valera, Isabel}, doi = {10.1609/aaai.v39i14.33626}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {2}, month = apr, number = {14}, open_access = {Gold}, pages = {14832–14839}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {A Practical Approach to Causal Inference over Time}, visible_on_website = {YES}, volume = {39}, year = {2025} }
The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR

Laura State, Alejandra Bringas Colmenarejo, Andrea Beretta, Salvatore Ruggieri, Franco Turini, and 1 more author

Artificial Intelligence and Law, Jan 2025

RESEARCH LINE
4

Abs Bib HTML

Explainable AI (XAI) provides methods to understand non-interpretable machine learning models. However, we have little knowledge about what legal experts expect from these explanations, including their legal compliance with, and value against European Union legislation. To close this gap, we present the Explanation Dialogues, an expert focus study to uncover the expectations, reasoning, and understanding of legal experts and practitioners towards XAI, with a specific focus on the European General Data Protection Regulation. The study consists of an online questionnaire and follow-up interviews, and is centered around a use-case in the credit domain. We extract both a set of hierarchical and interconnected codes using grounded theory, and present the standpoints of the participating experts towards XAI. We find that the presented explanations are hard to understand and lack information, and discuss issues that can arise from the different interests of the data controller and subject. Finally, we present a set of recommendations for developers of XAI methods, and indications of legal areas of discussion. Among others, recommendations address the presentation, choice, and content of an explanation, technical risks as well as the end-user, while we provide legal pointers to the contestability of explanations, transparency thresholds, intellectual property rights as well as the relationship between involved parties.
@article{SBB2025, author = {State, Laura and Bringas Colmenarejo, Alejandra and Beretta, Andrea and Ruggieri, Salvatore and Turini, Franco and Law, Stephanie}, doi = {10.1007/s10506-024-09430-w}, issn = {1572-8382}, journal = {Artificial Intelligence and Law}, line = {4}, month = jan, open_access = {Green}, publisher = {Springer Science and Business Media LLC}, title = {The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR}, visible_on_website = {YES}, year = {2025} }

2024

An Interactive Interface for Feature Space Navigation

Eleonora Cappuccio, Isacco Beretta, Marta Marchiori Manerba, and Salvatore Rinzivillo

Jun 2024

RESEARCH LINE
3

Abs Bib HTML

In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to provide users with an intuitive and direct way to navigate through the feature space, inspect model behavior, and perform what-if analyses via feature manipulations and visual feedback. We integrate multiple views including projections of high-dimensional data, decision boundary surfaces, and sensitivity indicators. The interface also supports real-time adjustments of feature values to observe the corresponding changes in the model predictions. Our experiments show that the system can help both novice and expert users to detect regions of uncertainty, identify influential features, and generate hypotheses for model improvement.
@inbook{CBM2024, author = {Cappuccio, Eleonora and Beretta, Isacco and Marchiori Manerba, Marta and Rinzivillo, Salvatore}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240184}, isbn = {9781643685229}, issn = {1879-8314}, line = {3}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {An Interactive Interface for Feature Space Navigation}, visible_on_website = {YES}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Mapping the landscape of ethical considerations in explainable AI research

Luca Nannini, Marta Marchiori Manerba, and Isacco Beretta

Ethics and Information Technology, Jun 2024

RESEARCH LINE
1

5

Abs Bib HTML

With its potential to contribute to the ethical governance of AI, explainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet the substantiation of these claims with rigorous ethical analysis and reflection remains largely unexamined. This contribution endeavours to scrutinise the relationship between XAI and ethical considerations. We conduct a systematic review of the XAI literature by applying a multi-stage filtering process and then developing a taxonomy for classifying the depth and quality of ethical engagement in the field. Our findings show that although a growing body of XAI research references ethical issues, the majority does so at a superficial level. We identify nine ethical themes (e.g., autonomy, justice, transparency) and show how they are engaged in the literature. We discuss trends, deficits, and future directions for integrating ethical considerations meaningfully into XAI research.
@article{NMB2024, author = {Nannini, Luca and Marchiori Manerba, Marta and Beretta, Isacco}, doi = {10.1007/s10676-024-09773-7}, issn = {1572-8439}, journal = {Ethics and Information Technology}, line = {1,5}, month = jun, number = {3}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Mapping the landscape of ethical considerations in explainable AI research}, visible_on_website = {YES}, volume = {26}, year = {2024} }
Requirements of eXplainable AI in Algorithmic Hiring

A. Beretta, G. Ercoli, A. Ferraro, R. Guidotti, A. Iommi, and 4 more authors

Dec 2024

RESEARCH LINE
1

Abs Bib HTML

AI models for ranking candidates to a job position are increasingly adopted. They bring a new layer of opaqueness in the way candidates are evaluated. We present preliminary research on stakeholder analysis and requirement elicitation for designing an explainability component in AI models for ranking candidates to a job position. (CEUR-WS)
@misc{BEF2024, author = {Beretta, A. and Ercoli, G. and Ferraro, A. and Guidotti, R. and Iommi, A. and Mastropietro, A. and Monreale, A. and Rotelli, D. and Ruggieri, S.}, line = {1}, month = dec, title = {Requirements of eXplainable AI in Algorithmic Hiring}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }

2023

Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }
Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
The Importance of Time in Causal Algorithmic Recourse

Isacco Beretta, and Martina Cinquini

Jun 2023

RESEARCH LINE
2

Abs Bib HTML

The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inbook{BC2023, author = {Beretta, Isacco and Cinquini, Martina}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_16}, isbn = {9783031440649}, issn = {1865-0937}, line = {2}, open_access = {NO}, pages = {283–298}, publisher = {Springer Nature Switzerland}, title = {The Importance of Time in Causal Algorithmic Recourse}, visible_on_website = {YES}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{PBP2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {4}, month = apr, open_access = {Gold}, pages = {1–9}, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, visible_on_website = {YES}, year = {2022} }

Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients

Himanshi Allahabadi, Julia Amann, Isabelle Balot, Andrea Beretta, Charles Binkley, and 52 more authors

IEEE Transactions on Technology and Society, Dec 2022

RESEARCH LINE

5

Abs Bib HTML

This article’s main contributions are twofold: 1) to demonstrate how to apply the general European Union’s High-Level Expert Group’s (EU HLEG) guidelines for trustworthy AI in practice for the domain of healthcare and 2) to investigate the research question of what does “trustworthy AI” mean at the time of the COVID-19 pandemic. To this end, we present the results of a post-hoc self-assessment to evaluate the trustworthiness of an AI system for predicting a multiregional score conveying the degree of lung compromise in COVID-19 patients, developed and verified by an interdisciplinary team with members from academia, public hospitals, and industry in time of pandemic. The AI system aims to help radiologists to estimate and communicate the severity of damage in a patient’s lung from Chest X-rays. It has been experimentally deployed in the radiology department of the ASST Spedali Civili clinic in Brescia, Italy, since December 2020 during pandemic time. The methodology we have applied for our post-hoc assessment, called Z-Inspection®, uses sociotechnical scenarios to identify ethical, technical, and domain-specific issues in the use of the AI system in the context of the pandemic.

@article{AAB2022,
  author = {Allahabadi, Himanshi and Amann, Julia and Balot, Isabelle and Beretta, Andrea and Binkley, Charles and Bozenhard, Jonas and Bruneault, Frederick and Brusseau, James and Candemir, Sema and Cappellini, Luca Alessandro and Chakraborty, Subrata and Cherciu, Nicoleta and Cociancig, Christina and Coffee, Megan and Ek, Irene and Espinosa-Leal, Leonardo and Farina, Davide and Fieux-Castagnet, Genevieve and Frauenfelder, Thomas and Gallucci, Alessio and Giuliani, Guya and Golda, Adam and van Halem, Irmhild and Hildt, Elisabeth and Holm, Sune and Kararigas, Georgios and Krier, Sebastien A. and Kuhne, Ulrich and Lizzi, Francesca and Madai, Vince I. and Markus, Aniek F. and Masis, Serg and Mathez, Emilie Wiinblad and Mureddu, Francesco and Neri, Emanuele and Osika, Walter and Ozols, Matiss and Panigutti, Cecilia and Parent, Brendan and Pratesi, Francesca and Moreno-Sanchez, Pedro A. and Sartor, Giovanni and Savardi, Mattia and Signoroni, Alberto and Sormunen, Hanna-Maria and Spezzatti, Andy and Srivastava, Adarsh and Stephansen, Annette F. and Theng, Lau Bee and Tithi, Jesmin Jahan and Tuominen, Jarno and Umbrello, Steven and Vaccher, Filippo and Vetter, Dennis and Westerlund, Magnus and Wurth, Renee and Zicari, Roberto V.},
  doi = {10.1109/tts.2022.3195114},
  issn = {2637-6415},
  journal = {IEEE Transactions on Technology and Society},
  line = {4,5},
  month = dec,
  number = {4},
  open_access = {Gold},
  pages = {272–289},
  publisher = {Institute of Electrical and Electronics Engineers (IEEE)},
  title = {Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients},
  visible_on_website = {YES},
  volume = {3},
  year = {2022}
}

User-driven counterfactual generator: a human centered exploration

Beretta I; Cappuccio E; Marchiori Manerba M

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

In this paper, we critically examine the limitations of the techno-solutionist approach to explanations in the context of counterfactual generation, reaffirming interactivity as a core value in the explanation interface between the model and the user.
@misc{BCM2022, author = {M, Beretta I; Cappuccio E; Marchiori Manerba}, line = {1,3}, month = dec, title = {User-driven counterfactual generator: a human centered exploration}, year = {2022} }

2025

Differentially Private FastSHAP for Federated Learning Model Explainability

Valerio Bonsignori, Luca Corbucci, Francesca Naretto, and Anna Monreale

In 2025 International Joint Conference on Neural Networks (IJCNN) , Jun 2025

RESEARCH LINE
1

5

Abs Bib HTML

Explaining the reasoning behind black-box model predictions while preserving user privacy is a significant challenge. This becomes even more complex in Federated Learning, where legal constraints restrict the data that clients can share with external entities. In this paper, we introduce Fastshap++, a method that adapts Fastshap to explain Federated Learning trained models. Unlike existing approaches, Fastshap++ mitigates client privacy risks by incorporating Differential Privacy into the explanation process and preventing the exchange of sensitive information between clients and external entities. We evaluate the effectiveness of Fastshap++ testing it on three different datasets, and comparing the explanations with those produced by a centralized explainer with access to clients’ training data. Lastly, we study the impact of varying levels of Differential Privacy to analyse the trade-offs between privacy and the quality of the explanations.
@inproceedings{BCN2025, address = {Rome, Italy}, author = {Bonsignori, Valerio and Corbucci, Luca and Naretto, Francesca and Monreale, Anna}, booktitle = {2025 International Joint Conference on Neural Networks (IJCNN)}, doi = {10.1109/ijcnn64981.2025.11227553}, line = {1,5}, month = jun, pages = {1–8}, publisher = {IEEE}, title = {Differentially Private FastSHAP for Federated Learning Model Explainability}, visible_on_website = {YES}, year = {2025} }
Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model

Fabio Michele Russo, Carlo Metta, Anna Monreale, Salvatore Rinzivillo, and Fabio Pinelli

Jun 2025

RESEARCH LINE
1

Abs Bib HTML

As predictive machine learning models become increasingly adopted and advanced, their role has evolved from merely predicting outcomes to actively shaping them. This evolution has underscored the importance of Trustworthy AI, highlighting the necessity to extend our focus beyond mere accuracy and toward a comprehensive understanding of these models’ behaviors within the specific contexts of their applications. To further progress in explainability, we introduce poem, Prefetched Offline Explanation Model, a model-agnostic, local explainability algorithm for image data. The algorithm generates exemplars, counterexemplars and saliency maps to provide quick and effective explanations suitable for time-sensitive scenarios. Leveraging an existing local algorithm, poem infers factual and counterfactual rules from data to create illustrative examples and opposite scenarios with an enhanced stability by design. A novel mechanism then matches incoming test points with an explanation base and produces diverse exemplars, informative saliency maps and believable counterexemplars. Experimental results indicate that poem outperforms its predecessor abele in speed and ability to generate more nuanced and varied exemplars alongside more insightful saliency maps and valuable counterexemplars.
@inbook{RMM2025, author = {Russo, Fabio Michele and Metta, Carlo and Monreale, Anna and Rinzivillo, Salvatore and Pinelli, Fabio}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_11}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {167–182}, publisher = {Springer Nature Switzerland}, title = {Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model}, visible_on_website = {YES}, year = {2025} }
An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items

Luca Corbucci, Javier Alejandro Borges Legrottaglie, Francesco Spinnato, Anna Monreale, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

Accurately identifying items forgotten during a supermarket visit and providing clear, interpretable explanations for recommending them remains an underexplored problem within the Next Basket Prediction (NBP) domain. Existing NBP approaches typically only focus on forecasting future purchases, without explicitly addressing the detection of unintentionally omitted items. This gap is partly due to the scarcity of real-world datasets that allow for the reliable estimation of forgotten items. Furthermore, most current NBP methods rely on black-box models, which lack transparency and limit the ability to justify recommendations to end users. In this paper, we formally introduce the forgotten-item prediction task and propose two novel interpretable-by-design algorithms. These methods are tailored to identify forgotten items while offering intuitive, human-understandable explanations. Experiments on a real-world retail dataset show our algorithms outperform state-of-the-art NBP baselines by 10–15% across multiple evaluation metrics.
@inbook{CBS2025, author = {Corbucci, Luca and Borges Legrottaglie, Javier Alejandro and Spinnato, Francesco and Monreale, Anna and Guidotti, Riccardo}, booktitle = {ECAI 2025}, doi = {10.3233/faia250912}, isbn = {9781643686318}, issn = {1879-8314}, line = {1}, month = oct, open_access = {Gold}, publisher = {IOS Press}, title = {An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items}, visible_on_website = {YES}, year = {2025} }
FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation

Xenia Heilmann, Luca Corbucci, Mattia Cerrato, and Anna Monreale

Dec 2025

RESEARCH LINE
5

Abs Bib

Federated Learning (FL) enables collaborative model training across multiple clients without sharing clients’ private data. However, fairness remains a key concern, as biases in local clients’ datasets can impact the entire federated system. Heterogeneous data distributions across clients may lead to models that are fairer for some clients than others. Although several fairness-enhancing solutions are present in the literature, most focus on mitigating bias for a single sensitive attribute, typically binary, overlooking the diverse and sometimes conflicting fairness needs of different clients. This limited perspective can limit the effectiveness of fairness interventions for the different clients. To support more robust and reproducible fairness research in FL, we aim to enable a consistent benchmarking of fairness-aware FL methods at both the global and client levels. In this paper, we contribute in three ways: (1) We introduce FeDa4Fair, a library to generate tabular datasets tailored to evaluating fair FL methods under heterogeneous client bias; (2) we release four bias-heterogeneous datasets and corresponding benchmarks to compare fairness mitigation methods in a controlled environment; (3) we provide ready-to-use functions for evaluating fairness outcomes for these datasets.
@misc{HCC2025, author = {Heilmann, Xenia and Corbucci, Luca and Cerrato, Mattia and Monreale, Anna}, line = {5}, month = dec, title = {FeDa4Fair: Client-Level Federated Datasets for Fairness Evaluation}, year = {2025} }
Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Dec 2025

RESEARCH LINE
1

5

Abs Bib HTML

During the last few years, the abundance of data has significantly boosted the performance of Machine Learning models, integrating them into several aspects of daily life. However, the rise of powerful Artificial Intelligence tools has introduced ethical and legal complexities. This paper proposes a computational framework to analyze the ethical and legal dimensions of Machine Learning models, focusing specifically on privacy concerns and interpretability. In fact, recently, the research community proposed privacy attacks able to reveal whether a record was part of the black-box training set or inferring variable values by accessing and querying a Machine Learning model. These attacks highlight privacy vulnerabilities and prove that GDPR regulation might be violated by making data or Machine Learning models accessible. At the same time, the complexity of these models, often labelled as “black-boxes”, has made the development of explanation methods indispensable to enhance trust and facilitate their acceptance and adoption in high-stake scenarios.
@misc{NMG2025, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, line = {1,5}, month = dec, publisher = {Trans. Data Priv. 18 (2), 67-93}, title = {Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.}, year = {2025} }

2024

Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

IEEE Access, Dec 2024

RESEARCH LINE
1

Abs Bib HTML

The current trend in time series classification is to develop highly accurate algorithms by combining multiple models in ensemble hybrids, representing time series in complex feature spaces, and extracting features from different representations. As a consequence, the best time series classifiers are black-box models, not understandable for humans. Even the approaches regarded as interpretable, such as shapelet-based ones, rely on randomization to maintain efficiency, which poses challenges for interpretability. We propose the Bag-Of-Receptive-Fields (BORF), a fast, interpretable, and deterministic time series transform. Building on the Bag-Of-Patterns, we bridge the gap between convolutional operators and discretization, enhancing the Symbolic Aggregate Approximation with dilation and stride to better capture temporal patterns at multiple scales. We propose an algorithmic speedup that reduces the time complexity associated with SAX-based classifiers, enabling the more flexible BORF, represented as a sparse multivariate tensor.
@article{SGM2024, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, doi = {10.1109/access.2024.3464743}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {137893–137912}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Explainable Authorship Identification in Cultural Heritage Applications

Mattia Setzu, Silvia Corbara, Anna Monreale, Alejandro Moreo, and Fabrizio Sebastiani

Journal on Computing and Cultural Heritage, Jun 2024

RESEARCH LINE
1

Abs Bib HTML

While a substantial amount of work has recently been devoted to enhance the performance of computational Authorship Identification (AId) systems, little to no attention has been paid to endowing AId systems with the ability to explain the reasons behind their predictions. This lacking substantially hinders the practical employment of AId methodologies, since the predictions returned by such systems are hardly useful unless they are supported with suitable explanations. In this paper, we explore the applicability of existing general-purpose eXplainable Artificial Intelligence (XAI) techniques to AId, with a special focus on explanations addressed to scholars working in cultural heritage. In particular, we assess the relative merits of three different types of XAI techniques (feature ranking, probing, factuals and counterfactual selection) on three different AId tasks (authorship attribution, authorship verification, same-authorship verification) by running experiments on real AId data. Our analysis shows that, while these techniques make important first steps towards explainable Authorship Identification, more work remains to be done in order to provide tools that can be profitably integrated in the workflows of scholars.
@article{SCM2024, author = {Setzu, Mattia and Corbara, Silvia and Monreale, Anna and Moreo, Alejandro and Sebastiani, Fabrizio}, doi = {10.1145/3654675}, issn = {1556-4711}, journal = {Journal on Computing and Cultural Heritage}, line = {1}, month = jun, number = {3}, open_access = {Gold}, pages = {1–23}, publisher = {Association for Computing Machinery (ACM)}, title = {Explainable Authorship Identification in Cultural Heritage Applications}, visible_on_website = {YES}, volume = {17}, year = {2024} }
Generative Model for Decision Trees

Riccardo Guidotti, Anna Monreale, Mattia Setzu, and Giulia Volpi

Proceedings of the AAAI Conference on Artificial Intelligence, Mar 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are among the most popular supervised models due to their interpretability and knowledge representation resembling human reasoning. Yet their design and tuning remain largely manual and analytic. In this work we place our proposal between discriminative-only design and full generative modeling: we design a generative model for decision trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. We then sample novel trees aimed at specific tasks, such as improving interpretability, compression, or fairness. Empirical results on synthetic and real data demonstrate that our generative model successfully produces new decision trees tailored to different desiderata while preserving predictive performance.
@article{GMS2024, author = {Guidotti, Riccardo and Monreale, Anna and Setzu, Mattia and Volpi, Giulia}, doi = {10.1609/aaai.v38i19.30104}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = mar, number = {19}, open_access = {Gold}, pages = {21116–21124}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Generative Model for Decision Trees}, visible_on_website = {YES}, volume = {38}, year = {2024} }
GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning

Rami Haffar, Francesca Naretto, David Sánchez, Anna Monreale, and Josep Domingo-Ferrer

In 2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) , Jun 2024

RESEARCH LINE
1

2

Abs Bib HTML

The increasing spread of artificial intelligence applications has led to decentralized frameworks that foster collaborative model training among multiple entities. One of such frameworks is federated learning, which ensures data availability in client nodes without requiring the central server to retain any data. Nevertheless, similar to centralized neural networks, interpretability remains a challenge in understanding the predictions of these decentralized frameworks. The limited access to data on the server side further complicates the applicability of explainers in such frameworks. To address this challenge, we propose GLOR-FLEX, a framework designed to generate rule-based global explanations from local explainers. GLOR-FLEX ensures client privacy by preventing the sharing of actual data between the clients and the server. The proposed framework initiates the process by constructing local decision trees on each client’s side to produce local explanations. Subsequently, by using rule extraction from these trees and strategically sorting and merging those rules, the server obtains a merged set of rules suitable to be used as a global explainer. We empirically evaluate the performance of GLOR-FLEX on three distinct tabular data sets, showing high fidelity scores between the explainers and both the local and global models. Our results support the effectiveness of GLOR-FLEX in generating accurate explanations that efficiently detect and explain the behavior of both local and global models.
@inproceedings{HNS2024, author = {Haffar, Rami and Naretto, Francesca and Sánchez, David and Monreale, Anna and Domingo-Ferrer, Josep}, booktitle = {2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)}, doi = {10.1109/fuzz-ieee60900.2024.10611878}, line = {1,2}, month = jun, open_access = {NO}, pages = {1–9}, publisher = {IEEE}, title = {GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning}, visible_on_website = {YES}, year = {2024} }
Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma

Giulia Peduzzi, Alessio Felici, Roberto Pellungrini, Francesca Giorgolo, Riccardo Farinella, and 7 more authors

Digestive and Liver Disease, Jun 2024

RESEARCH LINE
5

Abs Bib HTML

Background: Current knowledge on pancreatic ductal adenocarcinoma (PDAC) risk factors is limited and no study has comprehensively tested the exposome alongside genetic variability for disease susceptibility. We analyzed 347 exposure variables and a polygenic risk score in UK Biobank data (816 PDAC cases, 302,645 controls). Fifty-two associations passed Bonferroni correction. Known risk factors such as smoking, pancreatitis, diabetes, heavy alcohol use and high BMI were confirmed. Novel associations include mobile phone usage intensity and multiple stress-related lifestyle factors. PRS was associated with PDAC risk but no gene‚Äìenvironment interactions were detected. Conclusion: Stressful lifestyle and sedentary behavior may play a major role in PDAC susceptibility independently of genetics.
@article{PFP2023, author = {Peduzzi, Giulia and Felici, Alessio and Pellungrini, Roberto and Giorgolo, Francesca and Farinella, Riccardo and Gentiluomo, Manuel and Spinelli, Andrea and Capurso, Gabriele and Monreale, Anna and Canzian, Federico and Calderisi, Marco and Campa, Daniele}, doi = {10.1016/j.dld.2023.10.015}, issn = {1590-8658}, journal = {Digestive and Liver Disease}, line = {5}, month = jun, number = {6}, open_access = {Gold}, pages = {1054–1063}, publisher = {Elsevier BV}, title = {Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma}, visible_on_website = {YES}, volume = {56}, year = {2024} }
Requirements of eXplainable AI in Algorithmic Hiring

A. Beretta, G. Ercoli, A. Ferraro, R. Guidotti, A. Iommi, and 4 more authors

Dec 2024

RESEARCH LINE
1

Abs Bib HTML

AI models for ranking candidates to a job position are increasingly adopted. They bring a new layer of opaqueness in the way candidates are evaluated. We present preliminary research on stakeholder analysis and requirement elicitation for designing an explainability component in AI models for ranking candidates to a job position. (CEUR-WS)
@misc{BEF2024, author = {Beretta, A. and Ercoli, G. and Ferraro, A. and Guidotti, R. and Iommi, A. and Mastropietro, A. and Monreale, A. and Rotelli, D. and Ruggieri, S.}, line = {1}, month = dec, title = {Requirements of eXplainable AI in Algorithmic Hiring}, year = {2024} }

2023

Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{LSG2023, address = {Cham, Switzerland}, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Advances in Intelligent Data Analysis XXI}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, visible_on_website = {YES}, year = {2023} }

2022

Privacy Risk of Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Sep 2022

RESEARCH LINE
5

Abs Bib HTML

In this paper we propose to study a methodology that enables the evaluation of the privacy risk exposure of global explainers based on an interpretable classifier that imitates the global reasoning of a black-box classifier. The idea is to verify if the layer of interpretability added by the interpretable model can jeopardize the privacy protection of the training data used for learning the black-box classifier. In order to address this problem, we exploit a well-known attack model called membership inference attack (MIA). We then compute the privacy risk change ΔR due to the introduction of the global explainer c. The preliminary experimental results suggest that global explainers based on decision trees introduce a higher risk of privacy, increasing the percentage of records identified as members of the training dataset used to train the original black-box classifiers. These results suggest that in order to provide Trustworthy AI, it becomes fundamental to consider the relationship between different ethical values to identify possible values like transparency and privacy that may be in contrast, and studying solutions that enable the simultaneous satisfaction of more than one value.
@inbook{NMG2022, address = {Amsterdam, the Netherlands}, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220206}, issn = {1879-8314}, line = {5}, month = sep, open_access = {Gold}, pages = {249 - 251}, publisher = {IOS Press}, title = {Privacy Risk of Global Explainers}, visible_on_website = {YES}, year = {2022} }
Evaluating the Privacy Exposure of Interpretable Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

In 2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2022

RESEARCH LINE
5

Abs Bib HTML

In recent years we are witnessing the diffusion of AI systems based on powerful Machine Learning models which find application in many critical contexts such as medicine, financial market and credit scoring. In such a context it is particularly important to design Trustworthy AI systems while guaranteeing transparency, with respect to their decision reasoning and privacy protection. Although many works in the literature addressed the lack of transparency and the risk of privacy exposure of Machine Learning models, the privacy risks of explainers have not been appropriately studied. This paper presents a methodology for evaluating the privacy exposure raised by interpretable global explainers able to imitate the original black-box classifier. Our methodology exploits the well-known Membership Inference Attack. The experimental results highlight that global explainers based on interpretable trees lead to an increase in privacy exposure.
@inproceedings{NMG2022b, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi56440.2022.00012}, line = {5}, month = dec, open_access = {NO}, pages = {13–19}, publisher = {IEEE}, title = {Evaluating the Privacy Exposure of Interpretable Global Explainers}, visible_on_website = {YES}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }

2021

TRIPLEx: Triple Extraction for Explanation

Mattia Setzu, Anna Monreale, and Pasquale Minervini

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
1

2

Abs Bib HTML

Transformer-based models are used to solve a variety of Natural Language Processing tasks. Still, these models are opaque and poorly understandable for their users. Current approaches to explainability focus on token importance, in which the explanation consists of a set of tokens relevant to the prediction, and natural language explanations, in which the explanation is a generated piece of text. The latter are usually learned by design with models trained end-to-end to provide a prediction and an explanation, or rely on powerful external text generators to do the heavy lifting for them. In this paper we present TriplEX, an explainability algorithm for Transformer-based models fine-tuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks. TriplEX explains Transformers-based models by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation.
@inproceedings{SMM2022, author = {Setzu, Mattia and Monreale, Anna and Minervini, Pasquale}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00015}, line = {1,2}, month = dec, open_access = {NO}, pages = {44–53}, publisher = {IEEE}, title = {TRIPLEx: Triple Extraction for Explanation}, visible_on_website = {YES}, year = {2021} }
Designing Shapelets for Interpretable Data-Agnostic Classification

Riccardo Guidotti, and Anna Monreale

In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2021

RESEARCH LINE
1

Abs Bib HTML

Time series shapelets are discriminatory subsequences which are representative of a class, and their similarity to a time series can be used for successfully tackling the time series classification problem. The literature shows that Artificial Intelligence (AI) systems adopting classification models based on time series shapelets can be interpretable, more accurate, and significantly fast. Thus, in order to design a data-agnostic and interpretable classification approach, in this paper we first extend the notion of shapelets to different types of data, i.e., images, tabular and textual data. Then, based on this extended notion of shapelets we propose an interpretable data-agnostic classification method. Since the shapelets discovery can be time consuming, especially for data types more complex than time series, we exploit a notion of prototypes for finding candidate shapelets, and reducing both the time required to find a solution and the variance of shapelets. A wide experimentation on datasets of different types shows that the data-agnostic prototype-based shapelets returned by the proposed method empower an interpretable classification which is also fast, accurate, and stable. In addition, we show and we prove that shapelets can be at the basis of explainable AI methods.
@inproceedings{GM2021, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’21}, doi = {10.1145/3461702.3462553}, line = {1}, month = jul, open_access = {NO}, pages = {532–542}, publisher = {ACM}, series = {AIES ’21}, title = {Designing Shapelets for Interpretable Data-Agnostic Classification}, visible_on_website = {YES}, year = {2021} }
GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }
Deriving a Single Interpretable Model by Merging Tree-Based Classifiers

Valerio Bonsignori, Riccardo Guidotti, and Anna Monreale

May 2021

RESEARCH LINE
1

2

Abs Bib HTML

Decision tree classifiers have been proved to be among the most interpretable models due to their intuitive structure that illustrates decision processes in form of logical rules. Unfortunately, more complex tree-based classifiers such as oblique trees and random forests overcome the accuracy of decision trees at the cost of becoming non interpretable. In this paper, we propose a method that takes as input any tree-based classifier and returns a single decision tree able to approximate its behavior. Our proposal merges tree-based classifiers by an intensional and extensional approach and applies a post-hoc explanation strategy. Our experiments shows that the retrieved single decision tree is at least as accurate as the original tree-based model, faithful, and more interpretable.
@inbook{BGM2021, author = {Bonsignori, Valerio and Guidotti, Riccardo and Monreale, Anna}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-88942-5_27}, isbn = {9783030889425}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {347–357}, publisher = {Springer International Publishing}, title = {Deriving a Single Interpretable Model by Merging Tree-Based Classifiers}, visible_on_website = {YES}, year = {2021} }
Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals

Michele Resta, Anna Monreale, and Davide Bacciu

Entropy, Aug 2021

RESEARCH LINE
4

Abs Bib HTML

The biomedical field is characterized by an ever-increasing production of sequential data, which often come in the form of biosignals capturing the time-evolution of physiological processes, such as blood pressure and brain activity. This has motivated a large body of research dealing with the development of machine learning techniques for the predictive analysis of such biosignals. Unfortunately, in high-stakes decision making, such as clinical diagnosis, the opacity of machine learning models becomes a crucial aspect to be addressed in order to increase the trust and adoption of AI technology. In this paper, we propose a model agnostic explanation method, based on occlusion, that enables the learning of the input’s influence on the model predictions. We specifically target problems involving the predictive analysis of time-series data and the models that are typically used to deal with data of such nature, i.e., recurrent neural networks. Our approach is able to provide two different kinds of explanations: one suitable for technical experts, who need to verify the quality and correctness of machine learning models, and one suited to physicians, who need to understand the rationale underlying the prediction to make aware decisions. A wide experimentation on different physiological data demonstrates the effectiveness of our approach both in classification and regression tasks.
@article{RAB2021, author = {Resta, Michele and Monreale, Anna and Bacciu, Davide}, doi = {10.3390/e23081064}, issn = {1099-4300}, journal = {Entropy}, line = {4}, month = aug, number = {8}, open_access = {Gold}, pages = {1064}, publisher = {MDPI AG}, title = {Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals}, visible_on_website = {YES}, volume = {23}, year = {2021} }

2020

Global Explanations with Local Scoring

Mattia Setzu, Riccardo Guidotti, Anna Monreale, and Franco Turini

Aug 2020

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{SGM2019, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-43823-4_14}, isbn = {9783030438234}, issn = {1865-0937}, line = {1}, open_access = {NO}, pages = {159–171}, publisher = {Springer International Publishing}, title = {Global Explanations with Local Scoring}, visible_on_website = {YES}, year = {2020} }
Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Aug 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by “morphing” into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
@inbook{GMM2019, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-46150-8_12}, isbn = {9783030461508}, issn = {1611-3349}, line = {1,4}, pages = {189–205}, publisher = {Springer International Publishing}, title = {Black Box Explanation by Learning Image Exemplars in the Latent Feature Space}, visible_on_website = {YES}, year = {2020} }
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2020

RESEARCH LINE
1

4

Abs Bib HTML

We present an approach to explain the decisions of black box image classifiers through synthetic exemplar and counter-exemplar learnt in the latent feature space. Our explanation method exploits the latent representations learned through an adversarial autoencoder for generating a synthetic neighborhood of the image for which an explanation is required. A decision tree is trained on a set of images represented in the latent space, and its decision rules are used to generate exemplar images showing how the original image can be modified to stay within its class. Counterfactual rules are used to generate counter-exemplars showing how the original image can “morph” into another class. The explanation also comprehends a saliency map highlighting the areas that contribute to its classification, and areas that push it into another class. A wide and deep experimental evaluation proves that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability, besides providing the most useful and interpretable explanations.
@article{GMM2020, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, doi = {10.1609/aaai.v34i09.7116}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,4}, month = apr, number = {09}, open_access = {NO}, pages = {13665–13668}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations}, visible_on_website = {YES}, volume = {34}, year = {2020} }
Predicting and Explaining Privacy Risk Exposure in Mobility Data

Francesca Naretto, Roberto Pellungrini, Anna Monreale, Franco Maria Nardini, and Mirco Musolesi

Apr 2020

RESEARCH LINE
4

5

Abs Bib HTML

Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.
@inbook{NPM2020, author = {Naretto, Francesca and Pellungrini, Roberto and Monreale, Anna and Nardini, Franco Maria and Musolesi, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-61527-7_27}, isbn = {9783030615277}, issn = {1611-3349}, line = {4,5}, open_access = {NO}, pages = {403–418}, publisher = {Springer International Publishing}, title = {Predicting and Explaining Privacy Risk Exposure in Mobility Data}, visible_on_website = {YES}, year = {2020} }
Data-Agnostic Local Neighborhood Generation

Riccardo Guidotti, and Anna Monreale

In 2020 IEEE International Conference on Data Mining (ICDM) , Nov 2020

RESEARCH LINE
1

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, machine learning explanation, etc. In such contexts, it is important to generate data samples located within “local” areas surrounding specific instances. Local synthetic data can help the learning phase of predictive models, and it is fundamental for methods explaining the local behavior of obscure classifiers. The contribution of this paper is twofold. First, we introduce a method based on generative operators allowing the synthetic neighborhood generation by applying specific perturbations on a given input instance. The key factor consists in performing a data transformation that makes applicable to any type of data, i.e., data-agnostic. Second, we design a framework for evaluating the goodness of local synthetic neighborhoods exploiting both supervised and unsupervised methodologies. A deep experimentation shows the effectiveness of the proposed method.
@inproceedings{GM2020, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {2020 IEEE International Conference on Data Mining (ICDM)}, doi = {10.1109/icdm50108.2020.00122}, issn = {2374-8486}, line = {1}, month = nov, open_access = {NO}, pages = {1040–1045}, publisher = {IEEE}, title = {Data-Agnostic Local Neighborhood Generation}, visible_on_website = {YES}, year = {2020} }
Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{GMS2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, open_access = {NO}, pages = {167–176}, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, visible_on_website = {YES}, year = {2020} }
Rischi etico-legali dell’Intelligenza Artificiale

Monreale Anna

Dec 2020

RESEARCH LINE
5

Abs Bib HTML

Nowadays, a wide variety of personal data, describing directly or indirectly individuals’ activities, are collected and available, due to the spread of different apps, sensors and mobile devices. The availability of such data opens unprecedented opportunities of developing AI systems exploiting those data to provide a wide range of benefits in different domains. Unfortunately, in a critical domain like healthcare and justice the development and the application of AI systems can rise many ethical and legal concerns. This article discusses the ethical and legal implication in terms of privacy and transparency, also providing an overview on the different approaches to ethics across the world.
@misc{M2020, author = {Anna, Monreale}, issn = {2037-6677}, line = {5}, month = dec, title = {Rischi etico-legali dell’Intelligenza Artificiale}, year = {2020} }
Opening the black box: a primer for anti-discrimination

Ruggieri Salvatore, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pedreschi Dino, and 1 more author

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

The pervasive adoption of Artificial Intelligence (AI) models in the modern information society, requires counterbalancing the growing decision power demanded to AI models with risk assessment methodologies. In this paper, we consider the risk of discriminatory decisions and review approaches for discovering discrimination and for designing fair AI models. We highlight the tight relations between discrimination discovery and explainable AI, with the latter being a more general approach for understanding the behavior of black boxes.
@misc{RGG2020, address = {Italy}, author = {Salvatore, Ruggieri and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi and Franco, Turini}, line = {1}, month = dec, title = {Opening the black box: a primer for anti-discrimination}, year = {2020} }

2019

Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{PGG2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, visible_on_website = {YES}, volume = {33}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

4

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{GMG2019, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1,4}, month = nov, number = {6}, open_access = {Gold}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, visible_on_website = {YES}, volume = {34}, year = {2019} }
Explaining Multi-label Black-Box Classifiers for Health Applications

Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi

Aug 2019

RESEARCH LINE
1

4

Abs Bib HTML

Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{PGM2019, author = {Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino}, booktitle = {Precision Health and Medicine}, doi = {10.1007/978-3-030-24409-5_9}, isbn = {9783030244095}, issn = {1860-9503}, line = {1,4}, month = aug, pages = {97–110}, publisher = {Springer International Publishing}, title = {Explaining Multi-label Black-Box Classifiers for Health Applications}, visible_on_website = {YES}, year = {2019} }
The AI black box explanation problem

Guidotti Riccardo, Monreale Anna, and Pedreschi Dino

Dec 2019

RESEARCH LINE
1

2

3

Abs Bib HTML

The use of machine learning in decision-making has triggered an intense debate about “fair algorithms”. Given that fairness intuitions differ and can led to conflicting technical requirements, there is a pressing need to integrate ethical thinking into research and design of machine learning. We outline a framework showing how this can be done.
@misc{GMP2019, author = {Riccardo, Guidotti and Anna, Monreale and Dino, Pedreschi}, line = {1,2,3}, month = dec, publisher = {ERCIM – the European Research Consortium for Informatics and Mathematics}, title = {The AI black box explanation problem}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

3

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{GMR2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1,3}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, visible_on_website = {YES}, volume = {51}, year = {2018} }
Open the Black Box Data-Driven Explanation of Black Box Decision Systems

Pedreschi Dino, Giannotti Fosca, Guidotti Riccardo, Monreale Anna, Pappalardo Luca, and 2 more authors

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

Black box systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases hidden in the algorithms, due to human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We introduce the local-to-global framework for black box explanation, a novel approach with promising early results, which paves the road for a wide spectrum of future developments along three dimensions: (i) the language for expressing explanations in terms of highly expressive logic-based rules, with a statistical and causal interpretation; (ii) the inference of local explanations aimed at revealing the logic of the decision adopted for a specific instance by querying and auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of the many local explanations into simple global ones, with algorithms that optimize the quality and comprehensibility of explanations.
@misc{PGG2018, author = {Dino, Pedreschi and Fosca, Giannotti and Riccardo, Guidotti and Anna, Monreale and Luca, Pappalardo and Salvatore, Ruggieri and Franco, Turini}, doi = {1806.09936}, line = {1}, month = dec, publisher = {Arxive}, title = {Open the Black Box Data-Driven Explanation of Black Box Decision Systems}, year = {2018} }
Local Rule-Based Explanations of Black Box Decision Systems

Guidotti Riccardo, Monreale Anna, Ruggieri Salvatore, Pedreschi Dino, Turini Franco, and 1 more author

Dec 2018

RESEARCH LINE
1

Abs Bib HTML

The recent years have witnessed the rise of accurate but obscure decision systems which hide the logic of their internal decision processes to the users. The lack of explanations for the decisions of black box systems is a key ethical issue, and a limitation to the adoption of achine learning components in socially sensitive and safety-critical contexts. Therefore, we need explanations that reveals the reasons why a predictor takes a certain decision. In this paper we focus on the problem of black box outcome explanation, i.e., explaining the reasons of the decision taken on a specific instance. We propose LORE, an agnostic method able to provide interpretable and faithful explanations. LORE first leans a local interpretable predictor on a synthetic neighborhood generated by a genetic algorithm. Then it derives from the logic of the local interpretable predictor a meaningful explanation consisting of: a decision rule, which explains the reasons of the decision; and a set of counterfactual rules, suggesting the changes in the instance’s features that lead to a different outcome. Wide experiments show that LORE outperforms existing methods and baselines both in the quality of explanations and in the accuracy in mimicking the black box.
@misc{GMR2018a, author = {Riccardo, Guidotti and Anna, Monreale and Salvatore, Ruggieri and Dino, Pedreschi and Franco, Turini and Fosca, Giannotti}, doi = {1805.10820}, line = {1}, month = dec, publisher = {Arxive}, title = {Local Rule-Based Explanations of Black Box Decision Systems}, year = {2018} }

2023

Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories

Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda

Dec 2023

RESEARCH LINE
1

3

5

Abs Bib HTML

Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, identifying privacy risks is essential before deciding to release it publicly. Recent work has proposed using machine learning models for predicting privacy risk on raw mobility trajectories and using SHAP for risk explanation. However, applying SHAP to mobility data results in explanations of limited use both for privacy experts and end-users. In this work, we present EXPHLOT, a novel version of the Expert privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification to improve risk prediction while reducing computation time. We also devise an entropy-based mask to efficiently compute SHAP values and develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of privacy risk.
@inbook{NPR2023, author = {Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_22}, isbn = {9783031452758}, issn = {1611-3349}, line = {1,3,5}, open_access = {Gold}, pages = {325–340}, publisher = {Springer Nature Switzerland}, title = {EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories}, visible_on_website = {YES}, year = {2023} }
Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations

Eleonora Cappuccio, Daniele Fadda, Rosa Lanzilotti, and Salvatore Rinzivillo

In Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter , Sep 2023

RESEARCH LINE
1

2

3

Abs Bib HTML

The Human-Computer Interaction (HCI) community has long stressed the need for a more user-centered approach to Explainable Artificial Intelligence (XAI), a research area that aims at defining algorithms and tools to illustrate the predictions of the so-called black-box models. This approach can benefit from the fields of user-interface, user experience, and visual analytics. In this demo, we propose a visual-based tool, "F.I.P.E.R.", that shows interactive explanations combining rules and feature importance.
@inproceedings{CFR2023, author = {Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore}, booktitle = {Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter}, collection = {CHItaly 2023}, doi = {10.1145/3605390.3610811}, line = {1,2,3}, month = sep, open_access = {NO}, pages = {1–4}, publisher = {ACM}, series = {CHItaly 2023}, title = {Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations}, visible_on_website = {YES}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2023

Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{PBP2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {4}, month = apr, open_access = {Gold}, pages = {1–9}, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, visible_on_website = {YES}, year = {2022} }

Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients

Himanshi Allahabadi, Julia Amann, Isabelle Balot, Andrea Beretta, Charles Binkley, and 52 more authors

IEEE Transactions on Technology and Society, Dec 2022

RESEARCH LINE

5

Abs Bib HTML

This article’s main contributions are twofold: 1) to demonstrate how to apply the general European Union’s High-Level Expert Group’s (EU HLEG) guidelines for trustworthy AI in practice for the domain of healthcare and 2) to investigate the research question of what does “trustworthy AI” mean at the time of the COVID-19 pandemic. To this end, we present the results of a post-hoc self-assessment to evaluate the trustworthiness of an AI system for predicting a multiregional score conveying the degree of lung compromise in COVID-19 patients, developed and verified by an interdisciplinary team with members from academia, public hospitals, and industry in time of pandemic. The AI system aims to help radiologists to estimate and communicate the severity of damage in a patient’s lung from Chest X-rays. It has been experimentally deployed in the radiology department of the ASST Spedali Civili clinic in Brescia, Italy, since December 2020 during pandemic time. The methodology we have applied for our post-hoc assessment, called Z-Inspection®, uses sociotechnical scenarios to identify ethical, technical, and domain-specific issues in the use of the AI system in the context of the pandemic.

@article{AAB2022,
  author = {Allahabadi, Himanshi and Amann, Julia and Balot, Isabelle and Beretta, Andrea and Binkley, Charles and Bozenhard, Jonas and Bruneault, Frederick and Brusseau, James and Candemir, Sema and Cappellini, Luca Alessandro and Chakraborty, Subrata and Cherciu, Nicoleta and Cociancig, Christina and Coffee, Megan and Ek, Irene and Espinosa-Leal, Leonardo and Farina, Davide and Fieux-Castagnet, Genevieve and Frauenfelder, Thomas and Gallucci, Alessio and Giuliani, Guya and Golda, Adam and van Halem, Irmhild and Hildt, Elisabeth and Holm, Sune and Kararigas, Georgios and Krier, Sebastien A. and Kuhne, Ulrich and Lizzi, Francesca and Madai, Vince I. and Markus, Aniek F. and Masis, Serg and Mathez, Emilie Wiinblad and Mureddu, Francesco and Neri, Emanuele and Osika, Walter and Ozols, Matiss and Panigutti, Cecilia and Parent, Brendan and Pratesi, Francesca and Moreno-Sanchez, Pedro A. and Sartor, Giovanni and Savardi, Mattia and Signoroni, Alberto and Sormunen, Hanna-Maria and Spezzatti, Andy and Srivastava, Adarsh and Stephansen, Annette F. and Theng, Lau Bee and Tithi, Jesmin Jahan and Tuominen, Jarno and Umbrello, Steven and Vaccher, Filippo and Vetter, Dennis and Westerlund, Magnus and Wurth, Renee and Zicari, Roberto V.},
  doi = {10.1109/tts.2022.3195114},
  issn = {2637-6415},
  journal = {IEEE Transactions on Technology and Society},
  line = {4,5},
  month = dec,
  number = {4},
  open_access = {Gold},
  pages = {272–289},
  publisher = {Institute of Electrical and Electronics Engineers (IEEE)},
  title = {Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients},
  visible_on_website = {YES},
  volume = {3},
  year = {2022}
}

2021

Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche

Bosi Emanuele Panigutti Cecilia

il Diabete, Dec 2021

RESEARCH LINE

Bib HTML

@article{PB2021,
  author = {Panigutti Cecilia, Bosi Emanuele},
  doi = {10.30682/ildia2101f},
  issn = {1720-8335},
  journal = {il Diabete},
  line = {4},
  number = {1},
  open_access = {NO},
  publisher = {Bologna University Press Foundation},
  title = {Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche},
  visible_on_website = {YES},
  volume = {33},
  year = {2021}
}

FairLens: Auditing black-box clinical decision support systems

Cecilia Panigutti, Alan Perotti, André Panisson, Paolo Bajardi, and Dino Pedreschi

Information Processing & Management, Sep 2021

RESEARCH LINE
1

4

Abs Bib HTML

Highlights: We present a pipeline to detect and explain potential fairness issues in Clinical DSS. We study and compare different multi-label classification disparity measures. We explore ICD9 bias in MIMIC-IV, an openly available ICU benchmark dataset
@article{PPB2021, author = {Panigutti, Cecilia and Perotti, Alan and Panisson, André and Bajardi, Paolo and Pedreschi, Dino}, doi = {10.1016/j.ipm.2021.102657}, issn = {0306-4573}, journal = {Information Processing & Management}, line = {1,4}, month = sep, number = {5}, open_access = {Gold}, pages = {102657}, publisher = {Elsevier BV}, title = {FairLens: Auditing black-box clinical decision support systems}, visible_on_website = {YES}, volume = {58}, year = {2021} }

2020

Doctor XAI: an ontology-based approach to black-box sequential data classification explanations

Cecilia Panigutti, Alan Perotti, and Dino Pedreschi

In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , Jan 2020

RESEARCH LINE
1

3

4

Abs Bib HTML

Several recent advancements in Machine Learning involve blackbox models: algorithms that do not provide human-understandable explanations in support of their decisions. This limitation hampers the fairness, accountability and transparency of these models; the field of eXplainable Artificial Intelligence (XAI) tries to solve this problem providing human-understandable explanations for black-box models. However, healthcare datasets (and the related learning tasks) often present peculiar features, such as sequential data, multi-label predictions, and links to structured background knowledge. In this paper, we introduce Doctor XAI, a model-agnostic explainability technique able to deal with multi-labeled, sequential, ontology-linked data. We focus on explaining Doctor AI, a multilabel classifier which takes as input the clinical history of a patient in order to predict the next visit. Furthermore, we show how exploiting the temporal dimension in the data and the domain knowledge encoded in the medical ontology improves the quality of the mined explanations.
@inproceedings{PPP2020, author = {Panigutti, Cecilia and Perotti, Alan and Pedreschi, Dino}, booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency}, collection = {FAT* ’20}, doi = {10.1145/3351095.3372855}, line = {1,3,4}, month = jan, open_access = {NO}, pages = {629–639}, publisher = {ACM}, series = {FAT* ’20}, title = {Doctor XAI: an ontology-based approach to black-box sequential data classification explanations}, visible_on_website = {YES}, year = {2020} }

2019

Explaining Multi-label Black-Box Classifiers for Health Applications

Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi

Aug 2019

RESEARCH LINE
1

4

Abs Bib HTML

Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{PGM2019, author = {Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino}, booktitle = {Precision Health and Medicine}, doi = {10.1007/978-3-030-24409-5_9}, isbn = {9783030244095}, issn = {1860-9503}, line = {1,4}, month = aug, pages = {97–110}, publisher = {Springer International Publishing}, title = {Explaining Multi-label Black-Box Classifiers for Health Applications}, visible_on_website = {YES}, year = {2019} }

2025

Group Explainability Through Local Approximation

Mattia Setzu, Riccardo Guidotti, Dino Pedreschi, and Fosca Giannotti

Oct 2025

RESEARCH LINE
2

Abs Bib HTML

Machine learning models are becoming increasingly complex and widely adopted. Interpretable machine learning allows us to not only make predictions but also understand the rationale behind automated decisions through explanations. Explanations are typically characterized by their scope: local explanations are generated by local surrogate models for specific instances, while global explanations aim to approximate the behavior of the entire black-box model. In this paper, we break this dichotomy of locality to explore an underexamined area that lies between these two extremes: meso-level explanations. The goal of meso-level explainability is to provide explanations using a set of meso-level interpretable models, which capture patterns at an intermediate level of abstraction. To this end, we propose GrouX, an explainable-by-design algorithm that generates meso-level explanations in the form of feature importance scores. Our approach includes a partitioning phase that identifies meso groups, followed by the training of interpretable models within each group. We evaluate GrouX on a collection of tabular datasets, reporting both the accuracy and complexity of the resulting meso models, and compare it against other meso-level explainability algorithms. Additionally, we analyze the algorithm’s sensitivity to its hyperparameters to better understand its behavior and robustness.
@inbook{SGP2025, address = {ECAI 2025}, author = {Setzu, Mattia and Guidotti, Riccardo and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {ECAI 2025}, doi = {10.3233/faia250902}, isbn = {9781643686318}, issn = {1879-8314}, line = {2}, month = oct, open_access = {Gold}, pages = {952 - 958}, publisher = {IOS Press}, title = {Group Explainability Through Local Approximation}, visible_on_website = {YES}, year = {2025} }

2024

FairBelief - Assessing Harmful Beliefs in Language Models

Mattia Setzu, Marta Marchiori Manerba, Pasquale Minervini, and Debora Nozza

In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024) , Oct 2024

RESEARCH LINE
5

Abs Bib HTML

Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing.This paper proposes FairBelief, an analytical approach to capture and assess beliefs, i.e., propositions that an LM may embed with different degrees of confidence and that covertly influence its predictions. With FairBelief, we leverage prompting to study the behavior of several state-of-the-art LMs across different previously neglected axes, such as model scale and likelihood, assessing predictions on a fairness dataset specifically designed to quantify LMs’ outputs’ hurtfulness.Finally, we conclude with an in-depth qualitative assessment of the beliefs emitted by the models.We apply FairBelief to English LMs, revealing that, although these architectures enable high performances on diverse natural language processing tasks, they show hurtful beliefs about specific genders. Interestingly, training procedure and dataset, model scale, and architecture induce beliefs of different degrees of hurtfulness.
@inproceedings{SMM2024, author = {Setzu, Mattia and Marchiori Manerba, Marta and Minervini, Pasquale and Nozza, Debora}, booktitle = {Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024)}, doi = {10.18653/v1/2024.trustnlp-1.3}, line = {5}, open_access = {Gold}, pages = {27–39}, publisher = {Association for Computational Linguistics}, title = {FairBelief - Assessing Harmful Beliefs in Language Models}, visible_on_website = {YES}, year = {2024} }
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers

Guillermo Fernandez, Riccardo Guidotti, Fosca Giannotti, Mattia Setzu, Juan A. Aledo, and 2 more authors

Oct 2024

RESEARCH LINE
1

2

Abs Bib HTML

The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inbook{FGG2024, address = {Cham, Switzerland}, author = {Fernandez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and Gámez, Jose A. and Puerta, Jose M.}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_16}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,2}, open_access = {NO}, pages = {197–209}, publisher = {Springer Nature Switzerland}, title = {FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers}, visible_on_website = {YES}, year = {2024} }
Explainable Authorship Identification in Cultural Heritage Applications

Mattia Setzu, Silvia Corbara, Anna Monreale, Alejandro Moreo, and Fabrizio Sebastiani

Journal on Computing and Cultural Heritage, Jun 2024

RESEARCH LINE
1

Abs Bib HTML

While a substantial amount of work has recently been devoted to enhance the performance of computational Authorship Identification (AId) systems, little to no attention has been paid to endowing AId systems with the ability to explain the reasons behind their predictions. This lacking substantially hinders the practical employment of AId methodologies, since the predictions returned by such systems are hardly useful unless they are supported with suitable explanations. In this paper, we explore the applicability of existing general-purpose eXplainable Artificial Intelligence (XAI) techniques to AId, with a special focus on explanations addressed to scholars working in cultural heritage. In particular, we assess the relative merits of three different types of XAI techniques (feature ranking, probing, factuals and counterfactual selection) on three different AId tasks (authorship attribution, authorship verification, same-authorship verification) by running experiments on real AId data. Our analysis shows that, while these techniques make important first steps towards explainable Authorship Identification, more work remains to be done in order to provide tools that can be profitably integrated in the workflows of scholars.
@article{SCM2024, author = {Setzu, Mattia and Corbara, Silvia and Monreale, Anna and Moreo, Alejandro and Sebastiani, Fabrizio}, doi = {10.1145/3654675}, issn = {1556-4711}, journal = {Journal on Computing and Cultural Heritage}, line = {1}, month = jun, number = {3}, open_access = {Gold}, pages = {1–23}, publisher = {Association for Computing Machinery (ACM)}, title = {Explainable Authorship Identification in Cultural Heritage Applications}, visible_on_website = {YES}, volume = {17}, year = {2024} }
Data-Agnostic Pivotal Instances Selection for Decision-Making Models

Alessio Cascione, Mattia Setzu, and Riccardo Guidotti

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

As decision-making processes grow increasingly complex, machine learning tools have become essential in tackling business and societal challenges. However, many methodologies rely on complex models that are difficult for experts and users to interpret. We propose a hierarchical and interpretable pivot selection model inspired by Decision Trees, selecting representative pivotal instances based on similarity. The approach is data-agnostic and can leverage pretrained networks for data transformation. Experiments across tabular, text, image, and time-series datasets show superior performance to naive and state-of-the-art instance selectors, while minimizing the number of pivots and maintaining interpretability.
@inbook{CSG2024b, author = {Cascione, Alessio and Setzu, Mattia and Guidotti, Riccardo}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-031-70341-6_22}, isbn = {9783031703416}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {367–386}, publisher = {Springer Nature Switzerland}, title = {Data-Agnostic Pivotal Instances Selection for Decision-Making Models}, visible_on_website = {YES}, year = {2024} }
Generative Model for Decision Trees

Riccardo Guidotti, Anna Monreale, Mattia Setzu, and Giulia Volpi

Proceedings of the AAAI Conference on Artificial Intelligence, Mar 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are among the most popular supervised models due to their interpretability and knowledge representation resembling human reasoning. Yet their design and tuning remain largely manual and analytic. In this work we place our proposal between discriminative-only design and full generative modeling: we design a generative model for decision trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. We then sample novel trees aimed at specific tasks, such as improving interpretability, compression, or fairness. Empirical results on synthetic and real data demonstrate that our generative model successfully produces new decision trees tailored to different desiderata while preserving predictive performance.
@article{GMS2024, author = {Guidotti, Riccardo and Monreale, Anna and Setzu, Mattia and Volpi, Giulia}, doi = {10.1609/aaai.v38i19.30104}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = mar, number = {19}, open_access = {Gold}, pages = {21116–21124}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Generative Model for Decision Trees}, visible_on_website = {YES}, volume = {38}, year = {2024} }

2021

TRIPLEx: Triple Extraction for Explanation

Mattia Setzu, Anna Monreale, and Pasquale Minervini

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
1

2

Abs Bib HTML

Transformer-based models are used to solve a variety of Natural Language Processing tasks. Still, these models are opaque and poorly understandable for their users. Current approaches to explainability focus on token importance, in which the explanation consists of a set of tokens relevant to the prediction, and natural language explanations, in which the explanation is a generated piece of text. The latter are usually learned by design with models trained end-to-end to provide a prediction and an explanation, or rely on powerful external text generators to do the heavy lifting for them. In this paper we present TriplEX, an explainability algorithm for Transformer-based models fine-tuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks. TriplEX explains Transformers-based models by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation.
@inproceedings{SMM2022, author = {Setzu, Mattia and Monreale, Anna and Minervini, Pasquale}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00015}, line = {1,2}, month = dec, open_access = {NO}, pages = {44–53}, publisher = {IEEE}, title = {TRIPLEx: Triple Extraction for Explanation}, visible_on_website = {YES}, year = {2021} }
GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

4

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{SGM2021, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,4}, month = may, open_access = {Gold}, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, visible_on_website = {YES}, volume = {294}, year = {2021} }

2020

Global Explanations with Local Scoring

Mattia Setzu, Riccardo Guidotti, Anna Monreale, and Franco Turini

May 2020

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{SGM2019, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco}, booktitle = {Machine Learning and Knowledge Discovery in Databases}, doi = {10.1007/978-3-030-43823-4_14}, isbn = {9783030438234}, issn = {1865-0937}, line = {1}, open_access = {NO}, pages = {159–171}, publisher = {Springer International Publishing}, title = {Global Explanations with Local Scoring}, visible_on_website = {YES}, year = {2020} }

2025

Mathematical Foundation of Interpretable Equivariant Surrogate Models

Jacopo Joy Colombini, Filippo Bonchi, Francesco Giannini, Fosca Giannotti, Roberto Pellungrini, and 1 more author

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

This paper introduces a rigorous mathematical framework for neural network explainability, and more broadly for the explainability of equivariant operators called Group Equivariant Operators (GEOs), based on Group Equivariant Non-Expansive Operators (GENEOs) transformations. The central concept involves quantifying the distance between GEOs by measuring the non-commutativity of specific diagrams. Additionally, the paper proposes a definition of interpretability of GEOs according to a complexity measure that can be defined according to each user’s preferences. Moreover, we explore the formal properties of this framework and show how it can be applied in classical machine learning scenarios, like image classification with convolutional neural networks. (SpringerLink)
@inbook{CBG2025, address = {Istanbul, Turkey}, author = {Colombini, Jacopo Joy and Bonchi, Filippo and Giannini, Francesco and Giannotti, Fosca and Pellungrini, Roberto and Frosini, Patrizio}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_13}, isbn = {9783032083241}, issn = {1865-0937}, line = {1}, month = oct, open_access = {Gold}, pages = {294–318}, publisher = {Springer Nature Switzerland}, title = {Mathematical Foundation of Interpretable Equivariant Surrogate Models}, visible_on_website = {YES}, year = {2025} }
Ensemble Counterfactual Explanations for Churn Analysis

Samuele Tonati, Marzio Di Vece, Roberto Pellungrini, and Fosca Giannotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

Counterfactual explanations play a crucial role in interpreting and understanding the decision-making process of complex machine learning models, offering insights into why a particular prediction was made and how it could be altered. However, individual counterfactual explanations generated by different methods may vary significantly in terms of their quality, diversity, and coherence to the black-box prediction. This is especially important in financial applications such as churn analysis, where customer retention officers could explore different approaches and solutions with the clients to prevent churning. The officer’s capability to modify and explore different explanations is pivotal to his ability to provide feasible solutions. To address this challenge, we propose an evaluation framework through the implementation of an ensemble approach that combines state-of-the-art counterfactual generation methods and a linear combination score of desired properties to select the most appropriate explanation. We conduct our experiments on three publicly available churn datasets in different domains. Our experimental results demonstrate that the ensemble of counterfactual explanations provides more diverse and comprehensive insights into model behavior compared to individual methods alone that suffer from specific weaknesses. By aggregating, evaluating, and selecting multiple explanations, our approach enhances the diversity of the explanation, highlights common patterns, and mitigates the limitations of any single method, offering to the user the ability to tweak the explanation properties to their needs.
@inbook{TDP2025, author = {Tonati, Samuele and Di Vece, Marzio and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_21}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {332–347}, publisher = {Springer Nature Switzerland}, title = {Ensemble Counterfactual Explanations for Churn Analysis}, visible_on_website = {YES}, year = {2025} }

Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts

Andrea Pugnana, Riccardo Massidda, Francesco Giannini, Pietro Barbiero, Mateo Espinosa Zarlenga, and 4 more authors

Dec 2025

RESEARCH LINE

Bib

@misc{PMG2025,
  author = {Pugnana, Andrea and Massidda, Riccardo and Giannini, Francesco and Barbiero, Pietro and Zarlenga, Mateo Espinosa and Pellungrini, Roberto and Dominici, Gabriele and Giannotti, Fosca and Bacciu, Davide},
  line = {1,2},
  month = dec,
  title = {Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts},
  year = {2025}
}

2024

One-Shot Clustering for Federated Learning

Maciej Krzysztof Zuziak, Roberto Pellungrini, and Salvatore Rinzivillo

In 2024 IEEE International Conference on Big Data (BigData) , Dec 2024

RESEARCH LINE
1

Abs Bib HTML

Federated Learning (FL) is a widespread and well-adopted paradigm of decentralized learning that allows training one model from multiple sources without the need to directly transfer data between participating clients. Since its inception in 2015, it has been divided into numerous sub-fields that deal with application-specific issues, be it data heterogeneity or resource allocation. One such sub-field, Clustered Federated Learning (CFL), is dealing with the problem of clustering the population of clients into separate cohorts to deliver personalized models. Although few remarkable works have been published in this domain, the problem is still largely unexplored, as its basic assumption and settings are slightly different from standard FL. In this work, we present One-Shot Clustered Federated Learning (OCFL), a clustering-agnostic algorithm that can automatically detect the earliest suitable moment for clustering. Our algorithm is based on the computation of cosine similarity between gradients of the clients and a temperature measure that detects when the federated model starts to converge. We empirically evaluate our methodology by testing various one-shot clustering algorithms for over thirty different tasks on three benchmark datasets. Our experiments showcase the good performance of our approach when used to perform CFL in an automated manner without the need to adjust hyperparameters.
@inproceedings{ZPR2024, address = {Washington, DC, USA}, author = {Zuziak, Maciej Krzysztof and Pellungrini, Roberto and Rinzivillo, Salvatore}, booktitle = {2024 IEEE International Conference on Big Data (BigData)}, doi = {10.1109/bigdata62323.2024.10825763}, line = {1}, month = dec, open_access = {NO}, pages = {8108–8117}, publisher = {IEEE}, title = {One-Shot Clustering for Federated Learning}, visible_on_website = {YES}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma

Giulia Peduzzi, Alessio Felici, Roberto Pellungrini, Francesca Giorgolo, Riccardo Farinella, and 7 more authors

Digestive and Liver Disease, Jun 2024

RESEARCH LINE
5

Abs Bib HTML

Background: Current knowledge on pancreatic ductal adenocarcinoma (PDAC) risk factors is limited and no study has comprehensively tested the exposome alongside genetic variability for disease susceptibility. We analyzed 347 exposure variables and a polygenic risk score in UK Biobank data (816 PDAC cases, 302,645 controls). Fifty-two associations passed Bonferroni correction. Known risk factors such as smoking, pancreatitis, diabetes, heavy alcohol use and high BMI were confirmed. Novel associations include mobile phone usage intensity and multiple stress-related lifestyle factors. PRS was associated with PDAC risk but no gene‚Äìenvironment interactions were detected. Conclusion: Stressful lifestyle and sedentary behavior may play a major role in PDAC susceptibility independently of genetics.
@article{PFP2023, author = {Peduzzi, Giulia and Felici, Alessio and Pellungrini, Roberto and Giorgolo, Francesca and Farinella, Riccardo and Gentiluomo, Manuel and Spinelli, Andrea and Capurso, Gabriele and Monreale, Anna and Canzian, Federico and Calderisi, Marco and Campa, Daniele}, doi = {10.1016/j.dld.2023.10.015}, issn = {1590-8658}, journal = {Digestive and Liver Disease}, line = {5}, month = jun, number = {6}, open_access = {Gold}, pages = {1054–1063}, publisher = {Elsevier BV}, title = {Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma}, visible_on_website = {YES}, volume = {56}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }
Interpretable and Fair Mechanisms for Abstaining Classifiers

Daphne Lenders, Andrea Pugnana, Roberto Pellungrini, Toon Calders, Dino Pedreschi, and 1 more author

Dec 2024

RESEARCH LINE
1

5

Abs Bib HTML

Abstaining classifiers have the option to refrain from providing a prediction for instances that are difficult to classify. The abstention mechanism is designed to trade off the classifier‚Äôs performance on the accepted data while ensuring a minimum number of predictions. In this setting, fairness concerns often arise when the abstention mechanism reduces errors only for majority groups, increasing disparities across demographic groups. We introduce Interpretable and Fair Abstaining Classifier (IFAC), an algorithm that can reject predictions based on uncertainty and unfairness. By rejecting potentially unfair predictions, our method reduces disparities across groups of the non-rejected data. The unfairness-based rejections rely on interpretable rule-based fairness checks and situation testing, enabling transparent review and decision-making.
@misc{LPP2024, author = {Lenders, Daphne and Pugnana, Andrea and Pellungrini, Roberto and Calders, Toon and Pedreschi, Dino and Giannotti, Fosca}, doi = {[75,46,72,75,50,73,78]. }, line = {1,5}, month = dec, title = {Interpretable and Fair Mechanisms for Abstaining Classifiers}, year = {2024} }

2023

Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic

Clara Punzi, Aleksandra Maslennikova, Gizem Gezici, Roberto Pellungrini, and Fosca Giannotti

Dec 2023

RESEARCH LINE
1

4

Abs Bib HTML

Pandemic vaccination campaigns must account for vaccine skepticism as an obstacle to overcome. Using machine learning to identify behavioral and psychological patterns in public survey datasets can provide valuable insights and inform vaccination campaigns based on empirical evidence. However, we argue that the adoption of local and global explanation methodologies can provide additional support to health practitioners by suggesting personalized communication strategies and revealing potential demographic, social, or structural barriers to vaccination requiring systemic changes. In this paper, we first implement a chain classification model for the adoption of the vaccine during the H1N1 influenza outbreak taking seasonal vaccination information into account, and then compare it with a binary classifier for vaccination to better understand the overall patterns in the data. Following that, we derive and compare global explanations using post-hoc methodologies and interpretable-by-design models. Our findings indicate that socio-demographic factors play a distinct role in the H1N1 vaccination as compared to the general vaccination. Nevertheless, medical recommendation and health insurance remain significant factors for both vaccinations. Then, we concentrated on the subpopulation of individuals who did not receive an H1N1 vaccination despite being at risk of developing severe symptoms. In an effort to assist practitioners in providing effective recommendations to patients, we present rules and counterfactuals for the selected instances based on local explanations. Finally, we raise concerns regarding gender and racial disparities in healthcare access by analysing the interaction effects of sensitive attributes on the model’s output.
@inbook{PMG2023, author = {Punzi, Clara and Maslennikova, Aleksandra and Gezici, Gizem and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_31}, isbn = {9783031440670}, issn = {1865-0937}, line = {1,4}, open_access = {Gold}, pages = {621–635}, publisher = {Springer Nature Switzerland}, title = {Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic}, visible_on_website = {YES}, year = {2023} }
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories

Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda

Dec 2023

RESEARCH LINE
1

3

5

Abs Bib HTML

Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, identifying privacy risks is essential before deciding to release it publicly. Recent work has proposed using machine learning models for predicting privacy risk on raw mobility trajectories and using SHAP for risk explanation. However, applying SHAP to mobility data results in explanations of limited use both for privacy experts and end-users. In this work, we present EXPHLOT, a novel version of the Expert privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification to improve risk prediction while reducing computation time. We also devise an entropy-based mask to efficiently compute SHAP values and develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of privacy risk.
@inbook{NPR2023, author = {Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_22}, isbn = {9783031452758}, issn = {1611-3349}, line = {1,3,5}, open_access = {Gold}, pages = {325–340}, publisher = {Springer Nature Switzerland}, title = {EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories}, visible_on_website = {YES}, year = {2023} }

2020

Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks

Francesca Naretto, Roberto Pellungrini, Franco Maria Nardini, and Fosca Giannotti

Dec 2020

RESEARCH LINE
4

5

Abs Bib HTML

The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.
@inbook{NPN2020, author = {Naretto, Francesca and Pellungrini, Roberto and Nardini, Franco Maria and Giannotti, Fosca}, booktitle = {ECML PKDD 2020 Workshops}, doi = {10.1007/978-3-030-65965-3_34}, isbn = {9783030659653}, issn = {1865-0937}, line = {4,5}, open_access = {NO}, pages = {501–516}, publisher = {Springer International Publishing}, title = {Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks}, visible_on_website = {YES}, year = {2020} }
Predicting and Explaining Privacy Risk Exposure in Mobility Data

Francesca Naretto, Roberto Pellungrini, Anna Monreale, Franco Maria Nardini, and Mirco Musolesi

Dec 2020

RESEARCH LINE
4

5

Abs Bib HTML

Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.
@inbook{NPM2020, author = {Naretto, Francesca and Pellungrini, Roberto and Monreale, Anna and Nardini, Franco Maria and Musolesi, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-61527-7_27}, isbn = {9783030615277}, issn = {1611-3349}, line = {4,5}, open_access = {NO}, pages = {403–418}, publisher = {Springer International Publishing}, title = {Predicting and Explaining Privacy Risk Exposure in Mobility Data}, visible_on_website = {YES}, year = {2020} }

2025

MASCOTS: Model-Agnostic Symbolic COunterfactual Explanations for Time Series

Dawid Płudowski, Francesco Spinnato, Piotr Wilczyński, Krzysztof Kotowski, Evridiki Vasileia Ntagiou, and 2 more authors

Sep 2025

RESEARCH LINE
1

Abs Bib HTML

Counterfactual explanations provide an intuitive way to understand model decisions by identifying minimal changes required to alter an outcome. However, applying counterfactual methods to time series models remains challenging due to temporal dependencies, high dimensionality, and the lack of an intuitive human-interpretable representation. We introduce MASCOTS, a method that leverages the Bag-of-Receptive-Fields representation alongside symbolic transformations inspired by Symbolic Aggregate Approximation. By operating in a symbolic feature space, it enhances interpretability while preserving fidelity to the original data and model. Unlike existing approaches that either depend on model structure or autoencoder-based sampling, MASCOTS directly generates meaningful and diverse counterfactual observations in a model-agnostic manner, operating on both univariate and multivariate data. We evaluate MASCOTS on univariate and multivariate benchmark datasets, demonstrating comparable validity, proximity, and plausibility to state-of-the-art methods, while significantly improving interpretability and sparsity. Its symbolic nature allows for explanations that can be expressed visually, in natural language, or through semantic representations, making counterfactual reasoning more accessible and actionable.
@inbook{PSW2025, author = {Płudowski, Dawid and Spinnato, Francesco and Wilczyński, Piotr and Kotowski, Krzysztof and Ntagiou, Evridiki Vasileia and Guidotti, Riccardo and Biecek, Przemysław}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-032-06078-5_6}, isbn = {9783032060785}, issn = {1611-3349}, line = {1}, month = sep, open_access = {Gold}, pages = {94–112}, publisher = {Springer Nature Switzerland}, title = {MASCOTS: Model-Agnostic Symbolic COunterfactual Explanations for Time Series}, visible_on_website = {YES}, year = {2025} }
An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items

Luca Corbucci, Javier Alejandro Borges Legrottaglie, Francesco Spinnato, Anna Monreale, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

Accurately identifying items forgotten during a supermarket visit and providing clear, interpretable explanations for recommending them remains an underexplored problem within the Next Basket Prediction (NBP) domain. Existing NBP approaches typically only focus on forecasting future purchases, without explicitly addressing the detection of unintentionally omitted items. This gap is partly due to the scarcity of real-world datasets that allow for the reliable estimation of forgotten items. Furthermore, most current NBP methods rely on black-box models, which lack transparency and limit the ability to justify recommendations to end users. In this paper, we formally introduce the forgotten-item prediction task and propose two novel interpretable-by-design algorithms. These methods are tailored to identify forgotten items while offering intuitive, human-understandable explanations. Experiments on a real-world retail dataset show our algorithms outperform state-of-the-art NBP baselines by 10–15% across multiple evaluation metrics.
@inbook{CBS2025, author = {Corbucci, Luca and Borges Legrottaglie, Javier Alejandro and Spinnato, Francesco and Monreale, Anna and Guidotti, Riccardo}, booktitle = {ECAI 2025}, doi = {10.3233/faia250912}, isbn = {9781643686318}, issn = {1879-8314}, line = {1}, month = oct, open_access = {Gold}, publisher = {IOS Press}, title = {An Interpretable Data-Driven Unsupervised Approach for the Prevention of Forgotten Items}, visible_on_website = {YES}, year = {2025} }

2024

Enhancing Echo State Networks with Gradient-based Explainability Methods

Francesco Spinnato, Andrea Cossu, Riccardo Guidotti, Andrea Ceni, Claudio Gallicchio, and 1 more author

In ESANN 2024 proceesdings , Oct 2024

RESEARCH LINE
1

Abs Bib HTML

In sequence classification problems, the readout typically receives only the final state of the reservoir. However, averaging all states can be beneficial. In this work, we assess whether a weighted average of hidden states can enhance Echo State Network performance. We propose a gradient-based explainable technique to guide the contribution of each hidden state toward the final prediction. Our approach outperforms the naive average and other baselines in time series classification, particularly on noisy data.
@inproceedings{SCG2024, author = {Spinnato, Francesco and Cossu, Andrea and Guidotti, Riccardo and Ceni, Andrea and Gallicchio, Claudio and Bacciu, Davide}, booktitle = {ESANN 2024 proceesdings}, collection = {ESANN 2024}, doi = {10.14428/esann/2024.es2024-78}, line = {1}, open_access = {Gold}, pages = {17–22}, publisher = {Ciaco - i6doc.com}, series = {ESANN 2024}, title = {Enhancing Echo State Networks with Gradient-based Explainability Methods}, visible_on_website = {YES}, year = {2024} }
Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

IEEE Access, Oct 2024

RESEARCH LINE
1

Abs Bib HTML

The current trend in time series classification is to develop highly accurate algorithms by combining multiple models in ensemble hybrids, representing time series in complex feature spaces, and extracting features from different representations. As a consequence, the best time series classifiers are black-box models, not understandable for humans. Even the approaches regarded as interpretable, such as shapelet-based ones, rely on randomization to maintain efficiency, which poses challenges for interpretability. We propose the Bag-Of-Receptive-Fields (BORF), a fast, interpretable, and deterministic time series transform. Building on the Bag-Of-Patterns, we bridge the gap between convolutional operators and discretization, enhancing the Symbolic Aggregate Approximation with dilation and stride to better capture temporal patterns at multiple scales. We propose an algorithmic speedup that reduces the time complexity associated with SAX-based classifiers, enabling the more flexible BORF, represented as a sparse multivariate tensor.
@article{SGM2024, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, doi = {10.1109/access.2024.3464743}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {137893–137912}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields}, visible_on_website = {YES}, volume = {12}, year = {2024} }
Drifting explanations in continual learning

Andrea Cossu, Francesco Spinnato, Riccardo Guidotti, and Davide Bacciu

Neurocomputing, Sep 2024

RESEARCH LINE
1

Abs Bib HTML

Continual Learning (CL) trains models on streams of data, with the aim of learning new information without forgetting previous knowledge. We study the behavior of different explanation methods in CL and propose CLEX (ContinuaL EXplanations), an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios, where forgetting is pronounced. We observed that models with similar predictive accuracy do not generate similar explanations. (openportal.isti.cnr.it)
@article{CSG2024, author = {Cossu, Andrea and Spinnato, Francesco and Guidotti, Riccardo and Bacciu, Davide}, doi = {10.1016/j.neucom.2024.127960}, issn = {0925-2312}, journal = {Neurocomputing}, line = {1}, month = sep, open_access = {Gold}, pages = {127960}, publisher = {Elsevier BV}, title = {Drifting explanations in continual learning}, visible_on_website = {YES}, volume = {597}, year = {2024} }

2023

Text to Time Series Representations: Towards Interpretable Predictive Models

Mattia Poggioli, Francesco Spinnato, and Riccardo Guidotti

Sep 2023

RESEARCH LINE
1

Abs Bib HTML

In this paper, we investigate the impact of converting text observations into time series observations to solve interpretable text classification through time series representations. By considering temporal dependencies, TSA can be used for various purposes, such as descriptive analysis, clustering, classification, and forecasting. We propose using shapelets in NLP by turning texts into time series. To perform this transformation, we design and implement TOTS, a framework to turn text to time series. TOTS exploits a range of different conversion alternatives for tokenization, feature extraction and aggregation. We formalize our Text to Time Series framework as a feature extraction and aggregation process, proposing a set of different conversion alternatives for each step. We then exploit interpretable models originally developed for time series (e.g., shapelet-based models) as interpretable text classifiers. Our experiments show promising results for classification and interpretability.
@inbook{PSG2023, author = {Poggioli, Mattia and Spinnato, Francesco and Guidotti, Riccardo}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_16}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {230–245}, publisher = {Springer Nature Switzerland}, title = {Text to Time Series Representations: Towards Interpretable Predictive Models}, visible_on_website = {YES}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.”
@article{SGM2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, open_access = {Gold}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, visible_on_website = {YES}, volume = {18}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{LSG2023, address = {Cham, Switzerland}, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Advances in Intelligent Data Analysis XXI}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, visible_on_website = {YES}, year = {2023} }
Modeling Events and Interactions through Temporal Processes – A Survey

Liguori Angelica, Caroprese Luciano, Minici Marco, Veloso Bruno, Spinnato Francesco, and 3 more authors

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

In real-world scenario, many phenomena produce a collection of events that occur in continuous time. Point Processes provide a natural mathematical framework for modeling these sequences of events. In this survey, we investigate probabilistic models for modeling event sequences through temporal processes. We revise the notion of event modeling and provide the mathematical foundations that characterize the literature on the topic. We define an ontology to categorize the existing approaches in terms of three families: simple, marked, and spatio-temporal point processes. For each family, we systematically review the existing approaches based based on deep learning. Finally, we analyze the scenarios where the proposed techniques can be used for addressing prediction and modeling aspects.
@misc{LCM2023, author = {Angelica, Liguori and Luciano, Caroprese and Marco, Minici and Bruno, Veloso and Francesco, Spinnato and Mirco, Nanni and Giuseppe, Manco and Joao, Gama}, doi = {10.48550/ARXIV.2303.06067}, line = {1}, month = dec, publisher = {Arxiv}, title = {Modeling Events and Interactions through Temporal Processes -- A Survey}, year = {2023} }

2022

Explaining Crash Predictions on Multivariate Time Series Data

Francesco Spinnato, Riccardo Guidotti, Mirco Nanni, Daniele Maccagnola, Giulia Paciello, and 1 more author

Dec 2022

RESEARCH LINE
4

Abs Bib HTML

In Assicurazioni Generali, an automatic decision-making model is used to check real-time multivariate time series and alert if a car crash happened. In such a way, a Generali operator can call the customer to provide first assistance. The high sensitivity of the model used, combined with the fact that the model is not interpretable, might cause the operator to call customers even though a car crash did not happen but only due to a harsh deviation or the fact that the road is bumpy. Our goal is to tackle the problem of interpretability for car crash prediction and propose an eXplainable Artificial Intelligence (XAI) workflow that allows gaining insights regarding the logic behind the deep learning predictive model adopted by Generali. We reach our goal by building an interpretable alternative to the current obscure model that also reduces the training data usage and the prediction time.
@inbook{SGN2022, address = {Cham, Switzerland}, author = {Spinnato, Francesco and Guidotti, Riccardo and Nanni, Mirco and Maccagnola, Daniele and Paciello, Giulia and Farina, Antonio Bencini}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_39}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {556–566}, publisher = {Springer Nature Switzerland}, title = {Explaining Crash Predictions on Multivariate Time Series Data}, visible_on_website = {YES}, year = {2022} }
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions

Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti

IEEE Access, Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Time series data is increasingly used in a wide range of fields, and it is often relied on in crucial applications and high-stakes decision-making. For instance, sensors generate time series data to recognize different types of anomalies through automatic decision-making systems. Typically, these systems are realized with machine learning models that achieve top-tier performance on time series classification tasks. Unfortunately, the logic behind their prediction is opaque and hard to understand from a human standpoint. Recently, we observed a consistent increase in the development of explanation methods for time series classification justifying the need to structure and review the field. In this work, we (a) present the first extensive literature review on Explainable AI (XAI) for time series classification, (b) categorize the research field through a taxonomy subdividing the methods into time points-based, subsequences-based and instance-based, and (c) identify open research directions regarding the type of explanations and the evaluation of explanations and interpretability.
@article{TSS2022, address = { New York, USA}, author = {Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo}, doi = {10.1109/access.2022.3207765}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {100700–100724}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions}, visible_on_website = {YES}, volume = {10}, year = {2022} }

2020

Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{GMS2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, open_access = {NO}, pages = {167–176}, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, visible_on_website = {YES}, year = {2020} }

2025

Differentially Private FastSHAP for Federated Learning Model Explainability

Valerio Bonsignori, Luca Corbucci, Francesca Naretto, and Anna Monreale

In 2025 International Joint Conference on Neural Networks (IJCNN) , Jun 2025

RESEARCH LINE
1

5

Abs Bib HTML

Explaining the reasoning behind black-box model predictions while preserving user privacy is a significant challenge. This becomes even more complex in Federated Learning, where legal constraints restrict the data that clients can share with external entities. In this paper, we introduce Fastshap++, a method that adapts Fastshap to explain Federated Learning trained models. Unlike existing approaches, Fastshap++ mitigates client privacy risks by incorporating Differential Privacy into the explanation process and preventing the exchange of sensitive information between clients and external entities. We evaluate the effectiveness of Fastshap++ testing it on three different datasets, and comparing the explanations with those produced by a centralized explainer with access to clients’ training data. Lastly, we study the impact of varying levels of Differential Privacy to analyse the trade-offs between privacy and the quality of the explanations.
@inproceedings{BCN2025, address = {Rome, Italy}, author = {Bonsignori, Valerio and Corbucci, Luca and Naretto, Francesca and Monreale, Anna}, booktitle = {2025 International Joint Conference on Neural Networks (IJCNN)}, doi = {10.1109/ijcnn64981.2025.11227553}, line = {1,5}, month = jun, pages = {1–8}, publisher = {IEEE}, title = {Differentially Private FastSHAP for Federated Learning Model Explainability}, visible_on_website = {YES}, year = {2025} }
Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Dec 2025

RESEARCH LINE
1

5

Abs Bib HTML

During the last few years, the abundance of data has significantly boosted the performance of Machine Learning models, integrating them into several aspects of daily life. However, the rise of powerful Artificial Intelligence tools has introduced ethical and legal complexities. This paper proposes a computational framework to analyze the ethical and legal dimensions of Machine Learning models, focusing specifically on privacy concerns and interpretability. In fact, recently, the research community proposed privacy attacks able to reveal whether a record was part of the black-box training set or inferring variable values by accessing and querying a Machine Learning model. These attacks highlight privacy vulnerabilities and prove that GDPR regulation might be violated by making data or Machine Learning models accessible. At the same time, the complexity of these models, often labelled as “black-boxes”, has made the development of explanation methods indispensable to enhance trust and facilitate their acceptance and adoption in high-stake scenarios.
@misc{NMG2025, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, line = {1,5}, month = dec, publisher = {Trans. Data Priv. 18 (2), 67-93}, title = {Evaluating the Privacy Exposure of Interpretable Global and Local Explainers.}, year = {2025} }

2024

GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning

Rami Haffar, Francesca Naretto, David Sánchez, Anna Monreale, and Josep Domingo-Ferrer

In 2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE) , Jun 2024

RESEARCH LINE
1

2

Abs Bib HTML

The increasing spread of artificial intelligence applications has led to decentralized frameworks that foster collaborative model training among multiple entities. One of such frameworks is federated learning, which ensures data availability in client nodes without requiring the central server to retain any data. Nevertheless, similar to centralized neural networks, interpretability remains a challenge in understanding the predictions of these decentralized frameworks. The limited access to data on the server side further complicates the applicability of explainers in such frameworks. To address this challenge, we propose GLOR-FLEX, a framework designed to generate rule-based global explanations from local explainers. GLOR-FLEX ensures client privacy by preventing the sharing of actual data between the clients and the server. The proposed framework initiates the process by constructing local decision trees on each client’s side to produce local explanations. Subsequently, by using rule extraction from these trees and strategically sorting and merging those rules, the server obtains a merged set of rules suitable to be used as a global explainer. We empirically evaluate the performance of GLOR-FLEX on three distinct tabular data sets, showing high fidelity scores between the explainers and both the local and global models. Our results support the effectiveness of GLOR-FLEX in generating accurate explanations that efficiently detect and explain the behavior of both local and global models.
@inproceedings{HNS2024, author = {Haffar, Rami and Naretto, Francesca and Sánchez, David and Monreale, Anna and Domingo-Ferrer, Josep}, booktitle = {2024 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE)}, doi = {10.1109/fuzz-ieee60900.2024.10611878}, line = {1,2}, month = jun, open_access = {NO}, pages = {1–9}, publisher = {IEEE}, title = {GLOR-FLEX: Local to Global Rule-Based EXplanations for Federated Learning}, visible_on_website = {YES}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories

Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda

Jun 2023

RESEARCH LINE
1

3

5

Abs Bib HTML

Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, identifying privacy risks is essential before deciding to release it publicly. Recent work has proposed using machine learning models for predicting privacy risk on raw mobility trajectories and using SHAP for risk explanation. However, applying SHAP to mobility data results in explanations of limited use both for privacy experts and end-users. In this work, we present EXPHLOT, a novel version of the Expert privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification to improve risk prediction while reducing computation time. We also devise an entropy-based mask to efficiently compute SHAP values and develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of privacy risk.
@inbook{NPR2023, author = {Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_22}, isbn = {9783031452758}, issn = {1611-3349}, line = {1,3,5}, open_access = {Gold}, pages = {325–340}, publisher = {Springer Nature Switzerland}, title = {EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories}, visible_on_website = {YES}, year = {2023} }

2022

Privacy Risk of Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Sep 2022

RESEARCH LINE
5

Abs Bib HTML

In this paper we propose to study a methodology that enables the evaluation of the privacy risk exposure of global explainers based on an interpretable classifier that imitates the global reasoning of a black-box classifier. The idea is to verify if the layer of interpretability added by the interpretable model can jeopardize the privacy protection of the training data used for learning the black-box classifier. In order to address this problem, we exploit a well-known attack model called membership inference attack (MIA). We then compute the privacy risk change ΔR due to the introduction of the global explainer c. The preliminary experimental results suggest that global explainers based on decision trees introduce a higher risk of privacy, increasing the percentage of records identified as members of the training dataset used to train the original black-box classifiers. These results suggest that in order to provide Trustworthy AI, it becomes fundamental to consider the relationship between different ethical values to identify possible values like transparency and privacy that may be in contrast, and studying solutions that enable the simultaneous satisfaction of more than one value.
@inbook{NMG2022, address = {Amsterdam, the Netherlands}, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220206}, issn = {1879-8314}, line = {5}, month = sep, open_access = {Gold}, pages = {249 - 251}, publisher = {IOS Press}, title = {Privacy Risk of Global Explainers}, visible_on_website = {YES}, year = {2022} }
Evaluating the Privacy Exposure of Interpretable Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

In 2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2022

RESEARCH LINE
5

Abs Bib HTML

In recent years we are witnessing the diffusion of AI systems based on powerful Machine Learning models which find application in many critical contexts such as medicine, financial market and credit scoring. In such a context it is particularly important to design Trustworthy AI systems while guaranteeing transparency, with respect to their decision reasoning and privacy protection. Although many works in the literature addressed the lack of transparency and the risk of privacy exposure of Machine Learning models, the privacy risks of explainers have not been appropriately studied. This paper presents a methodology for evaluating the privacy exposure raised by interpretable global explainers able to imitate the original black-box classifier. Our methodology exploits the well-known Membership Inference Attack. The experimental results highlight that global explainers based on interpretable trees lead to an increase in privacy exposure.
@inproceedings{NMG2022b, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi56440.2022.00012}, line = {5}, month = dec, open_access = {NO}, pages = {13–19}, publisher = {IEEE}, title = {Evaluating the Privacy Exposure of Interpretable Global Explainers}, visible_on_website = {YES}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Nov 2022

RESEARCH LINE
1

2

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{GMR2022, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1007/s10618-022-00878-5}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,2}, month = nov, number = {5}, open_access = {Gold}, pages = {2825–2862}, publisher = {Springer Science and Business Media LLC}, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, visible_on_website = {YES}, volume = {38}, year = {2022} }

2020

Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks

Francesca Naretto, Roberto Pellungrini, Franco Maria Nardini, and Fosca Giannotti

Nov 2020

RESEARCH LINE
4

5

Abs Bib HTML

The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.
@inbook{NPN2020, author = {Naretto, Francesca and Pellungrini, Roberto and Nardini, Franco Maria and Giannotti, Fosca}, booktitle = {ECML PKDD 2020 Workshops}, doi = {10.1007/978-3-030-65965-3_34}, isbn = {9783030659653}, issn = {1865-0937}, line = {4,5}, open_access = {NO}, pages = {501–516}, publisher = {Springer International Publishing}, title = {Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks}, visible_on_website = {YES}, year = {2020} }
Predicting and Explaining Privacy Risk Exposure in Mobility Data

Francesca Naretto, Roberto Pellungrini, Anna Monreale, Franco Maria Nardini, and Mirco Musolesi

Nov 2020

RESEARCH LINE
4

5

Abs Bib HTML

Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.
@inbook{NPM2020, author = {Naretto, Francesca and Pellungrini, Roberto and Monreale, Anna and Nardini, Franco Maria and Musolesi, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-030-61527-7_27}, isbn = {9783030615277}, issn = {1611-3349}, line = {4,5}, open_access = {NO}, pages = {403–418}, publisher = {Springer International Publishing}, title = {Predicting and Explaining Privacy Risk Exposure in Mobility Data}, visible_on_website = {YES}, year = {2020} }

2024

Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space

Simone Piaggesi, Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

IEEE Access, Nov 2024

RESEARCH LINE
1

Abs Bib HTML

We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities similarly to well-known dimensionality reduction techniques. Our approach introduces a transparent latent space optimized for interpretability of both counterfactual and prototypical explanations for tabular data. The approach enables the easy extraction of local and global explanations and ensures that the latent space preserves similarity relations, enabling meaningful prototypical and counterfactual examples for any classifier.
@article{PBG2024, author = {Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1109/access.2024.3496114}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {168983–169000}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space}, visible_on_website = {YES}, volume = {12}, year = {2024} }

2023

Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{BGG2023, address = {Netherlands}, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1,3}, month = jun, number = {5}, open_access = {Gold}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, visible_on_website = {YES}, volume = {37}, year = {2023} }

2022

Interpretable Latent Space to Enable Counterfactual Explanations

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Jun 2022

RESEARCH LINE
1

Abs Bib HTML

Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{BGG2023c, address = {Montpellier, France}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_37}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {525–540}, publisher = {Springer Nature Switzerland}, title = {Interpretable Latent Space to Enable Counterfactual Explanations}, visible_on_website = {YES}, year = {2022} }
Transparent Latent Space Counterfactual Explanations for Tabular Data

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{BGG2023b, address = {Shenzhen, China}, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)}, doi = {10.1109/dsaa54385.2022.10032407}, line = {1}, month = oct, open_access = {NO}, pages = {1–10}, publisher = {IEEE}, title = {Transparent Latent Space Counterfactual Explanations for Tabular Data}, visible_on_website = {YES}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Bodria, Francesco; Rinzivillo, Salvatore; Fadda, Daniele; Guidotti, Riccardo; Giannotti, and 2 more authors

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@misc{BRF2022, author = {Bodria and Rinzivillo, Francesco; and Fadda, Salvatore; and Guidotti, Daniele; and Giannotti, Riccardo; and Pedreschi, Fosca; and Dino}, doi = {10.2312/evs.20221098}, line = {1,3}, month = dec, title = {Explaining Black Box with Visual Exploration of Latent Space}, year = {2022} }

2025

Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model

Fabio Michele Russo, Carlo Metta, Anna Monreale, Salvatore Rinzivillo, and Fabio Pinelli

Dec 2025

RESEARCH LINE
1

Abs Bib HTML

As predictive machine learning models become increasingly adopted and advanced, their role has evolved from merely predicting outcomes to actively shaping them. This evolution has underscored the importance of Trustworthy AI, highlighting the necessity to extend our focus beyond mere accuracy and toward a comprehensive understanding of these models’ behaviors within the specific contexts of their applications. To further progress in explainability, we introduce poem, Prefetched Offline Explanation Model, a model-agnostic, local explainability algorithm for image data. The algorithm generates exemplars, counterexemplars and saliency maps to provide quick and effective explanations suitable for time-sensitive scenarios. Leveraging an existing local algorithm, poem infers factual and counterfactual rules from data to create illustrative examples and opposite scenarios with an enhanced stability by design. A novel mechanism then matches incoming test points with an explanation base and produces diverse exemplars, informative saliency maps and believable counterexemplars. Experimental results indicate that poem outperforms its predecessor abele in speed and ability to generate more nuanced and varied exemplars alongside more insightful saliency maps and valuable counterexemplars.
@inbook{RMM2025, author = {Russo, Fabio Michele and Metta, Carlo and Monreale, Anna and Rinzivillo, Salvatore and Pinelli, Fabio}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_11}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {167–182}, publisher = {Springer Nature Switzerland}, title = {Explainable AI in Time-Sensitive Scenarios: Prefetched Offline Explanation Model}, visible_on_website = {YES}, year = {2025} }

2024

Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }

2023

Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }

2022

Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

Sep 2022

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing models allowing interaction between decision systems and humans by making the decisions understandable. We propose a case study for skin lesion diagnosis showing how it is possible to provide explanations of the decisions of deep neural network trained to label skin lesions.
@inbook{MGY2021b, address = {Amsterdam, the Netherlands}, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {HHAI2022: Augmenting Human Intellect}, doi = {10.3233/faia220209}, issn = {1879-8314}, line = {1}, month = sep, open_access = {NO}, pages = {258 - 260}, publisher = {IOS Press}, title = {Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers}, visible_on_website = {YES}, year = {2022} }

2021

Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

In 2021 IEEE Symposium on Computers and Communications (ISCC) , Sep 2021

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing mechanisms allowing for an interaction between decision systems and humans by making the decisions of the formers understandable. This is particularly important in sensitive contexts like in the medical domain. We propose a use case study, for skin lesion diagnosis, illustrating how it is possible to provide the practitioner with explanations on the decisions of a state of the art deep neural network classifier trained to characterize skin lesions from examples. Our framework consists of a trained classifier onto which an explanation module operates. The latter is able to offer the practitioner exemplars and counterexemplars for the classification diagnosis thus allowing the physician to interact with the automatic diagnosis system. The exemplars are generated via an adversarial autoencoder. We illustrate the behavior of the system on representative examples.
@inproceedings{MGY2021, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {2021 IEEE Symposium on Computers and Communications (ISCC)}, doi = {10.1109/iscc53001.2021.9631485}, line = {1}, month = sep, open_access = {NO}, pages = {1–7}, publisher = {IEEE}, title = {Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling}, visible_on_website = {YES}, year = {2021} }

2024

An Interactive Interface for Feature Space Navigation

Eleonora Cappuccio, Isacco Beretta, Marta Marchiori Manerba, and Salvatore Rinzivillo

Jun 2024

RESEARCH LINE
3

Abs Bib HTML

In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to provide users with an intuitive and direct way to navigate through the feature space, inspect model behavior, and perform what-if analyses via feature manipulations and visual feedback. We integrate multiple views including projections of high-dimensional data, decision boundary surfaces, and sensitivity indicators. The interface also supports real-time adjustments of feature values to observe the corresponding changes in the model predictions. Our experiments show that the system can help both novice and expert users to detect regions of uncertainty, identify influential features, and generate hypotheses for model improvement.
@inbook{CBM2024, author = {Cappuccio, Eleonora and Beretta, Isacco and Marchiori Manerba, Marta and Rinzivillo, Salvatore}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240184}, isbn = {9781643685229}, issn = {1879-8314}, line = {3}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {An Interactive Interface for Feature Space Navigation}, visible_on_website = {YES}, year = {2024} }
Exploring Large Language Models Capabilities to Explain Decision Trees

Paulo Bruno Serafim, Pierluigi Crescenzi, Gizem Gezici, Eleonora Cappuccio, Salvatore Rinzivillo, and 1 more author

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are widely adopted in Machine Learning tasks due to their operation simplicity and interpretability aspects. However, following the decision process path taken by trees can be difficult in a complex scenario or in a case where a user has no familiarity with them. Prior research showed that converting outcomes to natural language is an accessible way to facilitate understanding for non-expert users in several tasks. More recently, there has been a growing effort to use Large Language Models (LLMs) as a tool for providing natural language texts. In this paper, we examine the proficiency of LLMs to explain decision tree predictions in simple terms through the generation of natural language explanations. By exploring different textual representations and prompt engineering strategies, we identify capabilities that strengthen LLMs as a competent explainer as well as highlight potential challenges and limitations, opening further research possibilities on natural language explanations for decision trees.
@inbook{SGC2024, author = {Serafim, Paulo Bruno and Crescenzi, Pierluigi and Gezici, Gizem and Cappuccio, Eleonora and Rinzivillo, Salvatore and Giannotti, Fosca}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240183}, isbn = {9781643685229}, issn = {1879-8314}, line = {1}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {Exploring Large Language Models Capabilities to Explain Decision Trees}, visible_on_website = {YES}, year = {2024} }
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers

Eleonora Cappuccio, Benedetta Muscato, Laura Pollacci, Marta Marchiori Manerba, Clara Punzi, and 5 more authors

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
@misc{CMP2024, address = {Aachen, Germany}, author = {Cappuccio, Eleonora and Muscato, Benedetta and Pollacci, Laura and Manerba, Marta Marchiori and Punzi, Clara and Mala, Chandana Sree and Lalli, Margherita and Gezici, Gizem and Natilli, Michela and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers}, year = {2024} }

2023

Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations

Eleonora Cappuccio, Daniele Fadda, Rosa Lanzilotti, and Salvatore Rinzivillo

In Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter , Sep 2023

RESEARCH LINE
1

2

3

Abs Bib HTML

The Human-Computer Interaction (HCI) community has long stressed the need for a more user-centered approach to Explainable Artificial Intelligence (XAI), a research area that aims at defining algorithms and tools to illustrate the predictions of the so-called black-box models. This approach can benefit from the fields of user-interface, user experience, and visual analytics. In this demo, we propose a visual-based tool, "F.I.P.E.R.", that shows interactive explanations combining rules and feature importance.
@inproceedings{CFR2023, author = {Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore}, booktitle = {Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter}, collection = {CHItaly 2023}, doi = {10.1145/3605390.3610811}, line = {1,2,3}, month = sep, open_access = {NO}, pages = {1–4}, publisher = {ACM}, series = {CHItaly 2023}, title = {Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations}, visible_on_website = {YES}, year = {2023} }

2022

User-driven counterfactual generator: a human centered exploration

Beretta I; Cappuccio E; Marchiori Manerba M

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

In this paper, we critically examine the limitations of the techno-solutionist approach to explanations in the context of counterfactual generation, reaffirming interactivity as a core value in the explanation interface between the model and the user.
@misc{BCM2022, author = {M, Beretta I; Cappuccio E; Marchiori Manerba}, line = {1,3}, month = dec, title = {User-driven counterfactual generator: a human centered exploration}, year = {2022} }

2024

A Frank System for Co-Evolutionary Hybrid Decision-Making

Federico Mazzoni, Riccardo Guidotti, and Alessio Malizia

Dec 2024

RESEARCH LINE
1

3

4

Abs Bib HTML

Hybrid decision-making systems combine human judgment with algorithmic recommendations, yet coordinating these two sources of information remains challenging. We present FRANK, a co-evolutionary framework enabling humans and AI agents to iteratively exchange feedback and refine decisions over time. FRANK integrates rule-based reasoning, preference modeling, and a learning module that adapts recommendations based on user interaction. Through simulated and real-user experiments, we show that the co-evolution process helps users converge toward more stable and accurate decisions while increasing perceived transparency. The system allows humans to override or modify machine suggestions while the AI agent reshapes its internal models in response to human rationale. FRANK thus promotes a collaborative decision environment where human expertise and machine learning strengthen each other.
@inbook{MBP2024, author = {Mazzoni, Federico and Guidotti, Riccardo and Malizia, Alessio}, booktitle = {Advances in Intelligent Data Analysis XXII}, doi = {10.1007/978-3-031-58553-1_19}, isbn = {9783031585531}, issn = {1611-3349}, line = {1,3,4}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {A Frank System for Co-Evolutionary Hybrid Decision-Making}, visible_on_website = {YES}, year = {2024} }

2025

A Practical Approach to Causal Inference over Time

Martina Cinquini, Isacco Beretta, Salvatore Ruggieri, and Isabel Valera

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

RESEARCH LINE
2

Abs Bib HTML

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting causal VAR framework allows us to perform causal inference over time from observational time series data. Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.
@article{CBR2025, author = {Cinquini, Martina and Beretta, Isacco and Ruggieri, Salvatore and Valera, Isabel}, doi = {10.1609/aaai.v39i14.33626}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {2}, month = apr, number = {14}, open_access = {Gold}, pages = {14832–14839}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {A Practical Approach to Causal Inference over Time}, visible_on_website = {YES}, volume = {39}, year = {2025} }
The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR

Laura State, Alejandra Bringas Colmenarejo, Andrea Beretta, Salvatore Ruggieri, Franco Turini, and 1 more author

Artificial Intelligence and Law, Jan 2025

RESEARCH LINE
4

Abs Bib HTML

Explainable AI (XAI) provides methods to understand non-interpretable machine learning models. However, we have little knowledge about what legal experts expect from these explanations, including their legal compliance with, and value against European Union legislation. To close this gap, we present the Explanation Dialogues, an expert focus study to uncover the expectations, reasoning, and understanding of legal experts and practitioners towards XAI, with a specific focus on the European General Data Protection Regulation. The study consists of an online questionnaire and follow-up interviews, and is centered around a use-case in the credit domain. We extract both a set of hierarchical and interconnected codes using grounded theory, and present the standpoints of the participating experts towards XAI. We find that the presented explanations are hard to understand and lack information, and discuss issues that can arise from the different interests of the data controller and subject. Finally, we present a set of recommendations for developers of XAI methods, and indications of legal areas of discussion. Among others, recommendations address the presentation, choice, and content of an explanation, technical risks as well as the end-user, while we provide legal pointers to the contestability of explanations, transparency thresholds, intellectual property rights as well as the relationship between involved parties.
@article{SBB2025, author = {State, Laura and Bringas Colmenarejo, Alejandra and Beretta, Andrea and Ruggieri, Salvatore and Turini, Franco and Law, Stephanie}, doi = {10.1007/s10506-024-09430-w}, issn = {1572-8382}, journal = {Artificial Intelligence and Law}, line = {4}, month = jan, open_access = {Green}, publisher = {Springer Science and Business Media LLC}, title = {The explanation dialogues: an expert focus study to understand requirements towards explanations within the GDPR}, visible_on_website = {YES}, year = {2025} }

2024

An Interactive Interface for Feature Space Navigation

Eleonora Cappuccio, Isacco Beretta, Marta Marchiori Manerba, and Salvatore Rinzivillo

Jun 2024

RESEARCH LINE
3

Abs Bib HTML

In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to provide users with an intuitive and direct way to navigate through the feature space, inspect model behavior, and perform what-if analyses via feature manipulations and visual feedback. We integrate multiple views including projections of high-dimensional data, decision boundary surfaces, and sensitivity indicators. The interface also supports real-time adjustments of feature values to observe the corresponding changes in the model predictions. Our experiments show that the system can help both novice and expert users to detect regions of uncertainty, identify influential features, and generate hypotheses for model improvement.
@inbook{CBM2024, author = {Cappuccio, Eleonora and Beretta, Isacco and Marchiori Manerba, Marta and Rinzivillo, Salvatore}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240184}, isbn = {9781643685229}, issn = {1879-8314}, line = {3}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {An Interactive Interface for Feature Space Navigation}, visible_on_website = {YES}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Local explanation methods, such as SHAP and LIME, are increasingly adopted to justify predictions of clinical decision support systems. However, their reliability and clinical usefulness remain limited by instability, lack of contextualization, and poor alignment with medical reasoning. In this work, we propose an enhanced pipeline for generating trustworthy local explanations in healthcare. Our approach incorporates domain constraints, medical ontologies, and temporal reasoning over patient histories. We evaluate the method on multiple clinical prediction tasks and compare it against standard explainability tools using expert-driven criteria. Results show that explanations become more stable and more aligned with clinically plausible factors. A qualitative analysis with clinicians further indicates improved interpretability and actionability, supporting safer and more transparent AI-assisted healthcare.
@article{MBP2024b, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/bioengineering11040369}, issn = {2306-5354}, journal = {Bioengineering}, line = {1}, month = apr, number = {4}, open_access = {Gold}, pages = {369}, publisher = {MDPI AG}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, visible_on_website = {YES}, volume = {11}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Apr 2024

RESEARCH LINE
1

2

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{MBG2024, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {1,2}, month = apr, number = {7}, open_access = {Gold}, pages = {753}, publisher = {MDPI AG}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, visible_on_website = {YES}, volume = {14}, year = {2024} }
Mapping the landscape of ethical considerations in explainable AI research

Luca Nannini, Marta Marchiori Manerba, and Isacco Beretta

Ethics and Information Technology, Jun 2024

RESEARCH LINE
1

5

Abs Bib HTML

With its potential to contribute to the ethical governance of AI, explainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet the substantiation of these claims with rigorous ethical analysis and reflection remains largely unexamined. This contribution endeavours to scrutinise the relationship between XAI and ethical considerations. We conduct a systematic review of the XAI literature by applying a multi-stage filtering process and then developing a taxonomy for classifying the depth and quality of ethical engagement in the field. Our findings show that although a growing body of XAI research references ethical issues, the majority does so at a superficial level. We identify nine ethical themes (e.g., autonomy, justice, transparency) and show how they are engaged in the literature. We discuss trends, deficits, and future directions for integrating ethical considerations meaningfully into XAI research.
@article{NMB2024, author = {Nannini, Luca and Marchiori Manerba, Marta and Beretta, Isacco}, doi = {10.1007/s10676-024-09773-7}, issn = {1572-8439}, journal = {Ethics and Information Technology}, line = {1,5}, month = jun, number = {3}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Mapping the landscape of ethical considerations in explainable AI research}, visible_on_website = {YES}, volume = {26}, year = {2024} }
Requirements of eXplainable AI in Algorithmic Hiring

A. Beretta, G. Ercoli, A. Ferraro, R. Guidotti, A. Iommi, and 4 more authors

Dec 2024

RESEARCH LINE
1

Abs Bib HTML

AI models for ranking candidates to a job position are increasingly adopted. They bring a new layer of opaqueness in the way candidates are evaluated. We present preliminary research on stakeholder analysis and requirement elicitation for designing an explainability component in AI models for ranking candidates to a job position. (CEUR-WS)
@misc{BEF2024, author = {Beretta, A. and Ercoli, G. and Ferraro, A. and Guidotti, R. and Iommi, A. and Mastropietro, A. and Monreale, A. and Rotelli, D. and Ruggieri, S.}, line = {1}, month = dec, title = {Requirements of eXplainable AI in Algorithmic Hiring}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }

2023

Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{MBG2023, address = {Berlin/Heidelberg, Germany}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3}, month = jun, number = {1}, open_access = {Gold}, pages = {183–195}, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, visible_on_website = {YES}, volume = {20}, year = {2023} }
Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
The Importance of Time in Causal Algorithmic Recourse

Isacco Beretta, and Martina Cinquini

Jun 2023

RESEARCH LINE
2

Abs Bib HTML

The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inbook{BC2023, author = {Beretta, Isacco and Cinquini, Martina}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_16}, isbn = {9783031440649}, issn = {1865-0937}, line = {2}, open_access = {NO}, pages = {283–298}, publisher = {Springer Nature Switzerland}, title = {The Importance of Time in Causal Algorithmic Recourse}, visible_on_website = {YES}, year = {2023} }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Transactions on Interactive Intelligent Systems, Dec 2023

RESEARCH LINE
1

3

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{PBF2023, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, doi = {10.1145/3587271}, issn = {2160-6463}, journal = {ACM Transactions on Interactive Intelligent Systems}, line = {1,3}, month = dec, number = {4}, open_access = {Gold}, pages = {1–35}, publisher = {Association for Computing Machinery (ACM)}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, visible_on_website = {YES}, volume = {13}, year = {2023} }

2022

Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{PBP2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {4}, month = apr, open_access = {Gold}, pages = {1–9}, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, visible_on_website = {YES}, year = {2022} }

Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients

Himanshi Allahabadi, Julia Amann, Isabelle Balot, Andrea Beretta, Charles Binkley, and 52 more authors

IEEE Transactions on Technology and Society, Dec 2022

RESEARCH LINE

5

Abs Bib HTML

This article’s main contributions are twofold: 1) to demonstrate how to apply the general European Union’s High-Level Expert Group’s (EU HLEG) guidelines for trustworthy AI in practice for the domain of healthcare and 2) to investigate the research question of what does “trustworthy AI” mean at the time of the COVID-19 pandemic. To this end, we present the results of a post-hoc self-assessment to evaluate the trustworthiness of an AI system for predicting a multiregional score conveying the degree of lung compromise in COVID-19 patients, developed and verified by an interdisciplinary team with members from academia, public hospitals, and industry in time of pandemic. The AI system aims to help radiologists to estimate and communicate the severity of damage in a patient’s lung from Chest X-rays. It has been experimentally deployed in the radiology department of the ASST Spedali Civili clinic in Brescia, Italy, since December 2020 during pandemic time. The methodology we have applied for our post-hoc assessment, called Z-Inspection®, uses sociotechnical scenarios to identify ethical, technical, and domain-specific issues in the use of the AI system in the context of the pandemic.

@article{AAB2022,
  author = {Allahabadi, Himanshi and Amann, Julia and Balot, Isabelle and Beretta, Andrea and Binkley, Charles and Bozenhard, Jonas and Bruneault, Frederick and Brusseau, James and Candemir, Sema and Cappellini, Luca Alessandro and Chakraborty, Subrata and Cherciu, Nicoleta and Cociancig, Christina and Coffee, Megan and Ek, Irene and Espinosa-Leal, Leonardo and Farina, Davide and Fieux-Castagnet, Genevieve and Frauenfelder, Thomas and Gallucci, Alessio and Giuliani, Guya and Golda, Adam and van Halem, Irmhild and Hildt, Elisabeth and Holm, Sune and Kararigas, Georgios and Krier, Sebastien A. and Kuhne, Ulrich and Lizzi, Francesca and Madai, Vince I. and Markus, Aniek F. and Masis, Serg and Mathez, Emilie Wiinblad and Mureddu, Francesco and Neri, Emanuele and Osika, Walter and Ozols, Matiss and Panigutti, Cecilia and Parent, Brendan and Pratesi, Francesca and Moreno-Sanchez, Pedro A. and Sartor, Giovanni and Savardi, Mattia and Signoroni, Alberto and Sormunen, Hanna-Maria and Spezzatti, Andy and Srivastava, Adarsh and Stephansen, Annette F. and Theng, Lau Bee and Tithi, Jesmin Jahan and Tuominen, Jarno and Umbrello, Steven and Vaccher, Filippo and Vetter, Dennis and Westerlund, Magnus and Wurth, Renee and Zicari, Roberto V.},
  doi = {10.1109/tts.2022.3195114},
  issn = {2637-6415},
  journal = {IEEE Transactions on Technology and Society},
  line = {4,5},
  month = dec,
  number = {4},
  open_access = {Gold},
  pages = {272–289},
  publisher = {Institute of Electrical and Electronics Engineers (IEEE)},
  title = {Assessing Trustworthy AI in Times of COVID-19: Deep Learning for Predicting a Multiregional Score Conveying the Degree of Lung Compromise in COVID-19 Patients},
  visible_on_website = {YES},
  volume = {3},
  year = {2022}
}

User-driven counterfactual generator: a human centered exploration

Beretta I; Cappuccio E; Marchiori Manerba M

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

In this paper, we critically examine the limitations of the techno-solutionist approach to explanations in the context of counterfactual generation, reaffirming interactivity as a core value in the explanation interface between the model and the user.
@misc{BCM2022, author = {M, Beretta I; Cappuccio E; Marchiori Manerba}, line = {1,3}, month = dec, title = {User-driven counterfactual generator: a human centered exploration}, year = {2022} }

2025

SafeGen: safeguarding privacy and fairness through a genetic method

Martina Cinquini, Marta Marchiori Manerba, Federico Mazzoni, Francesca Pratesi, and Riccardo Guidotti

Machine Learning, Sep 2025

RESEARCH LINE
5

Abs Bib HTML

To ensure that Machine Learning systems produce unharmful outcomes, pursuing a joint optimization of performance and ethical profiles such as privacy and fairness is crucial. However, jointly optimizing these two ethical dimensions while maintaining predictive accuracy remains a fundamental challenge. Indeed, privacy-preserving techniques may worsen fairness and restrain the model’s ability to learn accurate statistical patterns, while data mitigation techniques may inadvertently compromise privacy. Aiming to bridge this gap, we propose safeGen, a preprocessing fairness enhancing and privacy-preserving method for tabular data. SafeGen employs synthetic data generation through a genetic algorithm to ensure that sensitive attributes are protected while maintaining the necessary statistical properties. We assess our method across multiple datasets, comparing it against state-of-the-art privacy-preserving and fairness approaches through a threefold evaluation: privacy preservation, fairness enhancement, and generated data plausibility. Through extensive experiments, we demonstrate that SafeGen consistently achieves strong anonymization while preserving or improving dataset fairness across several benchmarks. Additionally, through hybrid privacy-fairness constraints and the use of a genetic synthesizer, SafeGen ensures the plausibility of synthetic records while minimizing discrimination. Our findings demonstrate that modeling fairness and privacy within a unified generative method yields significantly better outcomes than addressing these constraints separately, reinforcing the importance of integrated approaches when multiple ethical objectives must be simultaneously satisfied.
@article{CMM2025, author = {Cinquini, Martina and Marchiori Manerba, Marta and Mazzoni, Federico and Pratesi, Francesca and Guidotti, Riccardo}, doi = {10.1007/s10994-025-06835-9}, issn = {1573-0565}, journal = {Machine Learning}, line = {5}, month = sep, number = {10}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {SafeGen: safeguarding privacy and fairness through a genetic method}, visible_on_website = {YES}, volume = {114}, year = {2025} }

2024

FairBelief - Assessing Harmful Beliefs in Language Models

Mattia Setzu, Marta Marchiori Manerba, Pasquale Minervini, and Debora Nozza

In Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024) , Sep 2024

RESEARCH LINE
5

Abs Bib HTML

Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing.This paper proposes FairBelief, an analytical approach to capture and assess beliefs, i.e., propositions that an LM may embed with different degrees of confidence and that covertly influence its predictions. With FairBelief, we leverage prompting to study the behavior of several state-of-the-art LMs across different previously neglected axes, such as model scale and likelihood, assessing predictions on a fairness dataset specifically designed to quantify LMs’ outputs’ hurtfulness.Finally, we conclude with an in-depth qualitative assessment of the beliefs emitted by the models.We apply FairBelief to English LMs, revealing that, although these architectures enable high performances on diverse natural language processing tasks, they show hurtful beliefs about specific genders. Interestingly, training procedure and dataset, model scale, and architecture induce beliefs of different degrees of hurtfulness.
@inproceedings{SMM2024, author = {Setzu, Mattia and Marchiori Manerba, Marta and Minervini, Pasquale and Nozza, Debora}, booktitle = {Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024)}, doi = {10.18653/v1/2024.trustnlp-1.3}, line = {5}, open_access = {Gold}, pages = {27–39}, publisher = {Association for Computational Linguistics}, title = {FairBelief - Assessing Harmful Beliefs in Language Models}, visible_on_website = {YES}, year = {2024} }
An Interactive Interface for Feature Space Navigation

Eleonora Cappuccio, Isacco Beretta, Marta Marchiori Manerba, and Salvatore Rinzivillo

Jun 2024

RESEARCH LINE
3

Abs Bib HTML

In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to provide users with an intuitive and direct way to navigate through the feature space, inspect model behavior, and perform what-if analyses via feature manipulations and visual feedback. We integrate multiple views including projections of high-dimensional data, decision boundary surfaces, and sensitivity indicators. The interface also supports real-time adjustments of feature values to observe the corresponding changes in the model predictions. Our experiments show that the system can help both novice and expert users to detect regions of uncertainty, identify influential features, and generate hypotheses for model improvement.
@inbook{CBM2024, author = {Cappuccio, Eleonora and Beretta, Isacco and Marchiori Manerba, Marta and Rinzivillo, Salvatore}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240184}, isbn = {9781643685229}, issn = {1879-8314}, line = {3}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {An Interactive Interface for Feature Space Navigation}, visible_on_website = {YES}, year = {2024} }
Social Bias Probing: Fairness Benchmarking for Language Models

Marta Marchiori Manerba, Karolina Stanczak, Riccardo Guidotti, and Isabelle Augenstein

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Jun 2024

RESEARCH LINE
5

Abs Bib HTML

While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limiting our understanding of bias complexities. This paper proposes a novel framework for probing language models for social biases by assessing disparate treatment, which involves treating individuals differently according to their affiliation with a sensitive demographic group. We curate SoFa, a large-scale benchmark designed to address the limitations of existing fairness collections. SoFa expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. Comparing our methodology with existing benchmarks, we reveal that biases within language models are more nuanced than acknowledged, indicating a broader scope of encoded biases than previously recognized. Benchmarking LMs on SoFa, we expose how identities expressing different religions lead to the most pronounced disparate treatments across all models. Finally, our findings indicate that real-life adversities faced by various groups such as women and people with disabilities are mirrored in the behavior of these models. (arXiv)
@inproceedings{MSG2024, author = {Marchiori Manerba, Marta and Stanczak, Karolina and Guidotti, Riccardo and Augenstein, Isabelle}, booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing}, doi = {10.18653/v1/2024.emnlp-main.812}, line = {5}, open_access = {Gold}, pages = {14653–14671}, publisher = {Association for Computational Linguistics}, title = {Social Bias Probing: Fairness Benchmarking for Language Models}, visible_on_website = {YES}, year = {2024} }
Mapping the landscape of ethical considerations in explainable AI research

Luca Nannini, Marta Marchiori Manerba, and Isacco Beretta

Ethics and Information Technology, Jun 2024

RESEARCH LINE
1

5

Abs Bib HTML

With its potential to contribute to the ethical governance of AI, explainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet the substantiation of these claims with rigorous ethical analysis and reflection remains largely unexamined. This contribution endeavours to scrutinise the relationship between XAI and ethical considerations. We conduct a systematic review of the XAI literature by applying a multi-stage filtering process and then developing a taxonomy for classifying the depth and quality of ethical engagement in the field. Our findings show that although a growing body of XAI research references ethical issues, the majority does so at a superficial level. We identify nine ethical themes (e.g., autonomy, justice, transparency) and show how they are engaged in the literature. We discuss trends, deficits, and future directions for integrating ethical considerations meaningfully into XAI research.
@article{NMB2024, author = {Nannini, Luca and Marchiori Manerba, Marta and Beretta, Isacco}, doi = {10.1007/s10676-024-09773-7}, issn = {1572-8439}, journal = {Ethics and Information Technology}, line = {1,5}, month = jun, number = {3}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Mapping the landscape of ethical considerations in explainable AI research}, visible_on_website = {YES}, volume = {26}, year = {2024} }

2022

Investigating Debiasing Effects on Classification and Explainability

Marta Marchiori Manerba, and Riccardo Guidotti

In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2022

RESEARCH LINE
1

5

Abs Bib HTML

During each stage of a dataset creation and development process, harmful biases can be accidentally introduced, leading to models that perpetuates marginalization and discrimination of minorities, as the role of the data used during the training is critical. We propose an evaluation framework that investigates the impact on classification and explainability of bias mitigation preprocessing techniques used to assess data imbalances concerning minorities’ representativeness and mitigate the skewed distributions discovered. Our evaluation focuses on assessing fairness, explainability and performance metrics. We analyze the behavior of local model-agnostic explainers on the original and mitigated datasets to examine whether the proxy models learned by the explainability techniques to mimic the black-boxes disproportionately rely on sensitive attributes, demonstrating biases rooted in the explainers. We conduct several experiments about known biased datasets to demonstrate our proposal’s novelty and effectiveness for evaluation and bias detection purposes.
@inproceedings{MG2022, author = {Marchiori Manerba, Marta and Guidotti, Riccardo}, booktitle = {Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’22}, doi = {10.1145/3514094.3534170}, line = {1,5}, month = jul, open_access = {NO}, pages = {468–478}, publisher = {ACM}, series = {AIES ’22}, title = {Investigating Debiasing Effects on Classification and Explainability}, visible_on_website = {YES}, year = {2022} }
User-driven counterfactual generator: a human centered exploration

Beretta I; Cappuccio E; Marchiori Manerba M

Dec 2022

RESEARCH LINE
1

3

Abs Bib HTML

In this paper, we critically examine the limitations of the techno-solutionist approach to explanations in the context of counterfactual generation, reaffirming interactivity as a core value in the explanation interface between the model and the user.
@misc{BCM2022, author = {M, Beretta I; Cappuccio E; Marchiori Manerba}, line = {1,3}, month = dec, title = {User-driven counterfactual generator: a human centered exploration}, year = {2022} }

2025

A Practical Approach to Causal Inference over Time

Martina Cinquini, Isacco Beretta, Salvatore Ruggieri, and Isabel Valera

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2025

RESEARCH LINE
2

Abs Bib HTML

In this paper, we focus on estimating the causal effect of an intervention over time on a dynamical system. To that end, we formally define causal interventions and their effects over time on discrete-time stochastic processes (DSPs). Then, we show under which conditions the equilibrium states of a DSP, both before and after a causal intervention, can be captured by a structural causal model (SCM). With such an equivalence at hand, we provide an explicit mapping from vector autoregressive models (VARs), broadly applied in econometrics, to linear, but potentially cyclic and/or affected by unmeasured confounders, SCMs. The resulting causal VAR framework allows us to perform causal inference over time from observational time series data. Our experiments on synthetic and real-world datasets show that the proposed framework achieves strong performance in terms of observational forecasting while enabling accurate estimation of the causal effect of interventions on dynamical systems. We demonstrate, through a case study, the potential practical questions that can be addressed using the proposed causal VAR framework.
@article{CBR2025, author = {Cinquini, Martina and Beretta, Isacco and Ruggieri, Salvatore and Valera, Isabel}, doi = {10.1609/aaai.v39i14.33626}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {2}, month = apr, number = {14}, open_access = {Gold}, pages = {14832–14839}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {A Practical Approach to Causal Inference over Time}, visible_on_website = {YES}, volume = {39}, year = {2025} }
SafeGen: safeguarding privacy and fairness through a genetic method

Martina Cinquini, Marta Marchiori Manerba, Federico Mazzoni, Francesca Pratesi, and Riccardo Guidotti

Machine Learning, Sep 2025

RESEARCH LINE
5

Abs Bib HTML

To ensure that Machine Learning systems produce unharmful outcomes, pursuing a joint optimization of performance and ethical profiles such as privacy and fairness is crucial. However, jointly optimizing these two ethical dimensions while maintaining predictive accuracy remains a fundamental challenge. Indeed, privacy-preserving techniques may worsen fairness and restrain the model’s ability to learn accurate statistical patterns, while data mitigation techniques may inadvertently compromise privacy. Aiming to bridge this gap, we propose safeGen, a preprocessing fairness enhancing and privacy-preserving method for tabular data. SafeGen employs synthetic data generation through a genetic algorithm to ensure that sensitive attributes are protected while maintaining the necessary statistical properties. We assess our method across multiple datasets, comparing it against state-of-the-art privacy-preserving and fairness approaches through a threefold evaluation: privacy preservation, fairness enhancement, and generated data plausibility. Through extensive experiments, we demonstrate that SafeGen consistently achieves strong anonymization while preserving or improving dataset fairness across several benchmarks. Additionally, through hybrid privacy-fairness constraints and the use of a genetic synthesizer, SafeGen ensures the plausibility of synthetic records while minimizing discrimination. Our findings demonstrate that modeling fairness and privacy within a unified generative method yields significantly better outcomes than addressing these constraints separately, reinforcing the importance of integrated approaches when multiple ethical objectives must be simultaneously satisfied.
@article{CMM2025, author = {Cinquini, Martina and Marchiori Manerba, Marta and Mazzoni, Federico and Pratesi, Francesca and Guidotti, Riccardo}, doi = {10.1007/s10994-025-06835-9}, issn = {1573-0565}, journal = {Machine Learning}, line = {5}, month = sep, number = {10}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {SafeGen: safeguarding privacy and fairness through a genetic method}, visible_on_website = {YES}, volume = {114}, year = {2025} }
A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods

Martina Cinquini, Karima Makhlouf, Sami Zhioua, Catuscia Palamidessi, and Riccardo Guidotti

IEEE Access, Sep 2025

RESEARCH LINE
2

5

Abs Bib HTML

Causal discovery (CD) algorithms are increasingly applied to socially and ethically sensitive domains. However, their evaluation under realistic conditions remains challenging due to the scarcity of real-world datasets annotated with ground-truth causal structures. Whereas synthetic data generators support controlled benchmarking, they often overlook forms of bias, such as dependencies involving sensitive attributes, which may significantly affect the observed distribution and compromise the trustworthiness of downstream analysis. This paper introduces a novel synthetic data generation framework that enables controlled bias injection while preserving the causal relationships specified in a ground-truth causal graph. The framework aims to evaluate the reliability of CD methods by examining the impact of varying bias levels and outcome binarization thresholds. Experimental results show that even moderate bias levels can lead CD approaches to fail to correctly infer causal links, particularly those connecting sensitive attributes to decision outcomes. These findings underscore the need for expert validation and highlight the limitations of current CD methods in fairness-critical applications. Our proposal thus provides an essential tool for benchmarking and improving CD algorithms in biased, real-world data settings.
@article{CMZ2025, author = {Cinquini, Martina and Makhlouf, Karima and Zhioua, Sami and Palamidessi, Catuscia and Guidotti, Riccardo}, doi = {10.1109/access.2025.3573201}, issn = {2169-3536}, journal = {IEEE Access}, line = {2,5}, open_access = {Gold}, pages = {97376–97391}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {A Bias Injection Technique to Assess the Resilience of Causal Discovery Methods}, visible_on_website = {YES}, volume = {13}, year = {2025} }

2024

Causality-Aware Local Interpretable Model-Agnostic Explanations

Martina Cinquini, and Riccardo Guidotti

Sep 2024

RESEARCH LINE
1

2

Abs Bib HTML

A main drawback of eXplainable Artificial Intelligence (XAI) approaches is the feature independence assumption, hindering the study of potential variable dependencies. This leads to approximating black box behaviors by analysing the effects on randomly generated feature values that may rarely occur in the original samples. This paper addresses this issue by integrating causal knowledge in an XAI method to enhance transparency and enable users to assess the quality of the generated explanations. Specifically, we propose a novel extension to a widely used local and model-agnostic explainer, which encodes explicit causal relationships within the data surrounding the instance being explained. Extensive experiments show that our approach overcomes the original method in terms of faithfully replicating the black-box model’s mechanism and the consistency and reliability of the generated explanations. (arXiv)
@inbook{CG2024, author = {Cinquini, Martina and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-63800-8_6}, isbn = {9783031638008}, issn = {1865-0937}, line = {1,2}, open_access = {NO}, pages = {108–124}, publisher = {Springer Nature Switzerland}, title = {Causality-Aware Local Interpretable Model-Agnostic Explanations}, visible_on_website = {YES}, year = {2024} }

2023

The Importance of Time in Causal Algorithmic Recourse

Isacco Beretta, and Martina Cinquini

Sep 2023

RESEARCH LINE
2

Abs Bib HTML

The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inbook{BC2023, author = {Beretta, Isacco and Cinquini, Martina}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44064-9_16}, isbn = {9783031440649}, issn = {1865-0937}, line = {2}, open_access = {NO}, pages = {283–298}, publisher = {Springer Nature Switzerland}, title = {The Importance of Time in Causal Algorithmic Recourse}, visible_on_website = {YES}, year = {2023} }
Handling Missing Values in Local Post-hoc Explainability

Martina Cinquini, Fosca Giannotti, Riccardo Guidotti, and Andrea Mattei

Sep 2023

RESEARCH LINE
1

Abs Bib HTML

Missing data are quite common in real scenarios when using Artificial Intelligence (AI) systems for decision-making with tabular data and effectively handling them poses a significant challenge for such systems. While some machine learning models used by AI systems can tackle this problem, the existing literature lacks post-hoc explainability approaches able to deal with predictors that encounter missing data. In this paper, we extend a widely used local model-agnostic post-hoc explanation approach that enables explainability in the presence of missing values by incorporating state-of-the-art imputation methods within the explanation process. Since our proposal returns explanations in the form of feature importance, the user will be aware also of the importance of a missing value in a given record for a particular prediction. Extensive experiments show the effectiveness of the proposed method with respect to some baseline solutions relying on traditional data imputation.
@inbook{CGG2023, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo and Mattei, Andrea}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_14}, isbn = {9783031440670}, issn = {1865-0937}, line = {1}, open_access = {Gold}, pages = {256–278}, publisher = {Springer Nature Switzerland}, title = {Handling Missing Values in Local Post-hoc Explainability}, visible_on_website = {YES}, year = {2023} }

2021

Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery

Martina Cinquini, Fosca Giannotti, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
2

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, artificial intelligence explanation, etc. In all such contexts, it is important to generate plausible data samples. A common assumption of approaches widely used for data generation is the independence of the features. However, typically, the variables of a dataset de-pend on one another, and these dependencies are not considered in data generation leading to the creation of implausible records. The main problem is that dependencies among variables are typically unknown. In this paper, we design a synthetic dataset generator for tabular data that is able to discover nonlinear causalities among the variables and use them at generation time. State-of-the-art methods for nonlinear causal discovery are typically inefficient. We boost them by restricting the causal discovery among the features appearing in the frequent patterns efficiently retrieved by a pattern mining algorithm. To validate our proposal, we design a framework for generating synthetic datasets with known causalities. Wide experimentation on many synthetic datasets and real datasets with known causalities shows the effectiveness of the proposed method.
@inproceedings{CGG2021, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00016}, line = {2}, month = dec, open_access = {NO}, pages = {54–63}, publisher = {IEEE}, title = {Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery}, visible_on_website = {YES}, year = {2021} }

2025

Balancing Fairness and Interpretability in Clustering with FairParTree

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

The revolution involving Machine Learning has transformed data analytics, making algorithms important in decision-making processes across various domains, even in sensitive scenarios. Indeed, traditional clustering algorithms often lack interpretability and exhibit biases, leading to discriminatory practices and opaque decision-making. To overcome these limitations, we introduce FairParTree, a fair and interpretable clustering algorithm that integrates fairness constraints directly into the clustering process, ensuring that the resulting clusters do not disproportionately disadvantage any particular group. By leveraging the structure of decision trees, FairParTree enhances the interpretability of clustering results by providing clear and understandable motivations for cluster assignments through rule-based explanations. We evaluate FairParTree against state-of-the-art competitors. Through extensive experiments, we show that it maintains strong performances w.r.t. fairness, interpretability, and clustering quality across different dataset sizes, thus positioning itself as a competitive, fair, and interpretable clustering algorithm.
@inbook{LCM2025c, author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_5}, isbn = {9783032083241}, issn = {1865-0937}, line = {1}, month = oct, open_access = {NO}, pages = {104–127}, publisher = {Springer Nature Switzerland}, title = {Balancing Fairness and Interpretability in Clustering with FairParTree}, visible_on_website = {YES}, year = {2025} }
Balancing Fairness and Interpretability in Clustering with FairParTree

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Oct 2025

RESEARCH LINE
1

5

Abs Bib HTML

The revolution involving Machine Learning has transformed data analytics, making algorithms important in decision-making processes across various domains, even in sensitive scenarios. Indeed, traditional clustering algorithms often lack interpretability and exhibit biases, leading to discriminatory practices and opaque decision-making. To overcome these limitations, we introduce FairParTree, a fair and interpretable clustering algorithm that integrates fairness constraints directly into the clustering process, ensuring that the resulting clusters do not disproportionately disadvantage any particular group. By leveraging the structure of decision trees, FairParTree enhances the interpretability of clustering results by providing clear and understandable motivations for cluster assignments through rule-based explanations. We evaluate FairParTree against state-of-the-art competitors. Through extensive experiments, we show that it maintains strong performances w.r.t. fairness, interpretability, and clustering quality across different dataset sizes, thus positioning itself as a competitive, fair, and interpretable clustering algorithm.
@inbook{LCM2025, author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_5}, isbn = {9783032083241}, issn = {1865-0937}, line = {1,5}, month = oct, open_access = {Gold}, pages = {104–127}, publisher = {Springer Nature Switzerland}, title = {Balancing Fairness and Interpretability in Clustering with FairParTree}, visible_on_website = {YES}, year = {2025} }
Interpretable Instance-Based Learning Through Pairwise Distance Trees

Andrea Fedele, Alessio Cascione, Riccardo Guidotti, and Cristiano Landi

Sep 2025

RESEARCH LINE
1

Abs Bib HTML

Instance-based models offer natural interpretability by making decisions based on concrete examples. However, their transparency is often hindered by the use of complex similarity measures, which are difficult to interpret, especially in high-dimensional datasets. To address this issue, this paper presents a meta-learning framework that enhances the interpretability of instance-based models by replacing traditional, complex pairwise distance functions with interpretable pairwise distance trees. These trees are designed to prioritize simplicity and transparency while preserving the model’s effectiveness. By offering a clear decision-making process, the framework makes the instance selection more understandable. Also, the framework mitigates the computational burden of instance-based models, which typically require calculating all pairwise distances. Leveraging the generalization capabilities of pairwise distance trees and employing sampling strategies to select representative subsets, the method significantly reduces computational complexity. Our experiments demonstrate that the proposed approach improves computational efficiency with only a modest trade-off in accuracy while substantially enhancing the interpretability of the learned distance measure.
@inbook{FCG2025, author = {Fedele, Andrea and Cascione, Alessio and Guidotti, Riccardo and Landi, Cristiano}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-032-06078-5_1}, isbn = {9783032060785}, issn = {1611-3349}, line = {1}, month = sep, open_access = {Gold}, pages = {3–21}, publisher = {Springer Nature Switzerland}, title = {Interpretable Instance-Based Learning Through Pairwise Distance Trees}, visible_on_website = {YES}, year = {2025} }

A Note on Methods for Explainable Malware Analysis

Cristiano Landi, Alessio Cascione, Marta Marchiori Manerba, and Riccardo Guidotti

Dec 2025

RESEARCH LINE

Bib

@misc{LCM2025b,
  author = {Landi, Cristiano and Cascione, Alessio and Manerba, Marta Marchiori and Guidotti, Riccardo},
  line = {1},
  month = dec,
  title = {A Note on Methods for Explainable Malware Analysis},
  year = {2025}
}

2024

Shape-based Methods in Mobility Data Analysis: Effectiveness and Limitations

Cristiano Landi, and Riccardo Guidotti

Nov 2024

RESEARCH LINE
1

Abs Bib HTML

Although Mobility Data Analysis (MDA) has been explored for a long time, it still lags behind advancements in other fields.A common issue in MDA is the lack of methods’ standardization and reusability. On the other hand, for instance, in time series analysis, the existing methods are typically general-purpose, and it is possible to apply them across diverse datasets and applications without extensive customization. Still, in MDA, most contributions are ad-hoc and designed to address specific research questions, which limits their generalizability and reusability. Recently, some researchers explored the application of shapelet transform to trajectory data, i.e., extracting discriminatory sub-trajectories from training data to be used as classification features. Unlike current MDA methods, this line of research eliminates the need for feature engineering, greatly improving its ability to generalize. While shapelets on mobility data have shown state-of-the-art performance on public classification datasets, it is still not clear why they work. Are these subtrajectories merely proxies for geographic location, or do they also capture motion dynamics? We empirically show that shapelet-based approaches are a viable alternative to classical methods and flexible enough to solve MDA tasks related solely to trajectory shape, solely to movement dynamics, and those related to both. Additionally, we investigate the problem of Geographic Transferability, showing that such approaches offer a promising starting point for tackling this challenge.
@article{LG2024, author = {Landi, Cristiano and Guidotti, Riccardo}, doi = {10.21203/rs.3.rs-5369626/v1}, line = {1}, month = nov, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Shape-based Methods in Mobility Data Analysis: Effectiveness and Limitations}, visible_on_website = {YES}, year = {2024} }

2023

Interpretable Data Partitioning Through Tree-Based Clustering Methods

Riccardo Guidotti, Cristiano Landi, Andrea Beretta, Daniele Fadda, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{GLB2023, author = {Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-45275-8_33}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {492–507}, publisher = {Springer Nature Switzerland}, title = {Interpretable Data Partitioning Through Tree-Based Clustering Methods}, visible_on_website = {YES}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{LSG2023, address = {Cham, Switzerland}, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Advances in Intelligent Data Analysis XXI}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, visible_on_website = {YES}, year = {2023} }

2025

Ensemble Counterfactual Explanations for Churn Analysis

Samuele Tonati, Marzio Di Vece, Roberto Pellungrini, and Fosca Giannotti

Nov 2025

RESEARCH LINE
1

Abs Bib HTML

Counterfactual explanations play a crucial role in interpreting and understanding the decision-making process of complex machine learning models, offering insights into why a particular prediction was made and how it could be altered. However, individual counterfactual explanations generated by different methods may vary significantly in terms of their quality, diversity, and coherence to the black-box prediction. This is especially important in financial applications such as churn analysis, where customer retention officers could explore different approaches and solutions with the clients to prevent churning. The officer’s capability to modify and explore different explanations is pivotal to his ability to provide feasible solutions. To address this challenge, we propose an evaluation framework through the implementation of an ensemble approach that combines state-of-the-art counterfactual generation methods and a linear combination score of desired properties to select the most appropriate explanation. We conduct our experiments on three publicly available churn datasets in different domains. Our experimental results demonstrate that the ensemble of counterfactual explanations provides more diverse and comprehensive insights into model behavior compared to individual methods alone that suffer from specific weaknesses. By aggregating, evaluating, and selecting multiple explanations, our approach enhances the diversity of the explanation, highlights common patterns, and mitigates the limitations of any single method, offering to the user the ability to tweak the explanation properties to their needs.
@inbook{TDP2025, author = {Tonati, Samuele and Di Vece, Marzio and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_21}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {332–347}, publisher = {Springer Nature Switzerland}, title = {Ensemble Counterfactual Explanations for Churn Analysis}, visible_on_website = {YES}, year = {2025} }

2025

Interpretable Instance-Based Learning Through Pairwise Distance Trees

Andrea Fedele, Alessio Cascione, Riccardo Guidotti, and Cristiano Landi

Sep 2025

RESEARCH LINE
1

Abs Bib HTML

Instance-based models offer natural interpretability by making decisions based on concrete examples. However, their transparency is often hindered by the use of complex similarity measures, which are difficult to interpret, especially in high-dimensional datasets. To address this issue, this paper presents a meta-learning framework that enhances the interpretability of instance-based models by replacing traditional, complex pairwise distance functions with interpretable pairwise distance trees. These trees are designed to prioritize simplicity and transparency while preserving the model’s effectiveness. By offering a clear decision-making process, the framework makes the instance selection more understandable. Also, the framework mitigates the computational burden of instance-based models, which typically require calculating all pairwise distances. Leveraging the generalization capabilities of pairwise distance trees and employing sampling strategies to select representative subsets, the method significantly reduces computational complexity. Our experiments demonstrate that the proposed approach improves computational efficiency with only a modest trade-off in accuracy while substantially enhancing the interpretability of the learned distance measure.
@inbook{FCG2025, author = {Fedele, Andrea and Cascione, Alessio and Guidotti, Riccardo and Landi, Cristiano}, booktitle = {Machine Learning and Knowledge Discovery in Databases. Research Track}, doi = {10.1007/978-3-032-06078-5_1}, isbn = {9783032060785}, issn = {1611-3349}, line = {1}, month = sep, open_access = {Gold}, pages = {3–21}, publisher = {Springer Nature Switzerland}, title = {Interpretable Instance-Based Learning Through Pairwise Distance Trees}, visible_on_website = {YES}, year = {2025} }

2024

The ALTAI checklist as a tool to assess ethical and legal implications for a trustworthy AI development in education

Andrea Fedele, Clara Punzi, and Stefano Tramacere

Computer Law & Security Review, Jul 2024

RESEARCH LINE
5

Abs Bib HTML

The rapid proliferation of Artificial Intelligence (AI) applications in various domains of our lives has prompted a need for a shift towards a human-centered and trustworthy approach to AI. In this study we employ the Assessment List for Trustworthy Artificial Intelligence (ALTAI) checklist to evaluate the trustworthiness of Artificial Intelligence for Student Performance Prediction (AI4SPP), an AI-powered system designed to detect students at risk of school failure. We strongly support the ethical and legal development of AI and propose an implementation design where the user can choose to have access to each level of a three-tier outcome bundle: the AI prediction alone, the prediction along with its confidence level, and, lastly, local explanations for each grade prediction together with the previous two information. AI4SPP aims to raise awareness among educators and students regarding the factors contributing to low school performance, thereby facilitating the implementation of interventions not only to help students, but also to address biases within the school community. However, we also emphasize the ethical and legal concerns that could arise from a misuse of the AI4SPP tool. First of all, the collection and analysis of data, which is essential for the development of AI models, may lead to breaches of privacy, thus causing particularly adverse consequences in the case of vulnerable individuals. Furthermore, the system’s predictions may be influenced by unacceptable discrimination based on gender, ethnicity, or socio-economic background, leading to unfair actions. The ALTAI checklist serves as a valuable self-assessment tool during the design phase of AI systems, by means of which commonly overlooked weaknesses can be highlighted and addressed. In addition, the same checklist plays a crucial role throughout the AI system life cycle. Continuous monitoring of sensitive features within the dataset, alongside survey assessments to gauge users’ responses to the systems, is essential for gathering insights and intervening accordingly. We argue that adopting a critical approach to AI development is essential for societal progress, believing that it can evolve and accelerate over time without impeding openness to new technologies. By aligning with ethical principles and legal requirements, AI systems can make significant contributions to education while mitigating potential risks and ensuring a fair and inclusive learning environment.
@article{FPT2024, author = {Fedele, Andrea and Punzi, Clara and Tramacere, Stefano}, doi = {10.1016/j.clsr.2024.105986}, issn = {2212-473X}, journal = {Computer Law & Security Review}, line = {5}, month = jul, open_access = {Gold}, pages = {105986}, publisher = {Elsevier BV}, title = {The ALTAI checklist as a tool to assess ethical and legal implications for a trustworthy AI development in education}, visible_on_website = {YES}, volume = {53}, year = {2024} }
Explaining Siamese networks in few-shot learning

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Machine Learning, Apr 2024

RESEARCH LINE
1

Abs Bib HTML

Siamese neural networks are widely used in few-shot learning tasks thanks to their ability to compare pairs of samples and generalize from very limited labeled data. However, their internal decision-making process remains opaque, since similarity-based representations do not provide intuitive explanations for end users. In this work, we investigate how to explain Siamese networks by attributing contribution scores to both input samples involved in the comparison. We introduce an explanation method specifically tailored to pairwise architectures, producing two synchronized saliency maps that highlight which regions of the support and query examples drive the similarity judgment. We evaluate the approach on image-based few-shot classification benchmarks, showing that the explanations highlight semantically meaningful structures and remain consistent across different evaluation episodes.
@article{FGP2024, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, doi = {10.1007/s10994-024-06529-8}, issn = {1573-0565}, journal = {Machine Learning}, line = {1}, month = apr, number = {10}, open_access = {Gold}, pages = {7723–7760}, publisher = {Springer Science and Business Media LLC}, title = {Explaining Siamese networks in few-shot learning}, visible_on_website = {YES}, volume = {113}, year = {2024} }

2023

Explain and interpret few-shot learning

Fedele A.

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

Recent advancements in Artificial Intelligence have been fueled by vast datasets, powerful computing resources, and sophisticated algorithms. However, traditional Machine Learning models face limitations in handling scarce data. Few-Shot Learning (FSL) offers a promising solution by training models on a small number of examples per class. This manuscript introduces FXI-FSL, a framework for eXplainability and Interpretability in FSL, which aims to develop post-hoc explainability algorithms and interpretableby-design alternatives. A noteworthy contribution is the SIamese Network EXplainer (SINEX), a post-hoc approach shedding light on Siamese Network behavior. The proposed framework seeks to unveil the rationale behind FSL models, instilling trust in their real-world applications. Moreover, it emerges as a safeguard for developers, facilitating models fine-tuning prior to deployment, and as a guide for end users navigating the decisions of these models.
@misc{F2023, author = {A., Fedele}, line = {1}, month = dec, title = {Explain and interpret few-shot learning}, year = {2023} }

2022

Explaining Siamese Networks in Few-Shot Learning for Audio Data

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

Dec 2022

RESEARCH LINE
4

Abs Bib HTML

Machine learning models are not able to generalize correctly when queried on samples belonging to class distributions that were never seen during training. This is a critical issue, since real world applications might need to quickly adapt without the necessity of re-training. To overcome these limitations, few-shot learning frameworks have been proposed and their applicability has been studied widely for computer vision tasks. Siamese Networks learn pairs similarity in form of a metric that can be easily extended on new unseen classes. Unfortunately, the downside of such systems is the lack of explainability. We propose a method to explain the outcomes of Siamese Networks in the context of few-shot learning for audio data. This objective is pursued through a local perturbation-based approach that evaluates segments-weighted-average contributions to the final outcome considering the interplay between different areas of the audio spectrogram. Qualitative and quantitative results demonstrate that our method is able to show common intra-class characteristics and erroneous reliance on silent sections.
@inbook{FGP2022, address = {Cham, Switzerland}, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-18840-4_36}, isbn = {9783031188404}, issn = {1611-3349}, line = {4}, open_access = {NO}, pages = {509–524}, publisher = {Springer Nature Switzerland}, title = {Explaining Siamese Networks in Few-Shot Learning for Audio Data}, visible_on_website = {YES}, year = {2022} }

2024

The ALTAI checklist as a tool to assess ethical and legal implications for a trustworthy AI development in education

Andrea Fedele, Clara Punzi, and Stefano Tramacere

Computer Law & Security Review, Jul 2024

RESEARCH LINE
5

Abs Bib HTML

The rapid proliferation of Artificial Intelligence (AI) applications in various domains of our lives has prompted a need for a shift towards a human-centered and trustworthy approach to AI. In this study we employ the Assessment List for Trustworthy Artificial Intelligence (ALTAI) checklist to evaluate the trustworthiness of Artificial Intelligence for Student Performance Prediction (AI4SPP), an AI-powered system designed to detect students at risk of school failure. We strongly support the ethical and legal development of AI and propose an implementation design where the user can choose to have access to each level of a three-tier outcome bundle: the AI prediction alone, the prediction along with its confidence level, and, lastly, local explanations for each grade prediction together with the previous two information. AI4SPP aims to raise awareness among educators and students regarding the factors contributing to low school performance, thereby facilitating the implementation of interventions not only to help students, but also to address biases within the school community. However, we also emphasize the ethical and legal concerns that could arise from a misuse of the AI4SPP tool. First of all, the collection and analysis of data, which is essential for the development of AI models, may lead to breaches of privacy, thus causing particularly adverse consequences in the case of vulnerable individuals. Furthermore, the system’s predictions may be influenced by unacceptable discrimination based on gender, ethnicity, or socio-economic background, leading to unfair actions. The ALTAI checklist serves as a valuable self-assessment tool during the design phase of AI systems, by means of which commonly overlooked weaknesses can be highlighted and addressed. In addition, the same checklist plays a crucial role throughout the AI system life cycle. Continuous monitoring of sensitive features within the dataset, alongside survey assessments to gauge users’ responses to the systems, is essential for gathering insights and intervening accordingly. We argue that adopting a critical approach to AI development is essential for societal progress, believing that it can evolve and accelerate over time without impeding openness to new technologies. By aligning with ethical principles and legal requirements, AI systems can make significant contributions to education while mitigating potential risks and ensuring a fair and inclusive learning environment.
@article{FPT2024, author = {Fedele, Andrea and Punzi, Clara and Tramacere, Stefano}, doi = {10.1016/j.clsr.2024.105986}, issn = {2212-473X}, journal = {Computer Law & Security Review}, line = {5}, month = jul, open_access = {Gold}, pages = {105986}, publisher = {Elsevier BV}, title = {The ALTAI checklist as a tool to assess ethical and legal implications for a trustworthy AI development in education}, visible_on_website = {YES}, volume = {53}, year = {2024} }
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers

Eleonora Cappuccio, Benedetta Muscato, Laura Pollacci, Marta Marchiori Manerba, Clara Punzi, and 5 more authors

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
@misc{CMP2024, address = {Aachen, Germany}, author = {Cappuccio, Eleonora and Muscato, Benedetta and Pollacci, Laura and Manerba, Marta Marchiori and Punzi, Clara and Mala, Chandana Sree and Lalli, Margherita and Gezici, Gizem and Natilli, Michela and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers}, year = {2024} }

2023

Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic

Clara Punzi, Aleksandra Maslennikova, Gizem Gezici, Roberto Pellungrini, and Fosca Giannotti

Dec 2023

RESEARCH LINE
1

4

Abs Bib HTML

Pandemic vaccination campaigns must account for vaccine skepticism as an obstacle to overcome. Using machine learning to identify behavioral and psychological patterns in public survey datasets can provide valuable insights and inform vaccination campaigns based on empirical evidence. However, we argue that the adoption of local and global explanation methodologies can provide additional support to health practitioners by suggesting personalized communication strategies and revealing potential demographic, social, or structural barriers to vaccination requiring systemic changes. In this paper, we first implement a chain classification model for the adoption of the vaccine during the H1N1 influenza outbreak taking seasonal vaccination information into account, and then compare it with a binary classifier for vaccination to better understand the overall patterns in the data. Following that, we derive and compare global explanations using post-hoc methodologies and interpretable-by-design models. Our findings indicate that socio-demographic factors play a distinct role in the H1N1 vaccination as compared to the general vaccination. Nevertheless, medical recommendation and health insurance remain significant factors for both vaccinations. Then, we concentrated on the subpopulation of individuals who did not receive an H1N1 vaccination despite being at risk of developing severe symptoms. In an effort to assist practitioners in providing effective recommendations to patients, we present rules and counterfactuals for the selected instances based on local explanations. Finally, we raise concerns regarding gender and racial disparities in healthcare access by analysing the interaction effects of sensitive attributes on the model’s output.
@inbook{PMG2023, author = {Punzi, Clara and Maslennikova, Aleksandra and Gezici, Gizem and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_31}, isbn = {9783031440670}, issn = {1865-0937}, line = {1,4}, open_access = {Gold}, pages = {621–635}, publisher = {Springer Nature Switzerland}, title = {Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic}, visible_on_website = {YES}, year = {2023} }

2025

Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts

Andrea Pugnana, Riccardo Massidda, Francesco Giannini, Pietro Barbiero, Mateo Espinosa Zarlenga, and 4 more authors

Dec 2025

RESEARCH LINE

Bib

@misc{PMG2025,
  author = {Pugnana, Andrea and Massidda, Riccardo and Giannini, Francesco and Barbiero, Pietro and Zarlenga, Mateo Espinosa and Pellungrini, Roberto and Dominici, Gabriele and Giannotti, Fosca and Bacciu, Davide},
  line = {1,2},
  month = dec,
  title = {Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts},
  year = {2025}
}

2024

Interpretable and Fair Mechanisms for Abstaining Classifiers

Daphne Lenders, Andrea Pugnana, Roberto Pellungrini, Toon Calders, Dino Pedreschi, and 1 more author

Dec 2024

RESEARCH LINE
1

5

Abs Bib HTML

Abstaining classifiers have the option to refrain from providing a prediction for instances that are difficult to classify. The abstention mechanism is designed to trade off the classifier‚Äôs performance on the accepted data while ensuring a minimum number of predictions. In this setting, fairness concerns often arise when the abstention mechanism reduces errors only for majority groups, increasing disparities across demographic groups. We introduce Interpretable and Fair Abstaining Classifier (IFAC), an algorithm that can reject predictions based on uncertainty and unfairness. By rejecting potentially unfair predictions, our method reduces disparities across groups of the non-rejected data. The unfairness-based rejections rely on interpretable rule-based fairness checks and situation testing, enabling transparent review and decision-making.
@misc{LPP2024, author = {Lenders, Daphne and Pugnana, Andrea and Pellungrini, Roberto and Calders, Toon and Pedreschi, Dino and Giannotti, Fosca}, doi = {[75,46,72,75,50,73,78]. }, line = {1,5}, month = dec, title = {Interpretable and Fair Mechanisms for Abstaining Classifiers}, year = {2024} }

2023

Topics in Selective Classification

Andrea Pugnana

Proceedings of the AAAI Conference on Artificial Intelligence, Jun 2023

RESEARCH LINE
1

2

Abs Bib HTML

Selective classification (also known as classification with a reject option, or learning to defer) extends a classifier with a selection function (reject option/strategy) to determine whether or not a prediction should be accepted. This mechanism allows the AI system to abstain in those instances where the classifier is more uncertain about the class to predict, introducing a trade-off between performance and coverage (the percentage of cases where the classifier does not abstain). The reject option has been extensively studied from a theoretical standpoint. However, state-of-the-art practical approaches and tools are model-specific, e.g., they are tailored to Deep Neural Networks and focused/experimented mainly on image datasets. In this work I developed a model-agnostic heuristics that is able to lift any (probabilistic) classifier into a selective classifier. The approach exploits both a cross-fitting strategy and results from quantile estimation to build the selective function. The algorithm is tested on several real-world datasets, showing improvements compared to existing methodologies.
@article{P2024, author = {Pugnana, Andrea}, doi = {10.1609/aaai.v37i13.26925}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,2}, month = jun, number = {13}, open_access = {Gold}, pages = {16129–16130}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Topics in Selective Classification}, visible_on_website = {YES}, volume = {37}, year = {2023} }
AUC-based Selective Classification

Pugnana Andrea, and Ruggieri Salvatore

Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
@misc{PR2023, author = {Andrea, Pugnana and Salvatore, Ruggieri}, line = {2}, month = dec, pages = {2494--2514}, title = {AUC-based Selective Classification}, year = {2023} }

2022

Methods and tools for causal discovery and causal inference

Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama

WIREs Data Mining and Knowledge Discovery, Jan 2022

RESEARCH LINE
2

Abs Bib HTML

Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{NPR2022, address = {Danvers, MS, USA}, author = {Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João}, doi = {10.1002/widm.1449}, issn = {1942-4795}, journal = {WIREs Data Mining and Knowledge Discovery}, line = {2}, month = jan, number = {2}, open_access = {Gold}, publisher = {Wiley}, title = {Methods and tools for causal discovery and causal inference}, visible_on_website = {YES}, volume = {12}, year = {2022} }

2025

Embracing Diversity: A Multi-Perspective Approach with Soft Labels

Benedetta Muscato, Praveen Bushipaka, Gizem Gezici, Lucia Passaro, Fosca Giannotti, and 1 more author

Sep 2025

RESEARCH LINE
4

5

Abs Bib HTML

In subjective tasks like stance detection, diverse human perspectives are often simplified into a single ground truth through label aggregation i.e. majority voting, potentially marginalizing minority viewpoints. This paper presents a Multi-Perspective framework for stance detection that explicitly incorporates annotation diversity by using soft labels derived from both human and large language model (LLM) annotations. Building on a stance detection dataset focused on controversial topics, we augment it with document summaries and new LLM-generated labels. We then compare two approaches: a baseline using aggregated hard labels, and a multi-perspective model trained on disaggregated soft labels that capture annotation distributions. Our findings show that multi-perspective models consistently outperform traditional baselines (higher F1-scores), with lower model confidence, reflecting task subjectivity. This work highlights the importance of modeling disagreement and promotes a shift toward more inclusive, perspective-aware NLP systems.
@inbook{MBG2025, author = {Muscato, Benedetta and Bushipaka, Praveen and Gezici, Gizem and Passaro, Lucia and Giannotti, Fosca and Cucinotta, Tommaso}, booktitle = {HHAI 2025}, doi = {10.3233/faia250654}, isbn = {9781643686110}, issn = {1879-8314}, line = {4,5}, month = sep, open_access = {Gold}, pages = {370--384}, publisher = {IOS Press}, title = {Embracing Diversity: A Multi-Perspective Approach with Soft Labels}, visible_on_website = {YES}, year = {2025} }
Towards Building a Trustworthy RAG-Based Chatbot for the Italian Public Administration

Chandana Sree Mala, Christian Maio, Mattia Proietti, Gizem Gezici, Fosca Giannotti, and 3 more authors

Sep 2025

RESEARCH LINE
3

5

Abs Bib HTML

Building a Trustworthy Retrieval-Augmented Generation (RAG) chatbot for Italy’s public sector presents challenges that go beyond selecting an appropriate Large Language Model. A major issue is the retrieval phase, where Italian text embedders often underperform compared to English and multilingual counterparts, hindering precise identification and contextualization of critical information. Regulatory constraints further complicate matters by disallowing closed source or cloud based models, forcing reliance on on-premise or fully open source solutions that may not fully address the linguistic complexities of Italian documents. In our study, we evaluate three embedding approaches using a publicly available Italian dataset: a monolingual Italian approach, a translation based method leveraging English only embedders with backward reference mapping, and a multilingual framework applied to both original and translated texts. Our methodology involves chunking documents into coherent segments, embedding them in a high dimensional semantic space, and measuring retrieval accuracy via top-k similarity searches. Our results indicate that the translation based approach significantly improves retrieval performance over Italian specific models, suggesting that bilingual mapping can effectively address both domain specific challenges and regulatory constraints in developing RAG pipelines for public administration.
@inbook{MDP2025, author = {Mala, Chandana Sree and di Maio, Christian and Proietti, Mattia and Gezici, Gizem and Giannotti, Fosca and Melacci, Stefano and Lenci, Alessandro and Gori, Marco}, booktitle = {HHAI 2025}, doi = {10.3233/faia250637}, isbn = {9781643686110}, issn = {1879-8314}, line = {3,5}, month = sep, open_access = {Gold}, pages = {196--204}, publisher = {IOS Press}, title = {Towards Building a Trustworthy RAG-Based Chatbot for the Italian Public Administration}, visible_on_website = {YES}, year = {2025} }
Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems

Benedetta Muscato, Lucia Passaro, Gizem Gezici, and Fosca Giannotti

In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence , Sep 2025

RESEARCH LINE
4

5

Abs Bib HTML

In the realm of Natural Language Processing (NLP), common approaches for handling human disagreement consist of aggregating annotators’ viewpoints to establish a single ground truth. However, prior studies show that disregarding individual opinions can lead to the side-effect of under-representing minority perspectives, especially in subjective tasks, where annotators may systematically disagree because of their preferences. Recognizing that labels reflect the diverse backgrounds, life experiences, and values of individuals, this study proposes a new multi-perspective approach using soft labels to encourage the development of the next generation of perspective-aware models—more inclusive and pluralistic. We conduct an extensive analysis across diverse subjective text classification tasks including hate speech, irony, abusive language, and stance detection, to highlight the importance of capturing human disagreements, often overlooked by traditional aggregation methods. Results show that the multi-perspective approach not only better approximates human label distributions, as measured by Jensen-Shannon Divergence (JSD), but also achieves superior classification performance (higher F1-scores), outperforming traditional approaches. However, our approach exhibits lower confidence in tasks like irony and stance detection, likely due to the inherent subjectivity present in the texts. Lastly, leveraging Explainable AI (XAI), we explore model uncertainty and uncover meaningful insights into model predictions. All implementation details are available at our github repo.
@inproceedings{MPG2025, author = {Muscato, Benedetta and Passaro, Lucia and Gezici, Gizem and Giannotti, Fosca}, booktitle = {Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence}, collection = {IJCAI-2025}, doi = {10.24963/ijcai.2025/1092}, line = {4,5}, month = sep, open_access = {Gold}, pages = {9827–9835}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, series = {IJCAI-2025}, title = {Perspectives in Play: A Multi-Perspective Approach for More Inclusive NLP Systems}, visible_on_website = {YES}, year = {2025} }

MAINLE: a Multi-Agent, Interactive, Natural Language Local Explainer of Classification Tasks

Paulo Bruno Serafim, Romula Ferrer Filho, STENIO Freitas, Gizem Gezici, Fosca Giannotti, and 2 more authors

Dec 2025

RESEARCH LINE

3

Bib

@misc{SFF2025,
  author = {Serafim, Paulo Bruno and Filho, Romula Ferrer and Freitas, STENIO and Gezici, Gizem and Giannotti, Fosca and Raimondi, Franco and Santos, Alexandre},
  line = {1,3},
  month = dec,
  title = {MAINLE: a Multi-Agent, Interactive, Natural Language Local Explainer of Classification Tasks},
  year = {2025}
}

"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy

Daniele Gambetta, Gizem Gezici, Fosca Giannotti, Dino Pedreschi, Alistair Knott, and 1 more author

Dec 2025

RESEARCH LINE
1

Bib
@misc{GGG2025, author = {Gambetta, Daniele and Gezici, Gizem and Giannotti, Fosca and Pedreschi, Dino and Knott, Alistair and Pappalardo, Luca}, line = {1}, month = dec, title = {"Learning by surprise": a new characterization and mitigation strategy of model collapse in LLM autophagy}, year = {2025} }

2024

Exploring Large Language Models Capabilities to Explain Decision Trees

Paulo Bruno Serafim, Pierluigi Crescenzi, Gizem Gezici, Eleonora Cappuccio, Salvatore Rinzivillo, and 1 more author

Jun 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are widely adopted in Machine Learning tasks due to their operation simplicity and interpretability aspects. However, following the decision process path taken by trees can be difficult in a complex scenario or in a case where a user has no familiarity with them. Prior research showed that converting outcomes to natural language is an accessible way to facilitate understanding for non-expert users in several tasks. More recently, there has been a growing effort to use Large Language Models (LLMs) as a tool for providing natural language texts. In this paper, we examine the proficiency of LLMs to explain decision tree predictions in simple terms through the generation of natural language explanations. By exploring different textual representations and prompt engineering strategies, we identify capabilities that strengthen LLMs as a competent explainer as well as highlight potential challenges and limitations, opening further research possibilities on natural language explanations for decision trees.
@inbook{SGC2024, author = {Serafim, Paulo Bruno and Crescenzi, Pierluigi and Gezici, Gizem and Cappuccio, Eleonora and Rinzivillo, Salvatore and Giannotti, Fosca}, booktitle = {HHAI 2024: Hybrid Human AI Systems for the Social Good}, doi = {10.3233/faia240183}, isbn = {9781643685229}, issn = {1879-8314}, line = {1}, month = jun, open_access = {Gold}, publisher = {IOS Press}, title = {Exploring Large Language Models Capabilities to Explain Decision Trees}, visible_on_website = {YES}, year = {2024} }
Multi-Perspective Stance Detection

Benedetta Muscato, Praveen Bushipaka, Gizem Gezici, Lucia Passaro, and Fosca Giannotti

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

Subjective NLP tasks usually rely on human annotations provided by multiple annotators, whose judgments may vary due to their diverse backgrounds and life experiences. Traditional methods often aggregate multiple annotations into a single ground truth, disregarding the diversity in perspectives that arises from annotator disagreement. In this preliminary study, we examine the effect of including multiple annotations on model accuracy in classification. Our methodology investigates the performance of perspectiveaware classification models in stance detection task and further inspects if annotator disagreement affects the model confidence. The results show that multi-perspective approach yields better classification performance outperforming the baseline which uses the single label. This entails that designing more inclusive perspective-aware AI models is not only an essential first step in implementing responsible and ethical AI, but it can also achieve superior results than using the traditional approaches.
@misc{MBG2024bb, address = {Aachen, Germany}, author = {Muscato, Benedetta and Bushipaka, Praveen and Gezici, Gizem and Passaro, Lucia and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Multi-Perspective Stance Detection}, year = {2024} }
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers

Eleonora Cappuccio, Benedetta Muscato, Laura Pollacci, Marta Marchiori Manerba, Clara Punzi, and 5 more authors

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
@misc{CMP2024, address = {Aachen, Germany}, author = {Cappuccio, Eleonora and Muscato, Benedetta and Pollacci, Laura and Manerba, Marta Marchiori and Punzi, Clara and Mala, Chandana Sree and Lalli, Margherita and Gezici, Gizem and Natilli, Michela and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers}, year = {2024} }
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }
The ethical impact assessment of selling life insurance to titanic passengers

Gezici, Gizem; Mannari, Chiara; Orlandi, and Lorenzo

Dec 2024

RESEARCH LINE
5

Abs Bib

The Artificial Intelligence Act (AIA) is a uniform legal framework to ensure that AI systems within the European Union (EU) are safe and comply with existing law on fundamental rights and constitutional values. The AIA adopts a risk-based approach with the aim of intending to regulate AI systems, especially categorised as high-risk, which have significant harmful impacts on the health, safety and fundamental rights of persons in the Union. The AIA is founded on the Ethics Guidelines of the High-Level Expert Group for Trustworthy AI, which are grounded in fundamental rights and reflect four ethical imperatives in order to ensure ethical and robust AI. While we acknowledge that ethics is not law, we advocate that the analysis of ethical risks can assist us in complying with laws, thereby facilitating the implementation of the AIA requirements. Thus, we first design an AI-driven Decision Support System for individual risk prediction in the insurance domain (categorised as high-risk by the AIA) based on the Titanic case, which is a popular benchmark dataset in machine learning. We then fulfill an ethical impact assessment of the Titanic case study, relying on the four ethical imperatives of respect for human autonomy, prevention of harm, fairness, and explicability, declared by the High-Level Expert Group for Trustworthy AI. In the context of this ethical impact assessment, we also refer to the questions in the ALTAI checklist. Our discussions regarding the ethical impact assessment in the insurance domain demonstrate that ethical principles can intersect but also create tensions (intriguingly, only in this particular context), for which there is no definitive solution. When tensions arise, which may result in unavoidable trade-offs, these trade-offs should be addressed in a rational and methodical manner, paying special attention to the context of the current case study being evaluated.
@misc{GMO2024, author = {Gezici and Mannari, Gizem; and Orlandi, Chiara; and Lorenzo}, line = {5}, month = dec, title = {The ethical impact assessment of selling life insurance to titanic passengers}, year = {2024} }
XAI in healthcare

Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti F.

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

The evolution of Explainable Artificial Intelligence (XAI) within healthcare represents a crucial turn towards more transparent, understandable, and patient-centric AI applications. The main objective is not only to increase the accuracy of AI models but also, and more importantly, to establish user trust in decision support systems through improving their interpretability. This extended abstract outlines the ongoing efforts and advancements of our lab addressing the challenges brought up by complex AI systems in healthcare domain. Currently, there are four main projects: Prostate Imaging Cancer AI, Liver Transplantation & Diabetes, Breast Cancer, and Doctor XAI, and ABELE.
@misc{GMB2024, author = {F., Gezici G.; Metta C; Beretta A.; Pellungrini R.; Rinzivillo S.; Pedreschi D.; Giannotti}, line = {4,5}, month = dec, title = {XAI in healthcare}, year = {2024} }
An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives

Benedetta Muscato, Chandana Sree Mala, Marta Marchiori Manerba, Gizem Gezici, and Fosca Giannotti

Dec 2024

RESEARCH LINE
3

Abs Bib HTML

The varied backgrounds and experiences of human annotators inject different opinions and potential biases into the data, inevitably leading to disagreements. Yet, traditional aggregation methods fail to capture individual judgments since they rely on the notion of a single ground truth. Our aim is to review prior contributions to pinpoint the shortcomings that might cause stereotypical content generation. As a preliminary study, our purpose is to investigate state-of-the-art approaches, primarily focusing on the following two research directions. First, we investigate how adding subjectivity aspects to LLMs might guarantee diversity. We then look into the alignment between humans and LLMs and discuss how to measure it. Considering existing gaps, our review explores possible methods to mitigate the perpetuation of biases targeting specific communities. However, we recognize the potential risk of disseminating sensitive information due to the utilization of socio-demographic data in the training process. These considerations underscore the inclusion of diverse perspectives while taking into account the critical importance of implementing robust safeguards to protect individuals’ privacy and prevent the inadvertent propagation of sensitive information.
@misc{MMM2024, address = {Torino, Italia}, author = {Muscato, Benedetta and Mala, Chandana Sree and Manerba, Marta Marchiori and Gezici, Gizem and Giannotti, Fosca}, line = {3}, month = dec, pages = {49--55}, publisher = {ELRA and ICCL}, title = {An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives}, year = {2024} }

2023

Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic

Clara Punzi, Aleksandra Maslennikova, Gizem Gezici, Roberto Pellungrini, and Fosca Giannotti

Dec 2023

RESEARCH LINE
1

4

Abs Bib HTML

Pandemic vaccination campaigns must account for vaccine skepticism as an obstacle to overcome. Using machine learning to identify behavioral and psychological patterns in public survey datasets can provide valuable insights and inform vaccination campaigns based on empirical evidence. However, we argue that the adoption of local and global explanation methodologies can provide additional support to health practitioners by suggesting personalized communication strategies and revealing potential demographic, social, or structural barriers to vaccination requiring systemic changes. In this paper, we first implement a chain classification model for the adoption of the vaccine during the H1N1 influenza outbreak taking seasonal vaccination information into account, and then compare it with a binary classifier for vaccination to better understand the overall patterns in the data. Following that, we derive and compare global explanations using post-hoc methodologies and interpretable-by-design models. Our findings indicate that socio-demographic factors play a distinct role in the H1N1 vaccination as compared to the general vaccination. Nevertheless, medical recommendation and health insurance remain significant factors for both vaccinations. Then, we concentrated on the subpopulation of individuals who did not receive an H1N1 vaccination despite being at risk of developing severe symptoms. In an effort to assist practitioners in providing effective recommendations to patients, we present rules and counterfactuals for the selected instances based on local explanations. Finally, we raise concerns regarding gender and racial disparities in healthcare access by analysing the interaction effects of sensitive attributes on the model’s output.
@inbook{PMG2023, author = {Punzi, Clara and Maslennikova, Aleksandra and Gezici, Gizem and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-031-44067-0_31}, isbn = {9783031440670}, issn = {1865-0937}, line = {1,4}, open_access = {Gold}, pages = {621–635}, publisher = {Springer Nature Switzerland}, title = {Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic}, visible_on_website = {YES}, year = {2023} }

2025

A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems

G. Barlacchi, M. Lalli, E. Ferragina, F. Giannotti, and L. Pappalardo

Dec 2025

RESEARCH LINE
4

Abs Bib HTML

Recommender systems continuously interact with users, creating feedback loops that shape both individual behavior and collective market dynamics. This paper introduces a simulation framework to model these loops in online retail environments, where recommenders are periodically retrained on evolving user–item interactions. Using the Amazon e-Commerce dataset, we analyze how different recommendation algorithms influence diversity, purchase concentration, and user homogenization over time. Results reveal a systematic trade-off: while the feedback loop increases individual diversity, it simultaneously reduces collective diversity and concentrates demand on a few popular items. Moreover, for some recommender systems, the feedback loop increases user homogenization over time, making user purchase profiles increasingly similar. These findings underscore the need for recommender designs that balance personalization with long-term diversity.
@misc{BLF2025, author = {Barlacchi, G. and Lalli, M. and Ferragina, E. and Giannotti, F. and Pappalardo, L.}, doi = {10.48550/arXiv.2510.14857}, line = {4}, month = dec, title = {A Simulation Framework for Studying Systemic Effects of Feedback Loops in Recommender Systems}, year = {2025} }

2024

Inference through innovation processes tested in the authorship attribution task

Giulio Tani Raffaelli, Margherita Lalli, and Francesca Tria

Communications Physics, Sep 2024

RESEARCH LINE
1

Abs Bib HTML

Urn models for innovation capture fundamental empirical laws shared by several real-world processes. The so-called urn model with triggering includes, as particular cases, the urn representation of the two-parameter Poisson-Dirichlet process and the Dirichlet process, seminal in Bayesian non-parametric inference. In this work, we leverage this connection to introduce a general approach for quantifying closeness between symbolic sequences and test it within the framework of the authorship attribution problem. The method demonstrates high accuracy when compared to other related methods in different scenarios, featuring a substantial gain in computational efficiency and theoretical transparency. Beyond the practical convenience, this work demonstrates how the recently established connection between urn models and non-parametric Bayesian inference can pave the way for designing more efficient inference methods. In particular, the hybrid approach that we propose allows us to relax the exchangeability hypothesis, which can be particularly relevant for systems exhibiting complex correlation patterns and non-stationary dynamics. (arXiv)
@article{TLT2024, author = {Tani Raffaelli, Giulio and Lalli, Margherita and Tria, Francesca}, doi = {10.1038/s42005-024-01714-6}, issn = {2399-3650}, journal = {Communications Physics}, line = {1}, month = sep, number = {1}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Inference through innovation processes tested in the authorship attribution task}, visible_on_website = {YES}, volume = {7}, year = {2024} }
Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers

Eleonora Cappuccio, Benedetta Muscato, Laura Pollacci, Marta Marchiori Manerba, Clara Punzi, and 5 more authors

Dec 2024

RESEARCH LINE
4

5

Abs Bib HTML

How newspapers cover news significantly impacts how facts are understood, perceived, and processed by the public. This is especially crucial when serious crimes are reported, e.g., in the case of femicides, where the description of the perpetrator and the victim builds a strong, often polarized opinion of this severe societal issue. This paper presents FMNews, a new dataset of articles reporting femicides extracted from Italian newspapers. Our core contribution aims to promote the development of a deeper framing and awareness of the phenomenon through an original resource available and accessible to the research community, facilitating further analyses on the topic. The paper also provides a preliminary study of the resulting collection through several example use cases and scenarios.
@misc{CMP2024, address = {Aachen, Germany}, author = {Cappuccio, Eleonora and Muscato, Benedetta and Pollacci, Laura and Manerba, Marta Marchiori and Punzi, Clara and Mala, Chandana Sree and Lalli, Margherita and Gezici, Gizem and Natilli, Michela and Giannotti, Fosca}, line = {4,5}, month = dec, title = {Beyond Headlines: A Corpus of Femicides News Coverage in Italian Newspapers}, year = {2024} }
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions

Luca Pappalardo, Emanuele Ferragina, Salvatore Citraro, Giuliano Cornacchia, Mirco Nanni, and 9 more authors

Dec 2024

RESEARCH LINE
3

4

5

Abs Bib HTML

Recommendation systems and assistants (in short, recommenders) are ubiquitous in online platforms and influence most actions of our day-to-day lives, suggesting items or providing solutions based on users’ preferences or requests. This survey analyses the impact of recommenders in four human-AI ecosystems: social media, online retail, urban mapping and generative AI ecosystems. Its scope is to systematise a fast-growing field in which terminologies employed to classify methodologies and outcomes are fragmented and unsystematic. We follow the customary steps of qualitative systematic review, gathering 144 articles from different disciplines to develop a parsimonious taxonomy of: methodologies employed (empirical, simulation, observational, controlled), outcomes observed (concentration, model collapse, diversity, echo chamber, filter bubble, inequality, polarisation, radicalisation, volume), and their level of analysis (individual, item, model, and systemic). We systematically discuss all findings of our survey substantively and methodologically, highlighting also potential avenues for future research. This survey is addressed to scholars and practitioners interested in different human-AI ecosystems, policymakers and institutional stakeholders who want to understand better the measurable outcomes of recommenders, and tech companies who wish to obtain a systematic view of the impact of their recommenders.
@misc{PFC2024, author = {Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino}, doi = {10.48550/arXiv.2407.01630}, line = {3,4,5}, month = dec, publisher = {arXiv}, title = {A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions}, year = {2024} }

2025

Ensemble Counterfactual Explanations for Churn Analysis

Samuele Tonati, Marzio Di Vece, Roberto Pellungrini, and Fosca Giannotti

Dec 2025

RESEARCH LINE
1

Abs Bib HTML

Counterfactual explanations play a crucial role in interpreting and understanding the decision-making process of complex machine learning models, offering insights into why a particular prediction was made and how it could be altered. However, individual counterfactual explanations generated by different methods may vary significantly in terms of their quality, diversity, and coherence to the black-box prediction. This is especially important in financial applications such as churn analysis, where customer retention officers could explore different approaches and solutions with the clients to prevent churning. The officer’s capability to modify and explore different explanations is pivotal to his ability to provide feasible solutions. To address this challenge, we propose an evaluation framework through the implementation of an ensemble approach that combines state-of-the-art counterfactual generation methods and a linear combination score of desired properties to select the most appropriate explanation. We conduct our experiments on three publicly available churn datasets in different domains. Our experimental results demonstrate that the ensemble of counterfactual explanations provides more diverse and comprehensive insights into model behavior compared to individual methods alone that suffer from specific weaknesses. By aggregating, evaluating, and selecting multiple explanations, our approach enhances the diversity of the explanation, highlights common patterns, and mitigates the limitations of any single method, offering to the user the ability to tweak the explanation properties to their needs.
@inbook{TDP2025, author = {Tonati, Samuele and Di Vece, Marzio and Pellungrini, Roberto and Giannotti, Fosca}, booktitle = {Discovery Science}, doi = {10.1007/978-3-031-78980-9_21}, isbn = {9783031789809}, issn = {1611-3349}, line = {1}, open_access = {NO}, pages = {332–347}, publisher = {Springer Nature Switzerland}, title = {Ensemble Counterfactual Explanations for Churn Analysis}, visible_on_website = {YES}, year = {2025} }

2024

Commodity-specific triads in the Dutch inter-industry production network

Marzio Di Vece, Frank P. Pijpers, and Diego Garlaschelli

Scientific Reports, Feb 2024

RESEARCH LINE
4

Abs Bib HTML

Triadic motifs are the smallest building blocks of higher-order interactions in complex networks and can be detected as over-occurrences with respect to null models with only pair-wise interactions. Recently, the motif structure of production networks has attracted attention in light of its possible role in the propagation of economic shocks. However, its characterization at the level of individual commodities is still poorly understood. Here we analyze both binary and weighted triadic motifs in the Dutch inter-industry production network disaggregated at the level of 187 commodity groups, which Statistics Netherlands reconstructed from National Accounts registers, surveys and known empirical data. We introduce appropriate null models that filter out node heterogeneity and the strong effects of link reciprocity and find that, while the aggregate network that overlays all products is characterized by a multitude of triadic motifs, most single-product layers feature no significant motif, and roughly 85% of the layers feature only two motifs or less. This result paves the way for identifying a simple ‘triadic fingerprint’ of each commodity and for reconstructing most product-specific networks from partial information in a pairwise fashion by controlling for their reciprocity structure. We discuss how these results can help statistical bureaus identify fine-grained information in structural analyses of interest for policymakers.
@article{DPG2024, author = {Di Vece, Marzio and Pijpers, Frank P. and Garlaschelli, Diego}, doi = {10.1038/s41598-024-53655-3}, issn = {2045-2322}, journal = {Scientific Reports}, line = {4}, month = feb, number = {1}, open_access = {Gold}, publisher = {Springer Science and Business Media LLC}, title = {Commodity-specific triads in the Dutch inter-industry production network}, visible_on_website = {YES}, volume = {14}, year = {2024} }

2023

Deterministic, quenched, and annealed parameter estimation for heterogeneous network models

Marzio Di Vece, Diego Garlaschelli, and Tiziano Squartini

Physical Review E, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

At least two, different approaches to define and solve statistical models for the analysis of economic systems exist: the typical, econometric one, interpreting the Gravity Model specification as the expected link weight of an arbitrary probability distribution, and the one rooted into statistical physics, constructing maximum-entropy distributions constrained to satisfy certain network properties. In a couple of recent, companion papers they have been successfully integrated within the framework induced by the constrained minimisation of the Kullback-Leibler divergence: specifically, two, broad classes of models have been devised, i.e. the integrated and the conditional ones, defined by different, probabilistic rules to place links, load them with weights and turn them into proper, econometric prescriptions. Still, the recipes adopted by the two approaches to estimate the parameters entering into the definition of each model differ. In econometrics, a likelihood that decouples the binary and weighted parts of a model, treating a network as deterministic, is typically maximised; to restore its random character, two alternatives exist: either solving the likelihood maximisation on each configuration of the ensemble and taking the average of the parameters afterwards or taking the average of the likelihood function and maximising the latter one. The difference between these approaches lies in the order in which the operations of averaging and maximisation are taken — a difference that is reminiscent of the quenched and annealed ways of averaging out the disorder in spin glasses. The results of the present contribution, devoted to comparing these recipes in the case of continuous, conditional network models, indicate that the annealed estimation recipe represents the best alternative to the deterministic one. (ar5iv.labs.arxiv.org)
@article{DGS2023, address = {Aachen, Germany}, author = {Di Vece, Marzio and Garlaschelli, Diego and Squartini, Tiziano}, doi = {10.1103/physreve.108.054301}, issn = {2470-0053}, journal = {Physical Review E}, line = {1}, month = nov, number = {5}, open_access = {Gold}, publisher = {American Physical Society (APS)}, title = {Deterministic, quenched, and annealed parameter estimation for heterogeneous network models}, visible_on_website = {YES}, volume = {108}, year = {2023} }

2025

Disentangled and Self-Explainable Node Representation Learning

Simone Piaggesi, André Panisson, and Megha Khosla

Dec 2025

RESEARCH LINE

Bib HTML

@misc{PPK2025,
  author = {Piaggesi, Simone and Panisson, André and Khosla, Megha},
  doi = {10.48550/arXiv.2410.21043},
  line = {1},
  month = dec,
  title = {Disentangled and Self-Explainable Node Representation Learning},
  year = {2025}
}

Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding

Simone Piaggesi, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Dec 2025

RESEARCH LINE
1

Bib HTML
@misc{PGG2025, author = {Piaggesi, Simone and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, line = {1}, month = dec, title = {Explanations Go Linear: Post-hoc Explainability for Tabular Data with Interpretable Meta-Encoding}, year = {2025} }

2024

DINE: Dimensional Interpretability of Node Embeddings

Simone Piaggesi, Megha Khosla, André Panisson, and Avishek Anand

IEEE Transactions on Knowledge and Data Engineering, Dec 2024

RESEARCH LINE
1

2

Abs Bib HTML

Graphs are ubiquitous due to their flexibility in representing social and technological systems as networks of interacting elements. Graph representation learning methods, such as node embeddings, are powerful approaches to map nodes into a latent vector space, allowing their use for various graph tasks. Despite their success, only few studies have focused on explaining node embeddings locally. Moreover, global explanations of node embeddings remain unexplored, limiting interpretability and debugging potentials. We address this gap by developing human-understandable explanations for dimensions in node embeddings. Towards that, we first develop new metrics that measure the global interpretability of embedding vectors based on the marginal contribution of the embedding dimensions to predicting graph structure. We say that an embedding dimension is more interpretable if it can faithfully map to an understandable sub-structure in the input graph – like community structure. Having observed that standard node embeddings have low interpretability, we then introduce DINE (Dimension-based Interpretable Node Embedding), a novel approach that can retrofit existing node embeddings by making them more interpretable without sacrificing their task performance. We conduct extensive experiments on synthetic and real-world graphs and show that we can simultaneously learn highly interpretable node embeddings with effective performance in link prediction.
@article{PKP2024, author = {Piaggesi, Simone and Khosla, Megha and Panisson, André and Anand, Avishek}, doi = {10.1109/tkde.2024.3425460}, issn = {2326-3865}, journal = {IEEE Transactions on Knowledge and Data Engineering}, line = {1,2}, month = dec, number = {12}, open_access = {Gold}, pages = {7986–7997}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {DINE: Dimensional Interpretability of Node Embeddings}, visible_on_website = {YES}, volume = {36}, year = {2024} }
Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space

Simone Piaggesi, Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

IEEE Access, Dec 2024

RESEARCH LINE
1

Abs Bib HTML

We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities similarly to well-known dimensionality reduction techniques. Our approach introduces a transparent latent space optimized for interpretability of both counterfactual and prototypical explanations for tabular data. The approach enables the easy extraction of local and global explanations and ensures that the latent space preserves similarity relations, enabling meaningful prototypical and counterfactual examples for any classifier.
@article{PBG2024, author = {Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1109/access.2024.3496114}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, open_access = {Gold}, pages = {168983–169000}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space}, visible_on_website = {YES}, volume = {12}, year = {2024} }

2025

Mathematical Foundation of Interpretable Equivariant Surrogate Models

Jacopo Joy Colombini, Filippo Bonchi, Francesco Giannini, Fosca Giannotti, Roberto Pellungrini, and 1 more author

Oct 2025

RESEARCH LINE
1

Abs Bib HTML

This paper introduces a rigorous mathematical framework for neural network explainability, and more broadly for the explainability of equivariant operators called Group Equivariant Operators (GEOs), based on Group Equivariant Non-Expansive Operators (GENEOs) transformations. The central concept involves quantifying the distance between GEOs by measuring the non-commutativity of specific diagrams. Additionally, the paper proposes a definition of interpretability of GEOs according to a complexity measure that can be defined according to each user’s preferences. Moreover, we explore the formal properties of this framework and show how it can be applied in classical machine learning scenarios, like image classification with convolutional neural networks. (SpringerLink)
@inbook{CBG2025, address = {Istanbul, Turkey}, author = {Colombini, Jacopo Joy and Bonchi, Filippo and Giannini, Francesco and Giannotti, Fosca and Pellungrini, Roberto and Frosini, Patrizio}, booktitle = {Explainable Artificial Intelligence}, doi = {10.1007/978-3-032-08324-1_13}, isbn = {9783032083241}, issn = {1865-0937}, line = {1}, month = oct, open_access = {Gold}, pages = {294–318}, publisher = {Springer Nature Switzerland}, title = {Mathematical Foundation of Interpretable Equivariant Surrogate Models}, visible_on_website = {YES}, year = {2025} }
Categorical Explaining Functors: Ensuring Coherence in Logical Explanations

Stefano Fioravanti, Francesco Giannini, Pietro Barbiero, Paolo Frazzetto, Roberto Confalonieri, and 2 more authors

In Proceedings of the TwentySecond International Conference on Principles of Knowledge Representation and Reasoning , Nov 2025

RESEARCH LINE
1

2

Abs Bib HTML

The most common methods in explainable artificial intelligence are post-hoc techniques which identify the most relevant features used by pretrained opaque models. Some of the most advanced post-hoc methods can generate explanations that account for the mutual interactions of input features in the form of logic rules. However, these methods frequently fail to guarantee the consistency of the extracted explanations with the model’s underlying reasoning. To bridge this gap, we propose a theoretically grounded approach to ensure coherence and fidelity of the extracted explanations, moving beyond the limitations of current heuristic-based approaches. To this end, drawing from category theory, we introduce an explaining functor which structurally preserves logical entailment between the explanation and the opaque model’s reasoning. As a proof of concept, we validate the proposed theoretical constructions on a synthetic benchmark verifying how the proposed approach significantly mitigates the generation of contradictory or unfaithful explanations. (arXiv)
@inproceedings{FGB2025, address = {California}, author = {Fioravanti, Stefano and Giannini, Francesco and Barbiero, Pietro and Frazzetto, Paolo and Confalonieri, Roberto and Zanasi, Fabio and Navarin, Nicolò}, booktitle = {Proceedings of the TwentySecond International Conference on Principles of Knowledge Representation and Reasoning}, collection = {KR-2025}, doi = {10.24963/kr.2025/30}, issn = {978-1-956792-08-9}, line = {1,2}, month = nov, open_access = {Gold}, pages = {306–315}, publisher = {International Joint Conferences on Artificial Intelligence Organization}, series = {KR-2025}, title = {Categorical Explaining Functors: Ensuring Coherence in Logical Explanations}, visible_on_website = {YES}, year = {2025} }

Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts

Andrea Pugnana, Riccardo Massidda, Francesco Giannini, Pietro Barbiero, Mateo Espinosa Zarlenga, and 4 more authors

Dec 2025

RESEARCH LINE

Bib

@misc{PMG2025,
  author = {Pugnana, Andrea and Massidda, Riccardo and Giannini, Francesco and Barbiero, Pietro and Zarlenga, Mateo Espinosa and Pellungrini, Roberto and Dominici, Gabriele and Giannotti, Fosca and Bacciu, Davide},
  line = {1,2},
  month = dec,
  title = {Deferring Concept Bottleneck Models: Learning to Defer Interventions to Inaccurate Experts},
  year = {2025}
}

DeepProofLog: Efficient Proving in Deep Stochastic Logic Programs

Ying Jiao, Rodrigo Castellano Ontiveros, Luc De Raedt, Marco Gori, Francesco Giannini, and 2 more authors

Dec 2025

RESEARCH LINE