Publications | Xai Project

2024

Fiper: a Visual-based Explanation Combining Rules and Feature Importance

Eleonora Cappuccio, Daniele Fadda, Rosa Lanzilotti, and Salvatore Rinzivillo

arXiv preprint arXiv:2404.16903, Dec 2024

RESEARCH LINE
3

Abs Bib HTML

Artificial Intelligence algorithms have now become pervasive in multiple high-stakes domains. However, their internal logic can be obscure to humans. Explainable Artificial Intelligence aims to design tools and techniques to illustrate the predictions of the so-called black-box algorithms. The Human-Computer Interaction community has long stressed the need for a more user-centered approach to Explainable AI. This approach can benefit from research in user interface, user experience, and visual analytics. This paper proposes a visual-based method to illustrate rules paired with feature importance. A user study with 15 participants was conducted comparing our visual method with the original output of the algorithm and textual representation to test its effectiveness with users.
@article{cappuccio2024fiper, author = {Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore}, journal = {arXiv preprint arXiv:2404.16903}, line = {3}, month = dec, title = {Fiper: a Visual-based Explanation Combining Rules and Feature Importance}, year = {2024}, }
Generative Model for Decision Trees

Riccardo Guidotti, Anna Monreale, Mattia Setzu, and Giulia Volpi

In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada , Dec 2024

RESEARCH LINE
1

Abs Bib HTML

Decision trees are among the most popular supervised mod- els due to their interpretability and knowledge representation resembling human reasoning. Commonly-used decision tree induction algorithms are based on greedy top-down strategies. Although these approaches are known to be an efficient heuris- tic, the resulting trees are only locally optimal and tend to have overly complex structures. On the other hand, optimal decision tree algorithms attempt to create an entire decision tree at once to achieve global optimality. We place our proposal between these approaches by designing a generative model for deci- sion trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. Then, it adopts a genetic procedure to explore such latent space to find a compact decision tree with good predictive performance. We compare our proposal against clas- sical tree induction methods, optimal approaches, and ensem- ble models. The results show that our proposal can generate accurate and shallow, i.e., interpretable, decision trees.
@inproceedings{DBLP:conf/aaai/GuidottiMSV24, author = {Guidotti, Riccardo and Monreale, Anna and Setzu, Mattia and Volpi, Giulia}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/conf/aaai/GuidottiMSV24.bib}, booktitle = {Thirty-Eighth {AAAI} Conference on Artificial Intelligence, {AAAI} 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, {IAAI} 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, {EAAI} 2014, February 20-27, 2024, Vancouver, Canada}, doi = {10.1609/AAAI.V38I19.30104}, editor = {Wooldridge, Michael J. and Dy, Jennifer G. and Natarajan, Sriraam}, line = {1}, month = dec, pages = {21116--21124}, publisher = {{AAAI} Press}, timestamp = {Tue, 02 Apr 2024 16:32:09 +0200}, title = {Generative Model for Decision Trees}, year = {2024} }
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers

Guillermo Fernández, Riccardo Guidotti, Fosca Giannotti, Mattia Setzu, Juan A. Aledo, and 2 more authors

In Advances in Intelligent Data Analysis XXII - 22nd International Symposium on Intelligent Data Analysis, IDA 2024, Stockholm, Sweden, April 24-26, 2024, Proceedings, Part II , Dec 2024

RESEARCH LINE
1

2

Abs Bib HTML

The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inproceedings{DBLP:conf/ida/FernandezGGSAGP24, author = {Fern{\'{a}}ndez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and G{\'{a}}mez, Jos{\'{e}} A. and Puerta, Jos{\'{e}} Miguel}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/conf/ida/FernandezGGSAGP24.bib}, booktitle = {Advances in Intelligent Data Analysis {XXII} - 22nd International Symposium on Intelligent Data Analysis, {IDA} 2024, Stockholm, Sweden, April 24-26, 2024, Proceedings, Part {II}}, doi = {10.1007/978-3-031-58553-1\_16}, editor = {Miliou, Ioanna and Piatkowski, Nico and Papapetrou, Panagiotis}, line = {1,2}, month = dec, pages = {197--209}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, timestamp = {Fri, 17 May 2024 21:42:12 +0200}, title = {FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers}, volume = {14642}, year = {2024} }
A Frank System for Co-Evolutionary Hybrid Decision-Making

Federico Mazzoni, Riccardo Guidotti, and Alessio Malizia

In Advances in Intelligent Data Analysis XXII - 22nd International Symposium on Intelligent Data Analysis, IDA 2024, Stockholm, Sweden, April 24-26, 2024, Proceedings, Part II , Dec 2024

RESEARCH LINE
1

4

Abs Bib HTML

We introduce FRANK, a human-in-the-loop system for co-evolutionary hybrid decision-making aiding the user to label records from an un-labeled dataset. FRANK employs incremental learning to “evolve” in parallel with the user’s decisions, by training an interpretable machine learning model on the records labeled by the user. Furthermore, advances state-of-the-art approaches by offering inconsistency controls, explanations, fairness checks, and bad-faith safeguards simultaneously. We evaluate our proposal by simulating the users’ behavior with various levels of expertise and reliance on FRANK’s suggestions. The experiments show that FRANK’s intervention leads to improvements in the accuracy and the fairness of the decisions.
@inproceedings{DBLP:conf/ida/MazzoniGM24, author = {Mazzoni, Federico and Guidotti, Riccardo and Malizia, Alessio}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/conf/ida/MazzoniGM24.bib}, booktitle = {Advances in Intelligent Data Analysis {XXII} - 22nd International Symposium on Intelligent Data Analysis, {IDA} 2024, Stockholm, Sweden, April 24-26, 2024, Proceedings, Part {II}}, doi = {10.1007/978-3-031-58553-1\_19}, editor = {Miliou, Ioanna and Piatkowski, Nico and Papapetrou, Panagiotis}, line = {1,4}, month = dec, pages = {236--248}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, timestamp = {Tue, 07 May 2024 20:09:13 +0200}, title = {A Frank System for Co-Evolutionary Hybrid Decision-Making}, volume = {14642}, year = {2024} }
AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems

Clara Punzi, Roberto Pellungrini, Mattia Setzu, Fosca Giannotti, and Dino Pedreschi

CoRR, Dec 2024

RESEARCH LINE
1

2

Abs Bib HTML

Everyday we increasingly rely on machine learning models to automate and support high-stake tasks and decisions. This growing presence means that humans are now constantly interacting with machine learning-based systems, training and using models everyday. Several different techniques in computer science literature account for the human interaction with machine learning systems, but their classification is sparse and the goals varied. This survey proposes a taxonomy of Hybrid Decision Making Systems, providing both a conceptual and technical framework for understanding how current computer science literature models interaction between humans and machines.
@article{DBLP:journals/corr/abs-2402-06287, author = {Punzi, Clara and Pellungrini, Roberto and Setzu, Mattia and Giannotti, Fosca and Pedreschi, Dino}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/journals/corr/abs-2402-06287.bib}, doi = {10.48550/ARXIV.2402.06287}, eprint = {2402.06287}, eprinttype = {arXiv}, journal = {CoRR}, line = {1,2}, month = dec, timestamp = {Tue, 07 May 2024 20:16:53 +0200}, title = {AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems}, volume = {abs/2402.06287}, year = {2024} }
FairBelief - Assessing Harmful Beliefs in Language Models

Mattia Setzu, Marta Marchiori Manerba, Pasquale Minervini, and Debora Nozza

CoRR, Dec 2024

RESEARCH LINE
2

5

Abs Bib HTML

Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing. This paper proposes FairBelief, an analytical approach to capture and assess beliefs, i.e., propositions that an LM may embed with different degrees of confidence and that covertly influence its predictions. With FairBelief, we leverage prompting to study the behavior of several state-of-the-art LMs across different previously neglected axes, such as model scale and likelihood, assessing predictions on a fairness dataset specifically designed to quantify LMs’ outputs’ hurtfulness. Finally, we conclude with an in-depth qualitative assessment of the beliefs emitted by the models. We apply FairBelief to English LMs, revealing that, although these architectures enable high performances on diverse natural language processing tasks, they show hurtful beliefs about specific genders. Interestingly, training procedure and dataset, model scale, and architecture induce beliefs of different degrees of hurtfulness.
@article{DBLP:journals/corr/abs-2402-17389, author = {Setzu, Mattia and Manerba, Marta Marchiori and Minervini, Pasquale and Nozza, Debora}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/journals/corr/abs-2402-17389.bib}, doi = {10.48550/ARXIV.2402.17389}, eprint = {2402.17389}, eprinttype = {arXiv}, journal = {CoRR}, line = {2,5}, month = dec, timestamp = {Tue, 07 May 2024 20:15:26 +0200}, title = {FairBelief - Assessing Harmful Beliefs in Language Models}, volume = {abs/2402.17389}, year = {2024} }
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

Diagnostics, Dec 2024

RESEARCH LINE
3

4

Abs Bib HTML

A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{diagnostics14070753, article-number = {753}, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.3390/diagnostics14070753}, issn = {2075-4418}, journal = {Diagnostics}, line = {3,4}, month = dec, number = {7}, pubmedid = {38611666}, title = {Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification}, volume = {14}, year = {2024} }
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence

Carlo Metta, Andrea Beretta, Roberto Pellungrini, Salvatore Rinzivillo, and Fosca Giannotti

Bioengineering, Dec 2024

RESEARCH LINE
1

4

Abs Bib HTML

This paper focuses on the use of local Explainable Artificial Intelligence (XAI) methods, particularly the Local Rule-Based Explanations (LORE) technique, within healthcare and medical settings. It emphasizes the critical role of interpretability and transparency in AI systems for diagnosing diseases, predicting patient outcomes, and creating personalized treatment plans. While acknowledging the complexities and inherent trade-offs between interpretability and model performance, our work underscores the significance of local XAI methods in enhancing decision-making processes in healthcare. By providing granular, case-specific insights, local XAI methods like LORE enhance physicians’ and patients’ understanding of machine learning models and their outcome. Our paper reviews significant contributions to local XAI in healthcare, highlighting its potential to improve clinical decision making, ensure fairness, and comply with regulatory standards.
@article{metta2024towards, author = {Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca}, journal = {Bioengineering}, line = {1,4}, month = dec, number = {4}, pages = {369}, publisher = {MDPI}, title = {Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence}, volume = {11}, year = {2024}, }
An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives

Benedetta Muscato, Chandana Sree Mala, Marta Marchiori Manerba, Gizem Gezici, and Fosca Giannotti

In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024 , May 2024

RESEARCH LINE
1

2

Abs Bib HTML

The varied backgrounds and experiences of human annotators inject different opinions and potential biases into the data, inevitably leading to disagreements. Yet, traditional aggregation methods fail to capture individual judgments since they rely on the notion of a single ground truth. Our aim is to review prior contributions to pinpoint the shortcomings that might cause stereotypical content generation. As a preliminary study, our purpose is to investigate state-of-the-art approaches, primarily focusing on the following two research directions. First, we investigate how adding subjectivity aspects to LLMs might guarantee diversity. We then look into the alignment between humans and LLMs and discuss how to measure it. Considering existing gaps, our review explores possible methods to mitigate the perpetuation of biases targeting specific communities. However, we recognize the potential risk of disseminating sensitive information due to the utilization of socio-demographic data in the training process. These considerations underscore the inclusion of diverse perspectives while taking into account the critical importance of implementing robust safeguards to protect individuals’ privacy and prevent the inadvertent propagation of sensitive information.
@inproceedings{muscato-etal-2024-overview, address = {Torino, Italia}, author = {Muscato, Benedetta and Mala, Chandana Sree and Marchiori Manerba, Marta and Gezici, Gizem and Giannotti, Fosca}, booktitle = {Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024}, editor = {Abercrombie, Gavin and Basile, Valerio and Bernadi, Davide and Dudy, Shiran and Frenda, Simona and Havens, Lucy and Tonelli, Sara}, line = {1,2}, month = may, pages = {49--55}, publisher = {ELRA and ICCL}, title = {An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives}, year = {2024} }

2023

The Importance of Time in Causal Algorithmic Recourse

Isacco Beretta, and Martina Cinquini

In World Conference on Explainable Artificial Intelligence , Dec 2023

RESEARCH LINE
2

Abs Bib HTML

The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inproceedings{beretta2023importance, author = {Beretta, Isacco and Cinquini, Martina}, booktitle = {World Conference on Explainable Artificial Intelligence}, line = {2}, month = dec, organization = {Springer}, pages = {283--298}, title = {The Importance of Time in Causal Algorithmic Recourse}, year = {2023}, }
Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{Bodria_2023, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1}, month = jun, number = {5}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, volume = {37}, year = {2023} }
Benchmarking and survey of explanation methods for black box models

Francesco Bodria, Fosca Giannotti, Riccardo Guidotti, Francesca Naretto, Dino Pedreschi, and 1 more author

Data Mining and Knowledge Discovery, Jun 2023

RESEARCH LINE
1

Abs Bib HTML

The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{Bodria_2023_Benchmarking, author = {Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore}, doi = {10.1007/s10618-023-00933-9}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1}, month = jun, number = {5}, pages = {1719–1778}, publisher = {Springer Science and Business Media LLC}, title = {Benchmarking and survey of explanation methods for black box models}, volume = {37}, year = {2023} }
Handling missing values in local post-hoc explainability

Martina Cinquini, Fosca Giannotti, Riccardo Guidotti, and Andrea Mattei

In World Conference on Explainable Artificial Intelligence , Dec 2023

RESEARCH LINE
1

Abs Bib HTML

Missing data are quite common in real scenarios when using Artificial Intelligence (AI) systems for decision-making with tabular data and effectively handling them poses a significant challenge for such systems. While some machine learning models used by AI systems can tackle this problem, the existing literature lacks post-hoc explainability approaches able to deal with predictors that encounter missing data. In this paper, we extend a widely used local model-agnostic post-hoc explanation approach that enables explainability in the presence of missing values by incorporating state-of-the-art imputation methods within the explanation process. Since our proposal returns explanations in the form of feature importance, the user will be aware also of the importance of a missing value in a given record for a particular prediction. Extensive experiments show the effectiveness of the proposed method with respect to some baseline solutions relying on traditional data imputation.
@inproceedings{cinquini2023handling, author = {Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo and Mattei, Andrea}, booktitle = {World Conference on Explainable Artificial Intelligence}, line = {1}, month = dec, organization = {Springer}, pages = {256--278}, title = {Handling missing values in local post-hoc explainability}, year = {2023}, }
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories

Francesca Naretto, Roberto Pellungrini, Salvatore Rinzivillo, and Daniele Fadda

In Discovery Science - 26th International Conference, DS 2023, Porto, Portugal, October 9-11, 2023, Proceedings , Dec 2023

RESEARCH LINE
3

5

Abs Bib HTML

Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across various domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, accurately identifying privacy risks is essential before deciding to release it to the public. Recent work has proposed the use of machine learning models for predicting privacy risk on raw mobility trajectories and the use of SHAP for risk explanation. However, applying SHAP to mobility data results in explanations that are of limited use both for privacy experts and end-users. In this work, we present a novel version of the EXPERT privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification, as ROCKET and INCEPTIONTIME, to improve risk prediction while reducing computation time. Additionally, we address two key issues with SHAP explanation on mobility data: first, we devise an entropy-based mask to efficiently compute SHAP values for privacy risk in mobility data; second, we develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of SHAP values and privacy risk.
@inproceedings{DBLP:conf/dis/NarettoPRF23, author = {Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/conf/dis/NarettoPRF23.bib}, booktitle = {Discovery Science - 26th International Conference, {DS} 2023, Porto, Portugal, October 9-11, 2023, Proceedings}, doi = {10.1007/978-3-031-45275-8\_22}, editor = {Bifet, Albert and Lorena, Ana Carolina and Ribeiro, Rita P. and Gama, Jo{\~{a}}o and Abreu, Pedro H.}, line = {3,5}, month = dec, pages = {325--340}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, timestamp = {Wed, 01 Nov 2023 08:59:01 +0100}, title = {{EXPHLOT:} EXplainable Privacy Assessment for Human LOcation Trajectories}, volume = {14276}, year = {2023} }
Declarative Reasoning on Explanations Using Constraint Logic Programming

Laura State, Salvatore Ruggieri, and Franco Turini

CoRR, Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Explaining opaque Machine Learning (ML) models is an increasingly relevant problem. Current explanation in AI (XAI) methods suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of abstraction and interactivity with the user. We propose REASONX, an explanation method based on Constraint Logic Programming (CLP). REASONX can provide declarative, interactive explanations for decision trees, which can be the ML models under analysis or global/local surrogate models of any black-box model. Users can express background or common sense knowledge using linear constraints and MILP optimization over features of factual and contrastive instances, and interact with the answer constraints at different levels of abstraction through constraint projection. We present here the architecture of REASONX, which consists of a Python layer, closer to the user, and a CLP layer. REASONX’s core execution engine is a Prolog meta-program with declarative semantics in terms of logic theories.
@article{DBLP:journals/corr/abs-2309-00422, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/journals/corr/abs-2309-00422.bib}, doi = {10.48550/ARXIV.2309.00422}, eprint = {2309.00422}, eprinttype = {arXiv}, journal = {CoRR}, line = {2}, month = dec, timestamp = {Mon, 11 Sep 2023 16:01:35 +0200}, title = {Declarative Reasoning on Explanations Using Constraint Logic Programming}, volume = {abs/2309.00422}, year = {2023} }
Modeling Events and Interactions through Temporal Processes – A Survey

Angelica Liguori, Luciano Caroprese, Marco Minici, Bruno Veloso, Francesco Spinnato, and 3 more authors

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

In real-world scenario, many phenomena produce a collection of events that occur in continuous time. Point Processes provide a natural mathematical framework for modeling these sequences of events. In this survey, we investigate probabilistic models for modeling event sequences through temporal processes. We revise the notion of event modeling and provide the mathematical foundations that characterize the literature on the topic. We define an ontology to categorize the existing approaches in terms of three families: simple, marked, and spatio-temporal point processes. For each family, we systematically review the existing approaches based based on deep learning. Finally, we analyze the scenarios where the proposed techniques can be used for addressing prediction and modeling aspects.
@misc{https://doi.org/10.48550/arxiv.2303.06067, author = {Liguori, Angelica and Caroprese, Luciano and Minici, Marco and Veloso, Bruno and Spinnato, Francesco and Nanni, Mirco and Manco, Giuseppe and Gama, Joao}, copyright = {arXiv.org perpetual, non-exclusive license}, doi = {10.48550/ARXIV.2303.06067}, keywords = {Machine Learning (cs.LG), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences}, line = {1}, month = dec, publisher = {arXiv}, title = {Modeling Events and Interactions through Temporal Processes -- A Survey}, year = {2023} }
Geolet: An Interpretable Model for Trajectory Classification

Cristiano Landi, Francesco Spinnato, Riccardo Guidotti, Anna Monreale, and Mirco Nanni

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{Landi_2023, author = {Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-031-30047-9_19}, isbn = {9783031300479}, issn = {1611-3349}, line = {1}, month = dec, pages = {236–248}, publisher = {Springer Nature Switzerland}, title = {Geolet: An Interpretable Model for Trajectory Classification}, year = {2023} }
Exposing Racial Dialect Bias in Abusive Language Detection: Can Explainability Play a Role?

Marta Marchiori Manerba, and Virginia Morini

Dec 2023

RESEARCH LINE
5

Abs Bib HTML

Biases can arise and be introduced during each phase of a supervised learning pipeline, eventually leading to harm. Within the task of automatic abusive language detection, this matter becomes particularly severe since unintended bias towards sensitive topics such as gender, sexual orientation, or ethnicity can harm underrepresented groups. The role of the datasets used to train these models is crucial to address these challenges. In this contribution, we investigate whether explainability methods can expose racial dialect bias attested within a popular dataset for abusive language detection. Through preliminary experiments, we found that pure explainability techniques cannot effectively uncover biases within the dataset under analysis: the rooted stereotypes are often more implicit and complex to retrieve.
@inbook{Manerba_2023, author = {Manerba, Marta Marchiori and Morini, Virginia}, booktitle = {Machine Learning and Principles and Practice of Knowledge Discovery in Databases}, doi = {10.1007/978-3-031-23618-1_32}, isbn = {9783031236181}, issn = {1865-0937}, line = {5}, month = dec, pages = {483–497}, publisher = {Springer Nature Switzerland}, title = {Exposing Racial Dialect Bias in Abusive Language Detection: Can Explainability Play a Role?}, year = {2023} }
Differentiable Causal Discovery with Smooth Acyclic Orientations

Riccardo Massidda, Francesco Landolfi, Martina Cinquini, and Davide Bacciu

In ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators , Dec 2023

RESEARCH LINE
2

Abs Bib HTML

Most differentiable causal discovery approaches constrain or regularize an optimization problem using a continuous relaxation of the acyclicity property. The cost of computing the relaxation is cubic on the number of nodes and thus affects the scalability of such techniques. In this work, we introduce COSMO, the first quadratic and constraint-free continuous optimization scheme. COSMO represents a directed acyclic graph as a priority vector on the nodes and an adjacency matrix. We prove that the priority vector represents a differentiable approximation of the acyclic orientation of the graph, and we demonstrate the existence of an upper bound on the orientation acyclicity. In addition to being asymptotically faster, our empirical analysis highlights how COSMO performs comparably to constrained methods for graph discovery.
@inproceedings{massidda2023differentiable, author = {Massidda, Riccardo and Landolfi, Francesco and Cinquini, Martina and Bacciu, Davide}, booktitle = {ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators}, line = {2}, month = dec, title = {Differentiable Causal Discovery with Smooth Acyclic Orientations}, year = {2023}, }
Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning

Carlo Metta, Andrea Beretta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and 2 more authors

International Journal of Data Science and Analytics, Jun 2023

RESEARCH LINE
1

3

4

Abs Bib HTML

A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{Metta_2023, author = {Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca}, doi = {10.1007/s41060-023-00401-z}, issn = {2364-4168}, journal = {International Journal of Data Science and Analytics}, line = {1,3,4}, month = jun, publisher = {Springer Science and Business Media LLC}, title = {Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning}, year = {2023} }
AUC-based Selective Classification

Andrea Pugnana, and Salvatore Ruggieri

In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , 25–27 apr 2023

RESEARCH LINE
1

2

Abs Bib HTML PDF

Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
@inproceedings{pmlr-v206-pugnana23a, author = {Pugnana, Andrea and Ruggieri, Salvatore}, booktitle = {Proceedings of The 26th International Conference on Artificial Intelligence and Statistics}, editor = {Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem}, line = {1,2}, month = {25--27 Apr}, pages = {2494--2514}, publisher = {PMLR}, series = {Proceedings of Machine Learning Research}, title = {AUC-based Selective Classification}, volume = {206}, year = {2023} }
Text to Time Series Representations: Towards Interpretable Predictive Models

Mattia Poggioli, Francesco Spinnato, and Riccardo Guidotti

Dec 2023

RESEARCH LINE
1

Abs Bib HTML

Time Series Analysis (TSA) and Natural Language Processing (NLP) are two domains of research that have seen a surge of interest in recent years. NLP focuses mainly on enabling computers to manipulate and generate human language, whereas TSA identifies patterns or components in time-dependent data. Given their different purposes, there has been limited exploration of combining them. In this study, we present an approach to convert text into time series to exploit TSA for exploring text properties and to make NLP approaches interpretable for humans. We formalize our Text to Time Series framework as a feature extraction and aggregation process, proposing a set of different conversion alternatives for each step. We experiment with our approach on several textual datasets, showing the conversion approach’s performance and applying it to the field of interpretable time series classification.
@inbook{Poggioli_2023, author = {Poggioli, Mattia and Spinnato, Francesco and Guidotti, Riccardo}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-031-45275-8_16}, isbn = {9783031452758}, issn = {1611-3349}, line = {1}, month = dec, pages = {230–245}, publisher = {Springer Nature Switzerland}, title = {Text to Time Series Representations: Towards Interpretable Predictive Models}, year = {2023} }
Topics in Selective Classification

Andrea Pugnana

Proceedings of the AAAI Conference on Artificial Intelligence, Sep 2023

RESEARCH LINE
1

2

Abs Bib HTML

In recent decades, advancements in information technology allowed Artificial Intelligence (AI) systems to predict future outcomes with unprecedented success. This brought the widespread deployment of these methods in many fields, intending to support decision-making. A pressing question is how to make AI systems robust to common challenges in real-life scenarios and trustworthy. In my work, I plan to explore ways to enhance the trustworthiness of AI through the selective classification framework. In this setting, the AI system can refrain from predicting whenever it is not confident enough, allowing it to trade off coverage, i.e. the percentage of instances that receive a prediction, for performance.
@article{Pugnana_2023, author = {Pugnana, Andrea}, doi = {10.1609/aaai.v37i13.26925}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1,2}, month = sep, number = {13}, pages = {16129-16130}, title = {Topics in Selective Classification}, volume = {37}, year = {2023} }
Understanding Any Time Series Classifier with a Subsequence-based Explainer

Francesco Spinnato, Riccardo Guidotti, Anna Monreale, Mirco Nanni, Dino Pedreschi, and 1 more author

ACM Transactions on Knowledge Discovery from Data, Nov 2023

RESEARCH LINE
1

Abs Bib HTML

The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.
@article{Spinnato_2023, author = {Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1145/3624480}, issn = {1556-472X}, journal = {ACM Transactions on Knowledge Discovery from Data}, line = {1}, month = nov, number = {2}, pages = {1–34}, publisher = {Association for Computing Machinery (ACM)}, title = {Understanding Any Time Series Classifier with a Subsequence-based Explainer}, volume = {18}, year = {2023} }
Reason to explain: Interactive contrastive explanations (REASONX)

Laura State, Salvatore Ruggieri, and Franco Turini

Dec 2023

RESEARCH LINE
2

3

Abs Bib HTML

Many high-performing machine learning models are not interpretable. As they are increasingly used in decision scenarios that can critically affect individuals, it is necessary to develop tools to better understand their outputs. Popular explanation methods include contrastive explanations. However, they suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of interactivity. While (dialogue-like) interactivity is important to better communicate an explanation, background knowledge has the potential to significantly improve their quality, e.g., by adapting the explanation to the needs of the end-user. To close this gap, we present REASONX, an explanation tool based on Constraint Logic Programming (CLP). REASONX provides interactive contrastive explanations that can be augmented by background knowledge, and allows to operate under a setting of under-specified information, leading to increased flexibility in the provided explanations. REASONX computes factual and contrastive decision rules, as well as closest contrastive examples. It provides explanations for decision trees, which can be the ML models under analysis, or global/local surrogate models of any ML model. While the core part of REASONX is built on CLP, we also provide a program layer that allows to compute the explanations via Python, making the tool accessible to a wider audience. We illustrate the capability of REASONX on a synthetic data set, and on a well-developed example in the credit domain. In both cases, we can show how REASONX can be flexibly used and tailored to the needs of the user.
@misc{state2023reason, archiveprefix = {arXiv}, author = {State, Laura and Ruggieri, Salvatore and Turini, Franco}, eprint = {2305.18143}, line = {2,3}, month = dec, primaryclass = {cs.AI}, title = {Reason to explain: Interactive contrastive explanations (REASONX)}, year = {2023}, }
Co-design of Human-centered, Explainable AI for Clinical Decision Support

Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, and 2 more authors

ACM Trans. Interact. Intell. Syst., Dec 2023

Abs Bib HTML

eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{10.1145/3587271, author = {Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore}, title = {Co-design of Human-centered, Explainable AI for Clinical Decision Support}, year = {2023}, issue_date = {December 2023}, publisher = {Association for Computing Machinery}, address = {New York, NY, USA}, volume = {13}, number = {4}, issn = {2160-6455}, doi = {10.1145/3587271}, journal = {ACM Trans. Interact. Intell. Syst.}, month = dec, articleno = {21}, numpages = {35}, keywords = {user study, human-computer interaction, clinical decision support systems, Explainable artificial intelligence} }

2022

Interpretable Latent Space to Enable Counterfactual Explanations

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{Bodria_2022_Interpretable, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-031-18840-4_37}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, month = dec, pages = {525–540}, publisher = {Springer Nature Switzerland}, title = {Interpretable Latent Space to Enable Counterfactual Explanations}, year = {2022} }
Transparent Latent Space Counterfactual Explanations for Tabular Data

Francesco Bodria, Riccardo Guidotti, Fosca Giannotti, and Dino Pedreschi

In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022

RESEARCH LINE
1

Abs Bib HTML

Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{Bodria_2022_Transparent, author = {Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)}, doi = {10.1109/dsaa54385.2022.10032407}, line = {1}, month = oct, publisher = {IEEE}, title = {Transparent Latent Space Counterfactual Explanations for Tabular Data}, year = {2022} }
Explaining Siamese Networks in Few-Shot Learning for Audio Data

Andrea Fedele, Riccardo Guidotti, and Dino Pedreschi

In Discovery Science - 25th International Conference, DS 2022, Montpellier, France, October 10-12, 2022, Proceedings , Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Machine learning models are not able to generalize correctly when queried on samples belonging to class distributions that were never seen during training. This is a critical issue, since real world applications might need to quickly adapt without the necessity of re-training. To overcome these limitations, few-shot learning frameworks have been proposed and their applicability has been studied widely for computer vision tasks. Siamese Networks learn pairs similarity in form of a metric that can be easily extended on new unseen classes. Unfortunately, the downside of such systems is the lack of explainability. We propose a method to explain the outcomes of Siamese Networks in the context of few-shot learning for audio data. This objective is pursued through a local perturbation-based approach that evaluates segments-weighted-average contributions to the final outcome considering the interplay between different areas of the audio spectrogram. Qualitative and quantitative results demonstrate that our method is able to show common intra-class characteristics and erroneous reliance on silent sections.
@inproceedings{DBLP:conf/dis/FedeleGP22, author = {Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/conf/dis/FedeleGP22.bib}, booktitle = {Discovery Science - 25th International Conference, {DS} 2022, Montpellier, France, October 10-12, 2022, Proceedings}, doi = {10.1007/978-3-031-18840-4\_36}, editor = {Poncelet, Pascal and Ienco, Dino}, line = {1}, month = dec, pages = {509--524}, publisher = {Springer}, series = {Lecture Notes in Computer Science}, timestamp = {Sat, 30 Sep 2023 09:38:59 +0200}, title = {Explaining Siamese Networks in Few-Shot Learning for Audio Data}, volume = {13601}, year = {2022} }
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions

Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti

IEEE Access, Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Time series classification (TSC) is a challenging task in machine learning and data mining. The increasing complexity of models used for TSC has led to a growing demand for explainable artificial intelligence (XAI) methods. This paper provides a comprehensive review of the state-of-the-art in XAI for TSC. We present a taxonomy of XAI methods for TSC and discuss their strengths and weaknesses. We also identify research directions for future work in this area.
@article{DBLP:journals/access/TheisslerSSG22, author = {Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo}, bibsource = {dblp computer science bibliography, https://dblp.org}, biburl = {https://dblp.org/rec/journals/access/TheisslerSSG22.bib}, doi = {10.1109/ACCESS.2022.3207765}, journal = {{IEEE} Access}, line = {1}, month = dec, pages = {100700--100724}, timestamp = {Tue, 18 Oct 2022 22:17:12 +0200}, title = {Explainable {AI} for Time Series Classification: {A} Review, Taxonomy and Research Directions}, volume = {10}, year = {2022} }
Stable and actionable explanations of black-box models through factual and counterfactual rules

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Francesca Naretto, Franco Turini, and 2 more authors

Data Mining and Knowledge Discovery, Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{Guidotti2023Stable, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, journal = {Data Mining and Knowledge Discovery}, line = {1}, month = dec, title = {Stable and actionable explanations of black-box models through factual and counterfactual rules}, year = {2022}, }
Counterfactual explanations and how to find them: literature review and benchmarking

Riccardo Guidotti

Data Mining and Knowledge Discovery, Apr 2022

RESEARCH LINE
1

Abs Bib HTML

Interpretable machine learning aims at unveiling the reasons behind predictions returned by uninterpretable classifiers. One of the most valuable types of explanation consists of counterfactuals. A counterfactual explanation reveals what should have been different in an instance to observe a diverse outcome. For instance, a bank customer asks for a loan that is rejected. The counterfactual explanation consists of what should have been different for the customer in order to have the loan accepted. Recently, there has been an explosion of proposals for counterfactual explainers. The aim of this work is to survey the most recent explainers returning counterfactual explanations. We categorize explainers based on the approach adopted to return the counterfactuals, and we label them according to characteristics of the method and properties of the counterfactuals returned. In addition, we visually compare the explanations, and we report quantitative benchmarking assessing minimality, actionability, stability, diversity, discriminative power, and running time. The results make evident that the current state of the art does not provide a counterfactual explainer able to guarantee all these properties simultaneously.
@article{Guidotti_2022, author = {Guidotti, Riccardo}, doi = {10.1007/s10618-022-00831-6}, issn = {1573-756X}, journal = {Data Mining and Knowledge Discovery}, line = {1}, month = apr, publisher = {Springer Science and Business Media LLC}, title = {Counterfactual explanations and how to find them: literature review and benchmarking}, year = {2022} }
Investigating Debiasing Effects on Classification and Explainability

Marta Marchiori Manerba, and Riccardo Guidotti

In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2022

RESEARCH LINE
1

Abs Bib HTML

Biases can arise and be introduced during each phase of a supervised learning pipeline, eventually leading to harm. Within the task of automatic abusive language detection, this matter becomes particularly severe since unintended bias towards sensitive topics such as gender, sexual orientation, or ethnicity can harm underrepresented groups. The role of the datasets used to train these models is crucial to address these challenges. In this contribution, we investigate whether explainability methods can expose racial dialect bias attested within a popular dataset for abusive language detection. Through preliminary experiments, we found that pure explainability techniques cannot effectively uncover biases within the dataset under analysis: the rooted stereotypes are often more implicit and complex to retrieve.
@inproceedings{Marchiori_Manerba_2022, author = {Marchiori Manerba, Marta and Guidotti, Riccardo}, booktitle = {Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’22}, doi = {10.1145/3514094.3534170}, line = {1}, month = jul, publisher = {ACM}, series = {AIES ’22}, title = {Investigating Debiasing Effects on Classification and Explainability}, year = {2022} }
Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

Sep 2022

RESEARCH LINE
1

3

Abs Bib HTML

Explainable AI consists in developing models allowing interaction between decision systems and humans by making the decisions understandable. We propose a case study for skin lesion diagnosis showing how it is possible to provide explanations of the decisions of deep neural network trained to label skin lesions.
@inbook{Metta_2022, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {Frontiers in Artificial Intelligence and Applications}, doi = {10.3233/faia220209}, issn = {1879-8314}, line = {1,3}, month = sep, publisher = {IOS Press}, title = {Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers}, year = {2022} }
Evaluating the Privacy Exposure of Interpretable Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

In 2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2022

RESEARCH LINE
5

Abs Bib HTML

In recent years we are witnessing the diffusion of AI systems based on powerful Machine Learning models which find application in many critical contexts such as medicine, financial market and credit scoring. In such a context it is particularly important to design Trustworthy AI systems while guaranteeing transparency, with respect to their decision reasoning and privacy protection. Although many works in the literature addressed the lack of transparency and the risk of privacy exposure of Machine Learning models, the privacy risks of explainers have not been appropriately studied. This paper presents a methodology for evaluating the privacy exposure raised by interpretable global explainers able to imitate the original black-box classifier. Our methodology exploits the well-known Membership Inference Attack. The experimental results highlight that global explainers based on interpretable trees lead to an increase in privacy exposure.
@inproceedings{Naretto_2022, author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca}, booktitle = {2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi56440.2022.00012}, line = {5}, month = dec, publisher = {IEEE}, title = {Evaluating the Privacy Exposure of Interpretable Global Explainers}, year = {2022} }

Privacy Risk of Global Explainers

Francesca Naretto, Anna Monreale, and Fosca Giannotti

Sep 2022

RESEARCH LINE

Abs Bib HTML

@inbook{Naretto_2022_Privacy,
  author = {Naretto, Francesca and Monreale, Anna and Giannotti, Fosca},
  booktitle = {Frontiers in Artificial Intelligence and Applications},
  doi = {10.3233/faia220206},
  issn = {1879-8314},
  line = {5},
  month = sep,
  publisher = {IOS Press},
  title = {Privacy Risk of Global Explainers},
  year = {2022}
}

Methods and tools for causal discovery and causal inference

Ana Rita Nogueira, Andrea Pugnana, Salvatore Ruggieri, Dino Pedreschi, and João Gama

WIREs Data Mining and Knowledge Discovery, Jan 2022

RESEARCH LINE
1

Abs Bib HTML

Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{Nogueira_2022, author = {Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João}, doi = {10.1002/widm.1449}, issn = {1942-4795}, journal = {WIREs Data Mining and Knowledge Discovery}, line = {1}, month = jan, number = {2}, publisher = {Wiley}, title = {Methods and tools for causal discovery and causal inference}, volume = {12}, year = {2022} }
Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems

Cecilia Panigutti, Andrea Beretta, Fosca Giannotti, and Dino Pedreschi

In CHI Conference on Human Factors in Computing Systems , Apr 2022

RESEARCH LINE
1

3

4

Abs Bib HTML

The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{Panigutti_2022, author = {Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino}, booktitle = {CHI Conference on Human Factors in Computing Systems}, collection = {CHI ’22}, doi = {10.1145/3491102.3502104}, line = {1,3,4}, month = apr, publisher = {ACM}, series = {CHI ’22}, title = {Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems}, year = {2022} }
Explaining Crash Predictions on Multivariate Time Series Data

Francesco Spinnato, Riccardo Guidotti, Mirco Nanni, Daniele Maccagnola, Giulia Paciello, and 1 more author

Dec 2022

RESEARCH LINE
1

Abs Bib HTML

In Assicurazioni Generali, an automatic decision-making model is used to check real-time multivariate time series and alert if a car crash happened. In such a way, a Generali operator can call the customer to provide first assistance. The high sensitivity of the model used, combined with the fact that the model is not interpretable, might cause the operator to call customers even though a car crash did not happen but only due to a harsh deviation or the fact that the road is bumpy. Our goal is to tackle the problem of interpretability for car crash prediction and propose an eXplainable Artificial Intelligence (XAI) workflow that allows gaining insights regarding the logic behind the deep learning predictive model adopted by Generali. We reach our goal by building an interpretable alternative to the current obscure model that also reduces the training data usage and the prediction time.
@inbook{Spinnato_2022, author = {Spinnato, Francesco and Guidotti, Riccardo and Nanni, Mirco and Maccagnola, Daniele and Paciello, Giulia and Farina, Antonio Bencini}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-031-18840-4_39}, isbn = {9783031188404}, issn = {1611-3349}, line = {1}, month = dec, pages = {556–566}, publisher = {Springer Nature Switzerland}, title = {Explaining Crash Predictions on Multivariate Time Series Data}, year = {2022} }
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions

Andreas Theissler, Francesco Spinnato, Udo Schlegel, and Riccardo Guidotti

IEEE Access, Dec 2022

RESEARCH LINE
1

Abs Bib HTML

Time series data is increasingly used in a wide range of fields, and it is often relied on in crucial applications and high-stakes decision-making. For instance, sensors generate time series data to recognize different types of anomalies through automatic decision-making systems. Typically, these systems are realized with machine learning models that achieve top-tier performance on time series classification tasks. Unfortunately, the logic behind their prediction is opaque and hard to understand from a human standpoint. Recently, we observed a consistent increase in the development of explanation methods for time series classification justifying the need to structure and review the field. In this work, we (a) present the first extensive literature review on Explainable AI (XAI) for time series classification, (b) categorize the research field through a taxonomy subdividing the methods into time points-based, subsequences-based and instance-based, and (c) identify open research directions regarding the type of explanations and the evaluation of explanations and interpretability.
@article{Theissler_2022, author = {Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo}, doi = {10.1109/access.2022.3207765}, issn = {2169-3536}, journal = {IEEE Access}, line = {1}, month = dec, pages = {100700–100724}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions}, volume = {10}, year = {2022} }
Understanding peace through the world news

Vasiliki Voukelatou, Ioanna Miliou, Fosca Giannotti, and Luca Pappalardo

EPJ Data Science, Jan 2022

RESEARCH LINE
1

Abs Bib HTML

Peace is a principal dimension of well-being and is the way out of inequity and violence. Thus, its measurement has drawn the attention of researchers, policymakers, and peacekeepers. During the last years, novel digital data streams have drastically changed the research in this field. The current study exploits information extracted from a new digital database called Global Data on Events, Location, and Tone (GDELT) to capture peace through the Global Peace Index (GPI). Applying predictive machine learning models, we demonstrate that news media attention from GDELT can be used as a proxy for measuring GPI at a monthly level. Additionally, we use explainable AI techniques to obtain the most important variables that drive the predictions. This analysis highlights each country’s profile and provides explanations for the predictions, and particularly for the errors and the events that drive these errors. We believe that digital data exploited by researchers, policymakers, and peacekeepers, with data science tools as powerful as machine learning, could contribute to maximizing the societal benefits and minimizing the risks to peace.
@article{Voukelatou_2022, author = {Voukelatou, Vasiliki and Miliou, Ioanna and Giannotti, Fosca and Pappalardo, Luca}, doi = {10.1140/epjds/s13688-022-00315-z}, issn = {2193-1127}, journal = {EPJ Data Science}, line = {1}, month = jan, number = {4}, publisher = {Springer Science and Business Media LLC}, title = {Understanding peace through the world news}, volume = {11}, year = {2022} }
Explaining Black Box with Visual Exploration of Latent Space

Francesco Bodria, Salvatore Rinzivillo, Daniele Fadda, Riccardo Guidotti, Fosca Giannotti, and 1 more author

In EuroVis 2022 - Short Papers , Jan 2022

Abs Bib HTML

Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@inproceedings{10.2312:evs.20221098, booktitle = {EuroVis 2022 - Short Papers}, editor = {Agus, Marco and Aigner, Wolfgang and Hoellt, Thomas}, title = {{Explaining Black Box with Visual Exploration of Latent Space}}, author = {Bodria, Francesco and Rinzivillo, Salvatore and Fadda, Daniele and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino}, year = {2022}, publisher = {The Eurographics Association}, isbn = {978-3-03868-184-7}, doi = {10.2312/evs.20221098}, }

2021

Explainable AI Within the Digital Transformation and Cyber Physical Systems: XAI Methods and Applications

Dec 2021

RESEARCH LINE
1

5

Abs Bib HTML

This book presents Explainable Artificial Intelligence (XAI), which aims at producing explainable models that enable human users to understand and appropriately trust the obtained results. The authors discuss the challenges involved in making machine learning-based AI explainable. Firstly, that the explanations must be adapted to different stakeholders (end-users, policy makers, industries, utilities etc.) with different levels of technical knowledge (managers, engineers, technicians, etc.) in different application domains. Secondly, that it is important to develop an evaluation framework and standards in order to measure the effectiveness of the provided explanations at the human and the technical levels. This book gathers research contributions aiming at the development and/or the use of XAI techniques in order to address the aforementioned challenges in different applications such as healthcare, finance, cybersecurity, and document summarization. It allows highlighting the benefits and requirements of using explainable models in different application domains in order to provide guidance to readers to select the most adapted models to their specified problem and conditions. Includes recent developments of the use of Explainable Artificial Intelligence (XAI) in order to address the challenges of digital transition and cyber-physical systems; Provides a textual scientific description of the use of XAI in order to address the challenges of digital transition and cyber-physical systems; Presents examples and case studies in order to increase transparency and understanding of the methodological concepts.
@book{2021, doi = {10.1007/978-3-030-76409-8}, isbn = {9783030764098}, line = {1,5}, month = dec, publisher = {Springer International Publishing}, title = {Explainable AI Within the Digital Transformation and Cyber Physical Systems: XAI Methods and Applications}, year = {2021} }
Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche

Cecilia Panigutti, and Emanuele Bosi

2021, Dec 2021

RESEARCH LINE
4

Abs Bib HTML

Major technological advances over the past two decades have made it possible to obtain, store and analyze a massive amount of data (also called Big Data) concerning every aspect of reality. The progress we are witnessing in this area transcends purely technical-scientific aspects, going on to pervade the sphere of daily life. Facial recognition in photos on social media (1), the personalization of content and shopping suggestions (2-3), and the management of interactions with voice assistants such as Siri and Alexa (4-5) are some examples of applications of artificial intelligence (AI) methods to analyze the huge volume of available data in real time. In the field of biomedical sciences we are witnessing a similarly rapid increase in the ability to produce and analyze data at multiple levels, with major implications for the diagnosis and treatment of many diseases. Although terms such as precision medicine and precision therapy are increasingly popular, the underlying AI concepts are often ignored by experts in the medical area. This article will discuss the basics of AI, advantages and limitations of the state of the art in medicine (and particularly in diabetology), and future prospects for routine applications.
@article{2021diabetologico, doi = {10.30682/ildia2101f}, issn = {1720-8335}, journal = {2021}, line = {4}, month = dec, number = {1,4}, publisher = {Bononia University Press}, title = {Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche}, volume = {33}, year = {2021}, author = {Panigutti, Cecilia and Bosi, Emanuele} }
Deep Learning in Biology and Medicine

Davide Bacciu, Paulo J G Lisboa, and Alfredo Vellido

Jun 2021

RESEARCH LINE
4

Abs Bib HTML

Biology, medicine and biochemistry have become data-centric fields for which Deep Learning methods are delivering groundbreaking results. Addressing high impact challenges, Deep Learning in Biology and Medicine provides an accessible and organic collection of Deep Learning essays on bioinformatics and medicine. It caters for a wide readership, ranging from machine learning practitioners and data scientists seeking methodological knowledge to address biomedical applications, to life science specialists in search of a gentle reference for advanced data analytics. With contributions from internationally renowned experts, the book covers foundational methodologies in a wide spectrum of life sciences applications, including electronic health record processing, diagnostic imaging, text processing, as well as omics-data processing. This survey of consolidated problems is complemented by a selection of advanced applications, including cheminformatics and biomedical interaction network analysis. A modern and mindful approach to the use of data-driven methodologies in the life sciences also requires careful consideration of the associated societal, ethical, legal and transparency challenges, which are covered in the concluding chapters of this book.
@book{Bacciu_2021, author = {Bacciu, Davide and Lisboa, Paulo J G and Vellido, Alfredo}, doi = {10.1142/q0322}, isbn = {9781800610941}, line = {4}, month = jun, publisher = {WORLD SCIENTIFIC (EUROPE)}, title = {Deep Learning in Biology and Medicine}, year = {2021} }
Deriving a Single Interpretable Model by Merging Tree-Based Classifiers

Valerio Bonsignori, Riccardo Guidotti, and Anna Monreale

Dec 2021

RESEARCH LINE
1

Abs Bib HTML

Decision tree classifiers have been proved to be among the most interpretable models due to their intuitive structure that illustrates decision processes in form of logical rules. Unfortunately, more complex tree-based classifiers such as oblique trees and random forests overcome the accuracy of decision trees at the cost of becoming non interpretable. In this paper, we propose a method that takes as input any tree-based classifier and returns a single decision tree able to approximate its behavior. Our proposal merges tree-based classifiers by an intensional and extensional approach and applies a post-hoc explanation strategy. Our experiments shows that the retrieved single decision tree is at least as accurate as the original tree-based model, faithful, and more interpretable.
@inbook{Bonsignori_2021, author = {Bonsignori, Valerio and Guidotti, Riccardo and Monreale, Anna}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-88942-5_27}, isbn = {9783030889425}, issn = {1611-3349}, line = {1}, month = dec, pages = {347–357}, publisher = {Springer International Publishing}, title = {Deriving a Single Interpretable Model by Merging Tree-Based Classifiers}, year = {2021} }
Trustworthy AI

Raja Chatila, Virginia Dignum, Michael Fisher, Fosca Giannotti, Katharina Morik, and 2 more authors

Dec 2021

RESEARCH LINE
5

Abs Bib HTML

Modern AI systems have become of widespread use in almost all sectors with a strong impact on our society. However, the very methods on which they rely, based on Machine Learning techniques for processing data to predict outcomes and to make decisions, are opaque, prone to bias and may produce wrong answers. Objective functions optimized in learning systems are not guaranteed to align with the values that motivated their definition. Properties such as transparency, verifiability, explainability, security, technical robustness and safety, are key to build operational governance frameworks, so that to make AI systems justifiably trustworthy and to align their development and use with human rights and values.
@inbook{Chatila_2021, author = {Chatila, Raja and Dignum, Virginia and Fisher, Michael and Giannotti, Fosca and Morik, Katharina and Russell, Stuart and Yeung, Karen}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-69128-8_2}, isbn = {9783030691288}, issn = {1611-3349}, line = {5}, month = dec, pages = {13–39}, publisher = {Springer International Publishing}, title = {Trustworthy AI}, year = {2021} }
Matrix Profile-Based Interpretable Time Series Classifier

Riccardo Guidotti, and Matteo D’Onofrio

Frontiers in Artificial Intelligence, Oct 2021

RESEARCH LINE
1

Abs Bib HTML

Time series classification (TSC) is a pervasive and transversal problem in various fields ranging from disease diagnosis to anomaly detection in finance. Unfortunately, the most effective models used by Artificial Intelligence (AI) systems for TSC are not interpretable and hide the logic of the decision process, making them unusable in sensitive domains. Recent research is focusing on explanation methods to pair with the obscure classifier to recover this weakness. However, a TSC approach that is transparent by design and is simultaneously efficient and effective is even more preferable. To this aim, we propose an interpretable TSC method based on the patterns, which is possible to extract from the Matrix Profile (MP) of the time series in the training set. A smart design of the classification procedure allows obtaining an efficient and effective transparent classifier modeled as a decision tree that expresses the reasons for the classification as the presence of discriminative subsequences. Quantitative and qualitative experimentation shows that the proposed method overcomes the state-of-the-art interpretable approaches.
@article{Guidotti_2021, author = {Guidotti, Riccardo and D’Onofrio, Matteo}, doi = {10.3389/frai.2021.699448}, issn = {2624-8212}, journal = {Frontiers in Artificial Intelligence}, line = {1}, month = oct, publisher = {Frontiers Media SA}, title = {Matrix Profile-Based Interpretable Time Series Classifier}, volume = {4}, year = {2021} }
Designing Shapelets for Interpretable Data-Agnostic Classification

Riccardo Guidotti, and Anna Monreale

In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2021

RESEARCH LINE
1

Abs Bib HTML

Time series shapelets are discriminatory subsequences which are representative of a class, and their similarity to a time series can be used for successfully tackling the time series classification problem. The literature shows that Artificial Intelligence (AI) systems adopting classification models based on time series shapelets can be interpretable, more accurate, and significantly fast. Thus, in order to design a data-agnostic and interpretable classification approach, in this paper we first extend the notion of shapelets to different types of data, i.e., images, tabular and textual data. Then, based on this extended notion of shapelets we propose an interpretable data-agnostic classification method. Since the shapelets discovery can be time consuming, especially for data types more complex than time series, we exploit a notion of prototypes for finding candidate shapelets, and reducing both the time required to find a solution and the variance of shapelets. A wide experimentation on datasets of different types shows that the data-agnostic prototype-based shapelets returned by the proposed method empower an interpretable classification which is also fast, accurate, and stable. In addition, we show and we prove that shapelets can be at the basis of explainable AI methods.
@inproceedings{Guidotti_2021_Designing, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society}, collection = {AIES ’21}, doi = {10.1145/3461702.3462553}, line = {1}, month = jul, publisher = {ACM}, series = {AIES ’21}, title = {Designing Shapelets for Interpretable Data-Agnostic Classification}, year = {2021} }
Evaluating local explanation methods on ground truth

Riccardo Guidotti

Artificial Intelligence, Feb 2021

RESEARCH LINE
1

Abs Bib HTML

Evaluating local explanation methods is a difficult task due to the lack of a shared and universally accepted definition of explanation. In the literature, one of the most common ways to assess the performance of an explanation method is to measure the fidelity of the explanation with respect to the classification of a black box model adopted by an Artificial Intelligent system for making a decision. However, this kind of evaluation only measures the degree of adherence of the local explainer in reproducing the behavior of the black box classifier with respect to the final decision. Therefore, the explanation provided by the local explainer could be different in the content even though it leads to the same decision of the AI system. In this paper, we propose an approach that allows to measure to which extent the explanations returned by local explanation methods are correct with respect to a synthetic ground truth explanation. Indeed, the proposed methodology enables the generation of synthetic transparent classifiers for which the reason for the decision taken, i.e., a synthetic ground truth explanation, is available by design. Experimental results show how the proposed approach allows to easily evaluate local explanations on the ground truth and to characterize the quality of local explanation methods.
@article{Guidotti_2021_Evaluating, author = {Guidotti, Riccardo}, doi = {10.1016/j.artint.2020.103428}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1}, month = feb, pages = {103428}, publisher = {Elsevier BV}, title = {Evaluating local explanation methods on ground truth}, volume = {291}, year = {2021} }
FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems

Marta Marchiori Manerba, and Riccardo Guidotti

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
5

Abs Bib HTML

At every stage of a supervised learning process, harmful biases can arise and be inadvertently introduced, ultimately leading to marginalization, discrimination, and abuse towards minorities. This phenomenon becomes particularly impactful in the sensitive real-world context of abusive language detection systems, where non-discrimination is difficult to assess. In addition, given the opaqueness of their internal behavior, the dynamics leading a model to a certain decision are often not clear nor accountable, and significant problems of trust could emerge. A robust value-oriented evaluation of models’ fairness is therefore necessary. In this paper, we present FairShades, a model-agnostic approach for auditing the outcomes of abusive language detection systems. Combining explainability and fairness evaluation, FairShades can identify unintended biases and sensitive categories towards which models are most discriminative. This objective is pursued through the auditing of meaningful counterfactuals generated within CheckList framework. We conduct several experiments on BERT-based models to demonstrate our proposal’s novelty and effectiveness for unmasking biases.
@inproceedings{Manerba_2021, author = {Manerba, Marta Marchiori and Guidotti, Riccardo}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00014}, line = {5}, month = dec, publisher = {IEEE}, title = {FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems}, year = {2021} }
Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling

Carlo Metta, Riccardo Guidotti, Yuan Yin, Patrick Gallinari, and Salvatore Rinzivillo

In 2021 IEEE Symposium on Computers and Communications (ISCC) , Sep 2021

RESEARCH LINE
1

Abs Bib HTML

Explainable AI consists in developing mechanisms allowing for an interaction between decision systems and humans by making the decisions of the formers understandable. This is particularly important in sensitive contexts like in the medical domain. We propose a use case study, for skin lesion diagnosis, illustrating how it is possible to provide the practitioner with explanations on the decisions of a state of the art deep neural network classifier trained to characterize skin lesions from examples. Our framework consists of a trained classifier onto which an explanation module operates. The latter is able to offer the practitioner exemplars and counterexemplars for the classification diagnosis thus allowing the physician to interact with the automatic diagnosis system. The exemplars are generated via an adversarial autoencoder. We illustrate the behavior of the system on representative examples.
@inproceedings{Metta_2021, author = {Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore}, booktitle = {2021 IEEE Symposium on Computers and Communications (ISCC)}, doi = {10.1109/iscc53001.2021.9631485}, line = {1}, month = sep, publisher = {IEEE}, title = {Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling}, year = {2021} }
FairLens: Auditing black-box clinical decision support systems

Cecilia Panigutti, Alan Perotti, André Panisson, Paolo Bajardi, and Dino Pedreschi

Information Processing & Management, Sep 2021

RESEARCH LINE
1

4

Abs Bib HTML

Highlights: We present a pipeline to detect and explain potential fairness issues in Clinical DSS. We study and compare different multi-label classification disparity measures. We explore ICD9 bias in MIMIC-IV, an openly available ICU benchmark dataset
@article{Panigutti_2021, author = {Panigutti, Cecilia and Perotti, Alan and Panisson, André and Bajardi, Paolo and Pedreschi, Dino}, doi = {10.1016/j.ipm.2021.102657}, issn = {0306-4573}, journal = {Information Processing & Management}, line = {1,4}, month = sep, number = {5}, pages = {102657}, publisher = {Elsevier BV}, title = {FairLens: Auditing black-box clinical decision support systems}, volume = {58}, year = {2021} }
Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals

Michele Resta, Anna Monreale, and Davide Bacciu

Entropy, Aug 2021

RESEARCH LINE
1

Abs Bib HTML

The biomedical field is characterized by an ever-increasing production of sequential data, which often come in the form of biosignals capturing the time-evolution of physiological processes, such as blood pressure and brain activity. This has motivated a large body of research dealing with the development of machine learning techniques for the predictive analysis of such biosignals. Unfortunately, in high-stakes decision making, such as clinical diagnosis, the opacity of machine learning models becomes a crucial aspect to be addressed in order to increase the trust and adoption of AI technology. In this paper, we propose a model agnostic explanation method, based on occlusion, that enables the learning of the input’s influence on the model predictions. We specifically target problems involving the predictive analysis of time-series data and the models that are typically used to deal with data of such nature, i.e., recurrent neural networks. Our approach is able to provide two different kinds of explanations: one suitable for technical experts, who need to verify the quality and correctness of machine learning models, and one suited to physicians, who need to understand the rationale underlying the prediction to make aware decisions. A wide experimentation on different physiological data demonstrates the effectiveness of our approach both in classification and regression tasks.
@article{Resta_2021, author = {Resta, Michele and Monreale, Anna and Bacciu, Davide}, doi = {10.3390/e23081064}, issn = {1099-4300}, journal = {Entropy}, line = {1}, month = aug, number = {8}, pages = {1064}, publisher = {MDPI AG}, title = {Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals}, volume = {23}, year = {2021} }
TRIPLEx: Triple Extraction for Explanation

Mattia Setzu, Anna Monreale, and Pasquale Minervini

In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021

RESEARCH LINE
1

2

Abs Bib HTML

Transformer-based models are used to solve a va-riety of Natural Language Processing tasks. Still, these models are opaque and poorly understandable for their users. Current approaches to explainability focus on token importance, in which the explanation consists of a set of tokens relevant to the prediction, and natural language explanations, in which the explanation is a generated piece of text. The latter are usually learned by design with models traind end-to-end to provide a prediction and an explanation, or rely on powerful external text generators to do the heavy lifting for them. In this paper we present TRIPLEX, an explainability algorithm for Transformer-based models fine-tuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks. TRIPLEX explains Transformers-based models by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation.
@inproceedings{Setzu_2021, author = {Setzu, Mattia and Monreale, Anna and Minervini, Pasquale}, booktitle = {2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi52975.2021.00015}, line = {1,2}, month = dec, publisher = {IEEE}, title = {TRIPLEx: Triple Extraction for Explanation}, year = {2021} }
GLocalX - From Local to Global Explanations of Black Box AI Models

Mattia Setzu, Riccardo Guidotti, Anna Monreale, Franco Turini, Dino Pedreschi, and 1 more author

Artificial Intelligence, May 2021

RESEARCH LINE
1

2

Abs Bib HTML

Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{Setzu_2021_GLocalX, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca}, doi = {10.1016/j.artint.2021.103457}, issn = {0004-3702}, journal = {Artificial Intelligence}, line = {1,2}, month = may, pages = {103457}, publisher = {Elsevier BV}, title = {GLocalX - From Local to Global Explanations of Black Box AI Models}, volume = {294}, year = {2021} }

2020

Explaining Any Time Series Classifier

Riccardo Guidotti, Anna Monreale, Francesco Spinnato, Dino Pedreschi, and Fosca Giannotti

In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020

RESEARCH LINE
1

Abs Bib HTML

We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{Guidotti_2020, author = {Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca}, booktitle = {2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)}, doi = {10.1109/cogmi50398.2020.00029}, line = {1}, month = oct, publisher = {IEEE}, title = {Explaining Any Time Series Classifier}, year = {2020} }
Black Box Explanation by Learning Image Exemplars in the Latent Feature Space

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by “morphing” into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
@inbook{Guidotti_2020_Black_Box, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-46150-8_12}, isbn = {9783030461508}, issn = {1611-3349}, line = {1}, month = dec, pages = {189–205}, publisher = {Springer International Publishing}, title = {Black Box Explanation by Learning Image Exemplars in the Latent Feature Space}, year = {2020} }
Data-Agnostic Local Neighborhood Generation

Riccardo Guidotti, and Anna Monreale

In 2020 IEEE International Conference on Data Mining (ICDM) . More Information can be found here , Nov 2020

RESEARCH LINE
1

5

Abs Bib HTML

Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, machine learning explanation, etc. In such contexts, it is important to generate data samples located within “local” areas surrounding specific instances. Local synthetic data can help the learning phase of predictive models, and it is fundamental for methods explaining the local behavior of obscure classifiers. The contribution of this paper is twofold. First, we introduce a method based on generative operators allowing the synthetic neighborhood generation by applying specific perturbations on a given input instance. The key factor consists in performing a data transformation that makes applicable to any type of data, i.e., data-agnostic. Second, we design a framework for evaluating the goodness of local synthetic neighborhoods exploiting both supervised and unsupervised methodologies. A deep experimentation shows the effectiveness of the proposed method.
@inproceedings{Guidotti_2020_Data-Agnostic, author = {Guidotti, Riccardo and Monreale, Anna}, booktitle = {2020 IEEE International Conference on Data Mining (ICDM)}, doi = {10.1109/icdm50108.2020.00122}, line = {1,5}, month = nov, publisher = {IEEE}, title = {Data-Agnostic Local Neighborhood Generation}, year = {2020} }
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations

Riccardo Guidotti, Anna Monreale, Stan Matwin, and Dino Pedreschi

Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2020

RESEARCH LINE
1

Abs Bib HTML

We present an approach to explain the decisions of black box image classifiers through synthetic exemplar and counter-exemplar learnt in the latent feature space. Our explanation method exploits the latent representations learned through an adversarial autoencoder for generating a synthetic neighborhood of the image for which an explanation is required. A decision tree is trained on a set of images represented in the latent space, and its decision rules are used to generate exemplar images showing how the original image can be modified to stay within its class. Counterfactual rules are used to generate counter-exemplars showing how the original image can “morph” into another class. The explanation also comprehends a saliency map highlighting the areas that contribute to its classification, and areas that push it into another class. A wide and deep experimental evaluation proves that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability, besides providing the most useful and interpretable explanations.
@article{Guidotti_2020_Image_Classifiers, author = {Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino}, doi = {10.1609/aaai.v34i09.7116}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = apr, number = {09}, pages = {13665–13668}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations}, volume = {34}, year = {2020} }
Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars

Orestis Lampridis, Riccardo Guidotti, and Salvatore Ruggieri

Dec 2020

RESEARCH LINE
1

Abs Bib HTML

We present xspells, a model-agnostic local approach for explaining the decisions of a black box model for sentiment classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. We report experiments on two datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, and usefulness, and that is comparable to it in terms of stability.
@inbook{Lampridis_2020, author = {Lampridis, Orestis and Guidotti, Riccardo and Ruggieri, Salvatore}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-61527-7_24}, isbn = {9783030615277}, issn = {1611-3349}, line = {1}, month = dec, pages = {357–373}, publisher = {Springer International Publishing}, title = {Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars}, year = {2020} }
Predicting and Explaining Privacy Risk Exposure in Mobility Data

Francesca Naretto, Roberto Pellungrini, Anna Monreale, Franco Maria Nardini, and Mirco Musolesi

Dec 2020

RESEARCH LINE
5

Abs Bib HTML

Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.
@inbook{Naretto_2020_Predicting, author = {Naretto, Francesca and Pellungrini, Roberto and Monreale, Anna and Nardini, Franco Maria and Musolesi, Mirco}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-61527-7_27}, isbn = {9783030615277}, issn = {1611-3349}, line = {5}, month = dec, pages = {403–418}, publisher = {Springer International Publishing}, title = {Predicting and Explaining Privacy Risk Exposure in Mobility Data}, year = {2020} }
Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks

Francesca Naretto, Roberto Pellungrini, Franco Maria Nardini, and Fosca Giannotti

Dec 2020

RESEARCH LINE
5

Abs Bib HTML

The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.
@inbook{Naretto_2020_Prediction, author = {Naretto, Francesca and Pellungrini, Roberto and Nardini, Franco Maria and Giannotti, Fosca}, booktitle = {Communications in Computer and Information Science}, doi = {10.1007/978-3-030-65965-3_34}, isbn = {9783030659653}, issn = {1865-0937}, line = {5}, month = dec, pages = {501–516}, publisher = {Springer International Publishing}, title = {Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks}, year = {2020} }
Doctor XAI: an ontology-based approach to black-box sequential data classification explanations

Cecilia Panigutti, Alan Perotti, and Dino Pedreschi

In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , Jan 2020

RESEARCH LINE
1

4

Abs Bib HTML

Several recent advancements in Machine Learning involve blackbox models: algorithms that do not provide human-understandable explanations in support of their decisions. This limitation hampers the fairness, accountability and transparency of these models; the field of eXplainable Artificial Intelligence (XAI) tries to solve this problem providing human-understandable explanations for black-box models. However, healthcare datasets (and the related learning tasks) often present peculiar features, such as sequential data, multi-label predictions, and links to structured background knowledge. In this paper, we introduce Doctor XAI, a model-agnostic explainability technique able to deal with multi-labeled, sequential, ontology-linked data. We focus on explaining Doctor AI, a multilabel classifier which takes as input the clinical history of a patient in order to predict the next visit. Furthermore, we show how exploiting the temporal dimension in the data and the domain knowledge encoded in the medical ontology improves the quality of the mined explanations.
@inproceedings{Panigutti_2020, author = {Panigutti, Cecilia and Perotti, Alan and Pedreschi, Dino}, booktitle = {Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency}, collection = {FAT* ’20}, doi = {10.1145/3351095.3372855}, line = {1,4}, month = jan, publisher = {ACM}, series = {FAT* ’20}, title = {Doctor XAI: an ontology-based approach to black-box sequential data classification explanations}, year = {2020} }
Global Explanations with Local Scoring

Mattia Setzu, Riccardo Guidotti, Anna Monreale, and Franco Turini

Dec 2020

RESEARCH LINE
1

2

Abs Bib HTML

Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{Setzu_2020, author = {Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco}, booktitle = {Communications in Computer and Information Science}, doi = {10.1007/978-3-030-43823-4_14}, isbn = {9783030438234}, issn = {1865-0937}, line = {1,2}, month = dec, pages = {159–171}, publisher = {Springer International Publishing}, title = {Global Explanations with Local Scoring}, year = {2020} }

2019

Helping Your Docker Images to Spread Based on Explainable Models

Riccardo Guidotti, Jacopo Soldani, Davide Neri, Antonio Brogi, and Dino Pedreschi

Dec 2019

RESEARCH LINE
1

Abs Bib HTML

Docker is on the rise in today’s enterprise IT. It permits shipping applications inside portable containers, which run from so-called Docker images. Docker images are distributed in public registries, which also monitor their popularity. The popularity of an image impacts on its actual usage, and hence on the potential revenues for its developers. In this paper, we present a solution based on interpretable decision tree and regression trees for estimating the popularity of a given Docker image, and for understanding how to improve an image to increase its popularity. The results presented in this work can provide valuable insights to Docker developers, helping them in spreading their images.
@inbook{Guidotti_2019_Helping, author = {Guidotti, Riccardo and Soldani, Jacopo and Neri, Davide and Brogi, Antonio and Pedreschi, Dino}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-10997-4_13}, isbn = {9783030109974}, issn = {1611-3349}, line = {1}, month = dec, pages = {205–221}, publisher = {Springer International Publishing}, title = {Helping Your Docker Images to Spread Based on Explainable Models}, year = {2019} }
Factual and Counterfactual Explanations for Black Box Decision Making

Riccardo Guidotti, Anna Monreale, Fosca Giannotti, Dino Pedreschi, Salvatore Ruggieri, and 1 more author

IEEE Intelligent Systems, Nov 2019

RESEARCH LINE
1

Abs Bib HTML

The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{Guidotti_2019Factual, author = {Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1109/mis.2019.2957223}, issn = {1941-1294}, journal = {IEEE Intelligent Systems}, line = {1}, month = nov, number = {6}, pages = {14–23}, publisher = {Institute of Electrical and Electronics Engineers (IEEE)}, title = {Factual and Counterfactual Explanations for Black Box Decision Making}, volume = {34}, year = {2019} }
Investigating Neighborhood Generation Methods for Explanations of Obscure Image Classifiers

Riccardo Guidotti, Anna Monreale, and Leonardo Cariaggi

Dec 2019

RESEARCH LINE
1

Abs Bib HTML

Given the wide use of machine learning approaches based on opaque prediction models, understanding the reasons behind decisions of black box decision systems is nowadays a crucial topic. We address the problem of providing meaningful explanations in the widely-applied image classification tasks. In particular, we explore the impact of changing the neighborhood generation function for a local interpretable model-agnostic explanator by proposing four different variants. All the proposed methods are based on a grid-based segmentation of the images, but each of them proposes a different strategy for generating the neighborhood of the image for which an explanation is required. A deep experimentation shows both improvements and weakness of each proposed approach.
@inbook{Guidotti_2019Investigating, author = {Guidotti, Riccardo and Monreale, Anna and Cariaggi, Leonardo}, booktitle = {Lecture Notes in Computer Science}, doi = {10.1007/978-3-030-16148-4_5}, isbn = {9783030161484}, issn = {1611-3349}, line = {1}, month = dec, pages = {55–68}, publisher = {Springer International Publishing}, title = {Investigating Neighborhood Generation Methods for Explanations of Obscure Image Classifiers}, year = {2019} }
Explaining Multi-label Black-Box Classifiers for Health Applications

Cecilia Panigutti, Riccardo Guidotti, Anna Monreale, and Dino Pedreschi

Aug 2019

RESEARCH LINE
1

4

Abs Bib HTML

Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{Panigutti_2019, author = {Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino}, booktitle = {Studies in Computational Intelligence}, doi = {10.1007/978-3-030-24409-5_9}, isbn = {9783030244095}, issn = {1860-9503}, line = {1,4}, month = aug, pages = {97–110}, publisher = {Springer International Publishing}, title = {Explaining Multi-label Black-Box Classifiers for Health Applications}, year = {2019} }
Meaningful Explanations of Black Box AI Decision Systems

Dino Pedreschi, Fosca Giannotti, Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, and 1 more author

Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019

RESEARCH LINE
1

Abs Bib HTML

Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{Pedreschi_2019, author = {Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco}, doi = {10.1609/aaai.v33i01.33019780}, issn = {2159-5399}, journal = {Proceedings of the AAAI Conference on Artificial Intelligence}, line = {1}, month = jul, number = {01}, pages = {9780–9784}, publisher = {Association for the Advancement of Artificial Intelligence (AAAI)}, title = {Meaningful Explanations of Black Box AI Decision Systems}, volume = {33}, year = {2019} }

2018

A Survey of Methods for Explaining Black Box Models

Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and 1 more author

ACM Computing Surveys, Aug 2018

RESEARCH LINE
1

Abs Bib HTML

In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{Guidotti_2018, author = {Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino}, doi = {10.1145/3236009}, issn = {1557-7341}, journal = {ACM Computing Surveys}, line = {1}, month = aug, number = {5}, pages = {1–42}, publisher = {Association for Computing Machinery (ACM)}, title = {A Survey of Methods for Explaining Black Box Models}, volume = {51}, year = {2018} }