Publications by year in reversed chronological order.
2024
Generative Model for Decision Trees
Riccardo
Guidotti, Anna
Monreale, Mattia
Setzu, and Giulia
Volpi
In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI
2024, Thirty-Sixth Conference on Innovative Applications of Artificial
Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances
in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver,
Canada , Dec 2024
Decision trees are among the most popular supervised mod- els due to their interpretability and knowledge representation resembling human reasoning. Commonly-used decision tree induction algorithms are based on greedy top-down strategies. Although these approaches are known to be an efficient heuris- tic, the resulting trees are only locally optimal and tend to have overly complex structures. On the other hand, optimal decision tree algorithms attempt to create an entire decision tree at once to achieve global optimality. We place our proposal between these approaches by designing a generative model for deci- sion trees. Our method first learns a latent decision tree space through a variational architecture using pre-trained decision tree models. Then, it adopts a genetic procedure to explore such latent space to find a compact decision tree with good predictive performance. We compare our proposal against clas- sical tree induction methods, optimal approaches, and ensem- ble models. The results show that our proposal can generate accurate and shallow, i.e., interpretable, decision trees.
@inproceedings{DBLP:conf/aaai/GuidottiMSV24,author={Guidotti, Riccardo and Monreale, Anna and Setzu, Mattia and Volpi, Giulia},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/conf/aaai/GuidottiMSV24.bib},booktitle={Thirty-Eighth {AAAI} Conference on Artificial Intelligence, {AAAI}
2024, Thirty-Sixth Conference on Innovative Applications of Artificial
Intelligence, {IAAI} 2024, Fourteenth Symposium on Educational Advances
in Artificial Intelligence, {EAAI} 2014, February 20-27, 2024, Vancouver,
Canada},doi={10.1609/AAAI.V38I19.30104},editor={Wooldridge, Michael J. and Dy, Jennifer G. and Natarajan, Sriraam},line={1},month=dec,pages={21116--21124},publisher={{AAAI} Press},timestamp={Tue, 02 Apr 2024 16:32:09 +0200},title={Generative Model for Decision Trees},year={2024},}
FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers
Guillermo
Fernández, Riccardo
Guidotti, Fosca
Giannotti, Mattia
Setzu, Juan A.
Aledo, and
2 more authors
In Advances in Intelligent Data Analysis XXII - 22nd International
Symposium on Intelligent Data Analysis, IDA 2024, Stockholm, Sweden,
April 24-26, 2024, Proceedings, Part II , Dec 2024
The need for explanation for new, complex machine learning models has caused the rise and growth of the field of eXplainable Artificial Intelligence. Different explanation types arise, such as local explanations which focus on the classification for a particular instance, or global explanations which aim to show a global overview of the inner workings of the model. In this paper, we propose FLocalX, a framework that builds a fuzzy global explanation expressed in terms of fuzzy rules by using local explanations as a starting point and a metaheuristic optimization process to obtain the result. An initial experimentation has been carried out with a genetic algorithm as the optimization process. Across several datasets, black-box algorithms and local explanation methods, FLocalX has been tested in terms of both fidelity of the resulting global explanation, and complexity The results show that FLocalX is successfully able to generate short and understandable global explanations that accurately imitate the classifier.
@inproceedings{DBLP:conf/ida/FernandezGGSAGP24,author={Fern{\'{a}}ndez, Guillermo and Guidotti, Riccardo and Giannotti, Fosca and Setzu, Mattia and Aledo, Juan A. and G{\'{a}}mez, Jos{\'{e}} A. and Puerta, Jos{\'{e}} Miguel},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/conf/ida/FernandezGGSAGP24.bib},booktitle={Advances in Intelligent Data Analysis {XXII} - 22nd International
Symposium on Intelligent Data Analysis, {IDA} 2024, Stockholm, Sweden,
April 24-26, 2024, Proceedings, Part {II}},doi={10.1007/978-3-031-58553-1\_16},editor={Miliou, Ioanna and Piatkowski, Nico and Papapetrou, Panagiotis},line={1},month=dec,pages={197--209},publisher={Springer},series={Lecture Notes in Computer Science},timestamp={Fri, 17 May 2024 21:42:12 +0200},title={FLocalX - Local to Global Fuzzy Explanations for Black Box Classifiers},volume={14642},year={2024},openaccess={false}}
A Frank System for Co-Evolutionary Hybrid Decision-Making
Federico
Mazzoni, Riccardo
Guidotti, and Alessio
Malizia
In Advances in Intelligent Data Analysis XXII - 22nd International
Symposium on Intelligent Data Analysis, IDA 2024, Stockholm, Sweden,
April 24-26, 2024, Proceedings, Part II , Apr 2024
We introduce FRANK, a human-in-the-loop system for co-evolutionary hybrid decision-making aiding the user to label records from an un-labeled dataset. FRANK employs incremental learning to “evolve” in parallel with the user’s decisions, by training an interpretable machine learning model on the records labeled by the user. Furthermore, advances state-of-the-art approaches by offering inconsistency controls, explanations, fairness checks, and bad-faith safeguards simultaneously. We evaluate our proposal by simulating the users’ behavior with various levels of expertise and reliance on FRANK’s suggestions. The experiments show that FRANK’s intervention leads to improvements in the accuracy and the fairness of the decisions.
@inproceedings{DBLP:conf/ida/MazzoniGM24,author={Mazzoni, Federico and Guidotti, Riccardo and Malizia, Alessio},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/conf/ida/MazzoniGM24.bib},booktitle={Advances in Intelligent Data Analysis {XXII} - 22nd International
Symposium on Intelligent Data Analysis, {IDA} 2024, Stockholm, Sweden,
April 24-26, 2024, Proceedings, Part {II}},doi={10.1007/978-3-031-58553-1\_19},editor={Miliou, Ioanna and Piatkowski, Nico and Papapetrou, Panagiotis},line={1},month=apr,pages={236--248},publisher={Springer},series={Lecture Notes in Computer Science},timestamp={Tue, 07 May 2024 20:09:13 +0200},title={A Frank System for Co-Evolutionary Hybrid Decision-Making},volume={14642},year={2024},openaccess={false},}
AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems
Clara
Punzi, Roberto
Pellungrini, Mattia
Setzu, Fosca
Giannotti, and Dino
Pedreschi
Everyday we increasingly rely on machine learning models to automate and support high-stake tasks and decisions. This growing presence means that humans are now constantly interacting with machine learning-based systems, training and using models everyday. Several different techniques in computer science literature account for the human interaction with machine learning systems, but their classification is sparse and the goals varied. This survey proposes a taxonomy of Hybrid Decision Making Systems, providing both a conceptual and technical framework for understanding how current computer science literature models interaction between humans and machines.
@article{DBLP:journals/corr/abs-2402-06287,author={Punzi, Clara and Pellungrini, Roberto and Setzu, Mattia and Giannotti, Fosca and Pedreschi, Dino},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/journals/corr/abs-2402-06287.bib},doi={10.48550/ARXIV.2402.06287},eprint={2402.06287},eprinttype={arXiv},journal={CoRR},line={1,2},month=dec,timestamp={Tue, 07 May 2024 20:16:53 +0200},title={AI, Meet Human: Learning Paradigms for Hybrid Decision Making Systems},volume={abs/2402.06287},year={2024},}
FairBelief - Assessing Harmful Beliefs in Language Models
Mattia
Setzu, Marta Marchiori
Manerba, Pasquale
Minervini, and Debora
Nozza
Language Models (LMs) have been shown to inherit undesired biases that might hurt minorities and underrepresented groups if such systems were integrated into real-world applications without careful fairness auditing. This paper proposes FairBelief, an analytical approach to capture and assess beliefs, i.e., propositions that an LM may embed with different degrees of confidence and that covertly influence its predictions. With FairBelief, we leverage prompting to study the behavior of several state-of-the-art LMs across different previously neglected axes, such as model scale and likelihood, assessing predictions on a fairness dataset specifically designed to quantify LMs’ outputs’ hurtfulness. Finally, we conclude with an in-depth qualitative assessment of the beliefs emitted by the models. We apply FairBelief to English LMs, revealing that, although these architectures enable high performances on diverse natural language processing tasks, they show hurtful beliefs about specific genders. Interestingly, training procedure and dataset, model scale, and architecture induce beliefs of different degrees of hurtfulness.
@article{DBLP:journals/corr/abs-2402-17389,author={Setzu, Mattia and Manerba, Marta Marchiori and Minervini, Pasquale and Nozza, Debora},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/journals/corr/abs-2402-17389.bib},doi={10.48550/ARXIV.2402.17389},eprint={2402.17389},eprinttype={arXiv},journal={CoRR},line={2,5},month=dec,timestamp={Tue, 07 May 2024 20:15:26 +0200},title={FairBelief - Assessing Harmful Beliefs in Language Models},volume={abs/2402.17389},year={2024}}
Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification
Carlo
Metta, Andrea
Beretta, Riccardo
Guidotti, Yuan
Yin, Patrick
Gallinari, and
2 more authors
A crucial challenge in critical settings like medical diagnosis is making deep learning models used in decision-making systems interpretable. Efforts in Explainable Artificial Intelligence (XAI) are underway to address this challenge. Yet, many XAI methods are evaluated on broad classifiers and fail to address complex, real-world issues, such as medical diagnosis. In our study, we focus on enhancing user trust and confidence in automated AI decision-making systems, particularly for diagnosing skin lesions, by tailoring an XAI method to explain an AI model’s ability to identify various skin lesion types. We generate explanations using synthetic images of skin lesions as examples and counterexamples, offering a method for practitioners to pinpoint the critical features influencing the classification outcome. A validation survey involving domain experts, novices, and laypersons has demonstrated that explanations increase trust and confidence in the automated decision system. Furthermore, our exploration of the model’s latent space reveals clear separations among the most common skin lesion classes, a distinction that likely arises from the unique characteristics of each class and could assist in correcting frequent misdiagnoses by human professionals.
@article{diagnostics14070753,article-number={753},author={Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca},doi={10.3390/diagnostics14070753},issn={2075-4418},journal={Diagnostics},line={3,4},month=dec,number={7},pubmedid={38611666},title={Advancing Dermatological Diagnostics: Interpretable AI for Enhanced Skin Lesion Classification},volume={14},year={2024},}
Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence
Carlo
Metta, Andrea
Beretta, Roberto
Pellungrini, Salvatore
Rinzivillo, and Fosca
Giannotti
This paper focuses on the use of local Explainable Artificial Intelligence (XAI) methods, particularly the Local Rule-Based Explanations (LORE) technique, within healthcare and medical settings. It emphasizes the critical role of interpretability and transparency in AI systems for diagnosing diseases, predicting patient outcomes, and creating personalized treatment plans. While acknowledging the complexities and inherent trade-offs between interpretability and model performance, our work underscores the significance of local XAI methods in enhancing decision-making processes in healthcare. By providing granular, case-specific insights, local XAI methods like LORE enhance physicians’ and patients’ understanding of machine learning models and their outcome. Our paper reviews significant contributions to local XAI in healthcare, highlighting its potential to improve clinical decision making, ensure fairness, and comply with regulatory standards.
@article{metta2024towards,author={Metta, Carlo and Beretta, Andrea and Pellungrini, Roberto and Rinzivillo, Salvatore and Giannotti, Fosca},journal={Bioengineering},line={1,4},month=dec,number={4},pages={369},publisher={MDPI},title={Towards Transparent Healthcare: Advancing Local Explanation Methods in Explainable Artificial Intelligence},volume={11},year={2024},openaccess={true},}
An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives
Benedetta
Muscato, Chandana Sree
Mala, Marta
Marchiori Manerba, Gizem
Gezici, and Fosca
Giannotti
In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024 , May 2024
The varied backgrounds and experiences of human annotators inject different opinions and potential biases into the data, inevitably leading to disagreements. Yet, traditional aggregation methods fail to capture individual judgments since they rely on the notion of a single ground truth. Our aim is to review prior contributions to pinpoint the shortcomings that might cause stereotypical content generation. As a preliminary study, our purpose is to investigate state-of-the-art approaches, primarily focusing on the following two research directions. First, we investigate how adding subjectivity aspects to LLMs might guarantee diversity. We then look into the alignment between humans and LLMs and discuss how to measure it. Considering existing gaps, our review explores possible methods to mitigate the perpetuation of biases targeting specific communities. However, we recognize the potential risk of disseminating sensitive information due to the utilization of socio-demographic data in the training process. These considerations underscore the inclusion of diverse perspectives while taking into account the critical importance of implementing robust safeguards to protect individuals’ privacy and prevent the inadvertent propagation of sensitive information.
@inproceedings{muscato-etal-2024-overview,address={Torino, Italia},author={Muscato, Benedetta and Mala, Chandana Sree and Marchiori Manerba, Marta and Gezici, Gizem and Giannotti, Fosca},booktitle={Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives) @ LREC-COLING 2024},editor={Abercrombie, Gavin and Basile, Valerio and Bernadi, Davide and Dudy, Shiran and Frenda, Simona and Havens, Lucy and Tonelli, Sara},line={1,2},month=may,pages={49--55},publisher={ELRA and ICCL},title={An Overview of Recent Approaches to Enable Diversity in Large Language Models through Aligning with Human Perspectives},year={2024},}
Explainable Authorship Identification in Cultural Heritage Applications
Mattia
Setzu, Silvia
Corbara, Anna
Monreale, Alejandro
Moreo, and Fabrizio
Sebastiani
Journal on Computing and Cultural Heritage, Jun 2024
While a substantial amount of work has recently been devoted to improving the accuracy of computational Authorship Identification (AId) systems for textual data, little to no attention has been paid to endowing AId systems with the ability to explain the reasons behind their predictions. This substantially hinders the practical application of AId methods, since the predictions returned by such systems are hardly useful unless they are supported by suitable explanations. In this article, we explore the applicability of existing general-purpose eXplainable Artificial Intelligence (XAI) techniques to AId, with a focus on explanations addressed to scholars working in cultural heritage. In particular, we assess the relative merits of three different types of XAI techniques (feature ranking, probing, factual and counterfactual selection) on three different AId tasks (authorship attribution, authorship verification and same-authorship verification) by running experiments on real AId textual data. Our analysis shows that, while these techniques make important first steps towards XAI, more work remains to be done to provide tools that can be profitably integrated into the workflows of scholars.
@article{Setzu_2024,title={Explainable Authorship Identification in Cultural Heritage Applications},volume={17},issn={1556-4711},url={http://dx.doi.org/10.1145/3654675},doi={10.1145/3654675},number={3},journal={Journal on Computing and Cultural Heritage},publisher={Association for Computing Machinery (ACM)},author={Setzu, Mattia and Corbara, Silvia and Monreale, Anna and Moreo, Alejandro and Sebastiani, Fabrizio},year={2024},month=jun,pages={1--23},line={4},}
Exploring Large Language Models Capabilities to Explain Decision Trees
Paulo Bruno
Serafim, Pierluigi
Crescenzi, Gizem
Gezici, Eleonora
Cappuccio, Salvatore
Rinzivillo, and
1 more author
Decision trees are widely adopted in Machine Learning tasks due to their operation simplicity and interpretability aspects. However, following the decision process path taken by trees can be difficult in a complex scenario or in a case where a user has no familiarity with them. Prior research showed that converting outcomes to natural language is an accessible way to facilitate understanding for non-expert users in several tasks. More recently, there has been a growing effort to use Large Language Models (LLMs) as a tool for providing natural language texts. In this paper, we examine the proficiency of LLMs to explain decision tree predictions in simple terms through the generation of natural language explanations. By exploring different textual representations and prompt engineering strategies, we identify capabilities that strengthen LLMs as a competent explainer as well as highlight potential challenges and limitations, opening further research possibilities on natural language explanations for decision trees.
@inbook{Serafim_2024,title={Exploring Large Language Models Capabilities to Explain Decision Trees},isbn={9781643685229},issn={1879-8314},doi={10.3233/faia240183},booktitle={HHAI 2024: Hybrid Human AI Systems for the Social Good},publisher={IOS Press},author={Serafim, Paulo Bruno and Crescenzi, Pierluigi and Gezici, Gizem and Cappuccio, Eleonora and Rinzivillo, Salvatore and Giannotti, Fosca},year={2024},month=jun,line={1,3}}
Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma
Giulia
Peduzzi, Alessio
Felici, Roberto
Pellungrini, Francesca
Giorgolo, Riccardo
Farinella, and
7 more authors
Background: The current knowledge on pancreatic ductal adenocarcinoma (PDAC) risk factors is limited and no study has comprehensively tested the exposome in combination with the genetic variability in relation to the disease susceptibility.Aim:The aim of this study was to analyze the exposome and its interaction with known genetic susceptibility loci, in relation to PDAC risk. Methods:A case-control study nested in UK Biobank cohort was conducted on 816 PDAC cases and 302,645 controls. A total of 347 exposure variables, and a polygenic risk score (PRS) were analyzed through logistic regression. Gene-environment interaction analyses were conducted. Results: A total of 52 associations under the Bonferroni corrected threshold of p < 1.46 × 10−4 were observed. Known risk factors such as smoking, pancreatitis, diabetes, PRS, heavy alcohol drinking and overweight were replicated in this study. As for novel associations, a clear indication for length and intensity of mobile phone use and the stress-related factors and stressful events with increase of PDAC risk was observed. Although the PRS was associated with PDAC risk (P = 2.09 × 10−9), statistically significant gene-exposome interactions were not identified. Conclusion: In conclusion, our results suggest that a stressful lifestyle and sedentary behaviors may play a major role in PDAC susceptibility independently from the genetic background.
@article{Peduzzi_2024,title={Analysis of exposome and genetic variability suggests stress as a major contributor for development of pancreatic ductal adenocarcinoma},volume={56},issn={1590-8658},doi={10.1016/j.dld.2023.10.015},number={6},journal={Digestive and Liver Disease},publisher={Elsevier BV},author={Peduzzi, Giulia and Felici, Alessio and Pellungrini, Roberto and Giorgolo, Francesca and Farinella, Riccardo and Gentiluomo, Manuel and Spinelli, Andrea and Capurso, Gabriele and Monreale, Anna and Canzian, Federico and Calderisi, Marco and Campa, Daniele},year={2024},month=jun,line={4},pages={1054–1063},openaccess={true}}
Interpretable and Fair Mechanisms for Abstaining Classifiers
Daphne
Lenders, Andrea
Pugnana, Roberto
Pellungrini, Toon
Calders, Dino
Pedreschi, and
1 more author
Abstaining classifiers have the option to refrain from providing a prediction for instances that are difficult to classify. The abstention mechanism is designed to trade off the classifier’s performance on the accepted data while ensuring a minimum number of predictions. In this setting, often fairness concerns arise when the abstention mechanism solely reduces errors for the majority groups of the data, resulting in increased performance differences across demographic groups. While there exist a bunch of methods that aim to reduce discrimination when abstaining, there is no mechanism that can do so in an explainable way. In this paper, we fill this gap by introducing Interpretable and Fair Abstaining Classifier (IFAC), an algorithm that can reject predictions both based on their uncertainty and their unfairness. By rejecting possibly unfair predictions, our method reduces error and positive decision rate differences across demographic groups of the non-rejected data. Since the unfairness-based rejections are based on an interpretable-by-design method, i.e., rule-based fairness checks and situation testing, we create a transparent process that can empower human decision-makers to review the unfair predictions and make more just decisions for them. This explainable aspect is especially important in light of recent AI regulations, mandating that any high-risk decision task should be overseen by human experts to reduce discrimination risks. (Code and Appendix for this work is available on: https://github.com/calathea21/IFAC).
@inbook{Lenders_2024,title={Interpretable and Fair Mechanisms for Abstaining Classifiers},isbn={9783031703683},issn={1611-3349},doi={10.1007/978-3-031-70368-3_25},booktitle={Machine Learning and Knowledge Discovery in Databases. Research Track},publisher={Springer Nature Switzerland},author={Lenders, Daphne and Pugnana, Andrea and Pellungrini, Roberto and Calders, Toon and Pedreschi, Dino and Giannotti, Fosca},year={2024},pages={416–433},line={4},openaccess={false}}
A Frank System for Co-Evolutionary Hybrid Decision-Making
Federico
Mazzoni, Riccardo
Guidotti, and Alessio
Malizia
We introduce Frank, a human-in-the-loop system for co-evolutionary hybrid decision-making aiding the user to label records from an un-labeled dataset. Frank employs incremental learning to “evolve” in parallel with the user’s decisions, by training an interpretable machine learning model on the records labeled by the user. Furthermore, advances state-of-the-art approaches by offering inconsistency controls, explanations, fairness checks, and bad-faith safeguards simultaneously. We evaluate our proposal by simulating the users’ behavior with various levels of expertise and reliance on Frank’s suggestions. The experiments show that Frank’s intervention leads to improvements in the accuracy and the fairness of the decisions.
@inbook{Mazzoni_2024,title={A Frank System for Co-Evolutionary Hybrid Decision-Making},isbn={9783031585531},issn={1611-3349},doi={10.1007/978-3-031-58553-1_19},booktitle={Advances in Intelligent Data Analysis XXII},publisher={Springer Nature Switzerland},author={Mazzoni, Federico and Guidotti, Riccardo and Malizia, Alessio},year={2024},pages={236–248},}
Explaining Siamese networks in few-shot learning
Andrea
Fedele, Riccardo
Guidotti, and Dino
Pedreschi
Machine learning models often struggle to generalize accurately when tested on new class distributions that were not present in their training data. This is a significant challenge for real-world applications that require quick adaptation without the need for retraining. To address this issue, few-shot learning frameworks, which includes models such as Siamese Networks, have been proposed. Siamese Networks learn similarity between pairs of records through a metric that can be easily extended to new, unseen classes. However, these systems lack interpretability, which can hinder their use in certain applications. To address this, we propose a data-agnostic method to explain the outcomes of Siamese Networks in the context of few-shot learning. Our explanation method is based on a post-hoc perturbation-based procedure that evaluates the contribution of individual input features to the final outcome. As such, it falls under the category of post-hoc explanation methods. We present two variants, one that considers each input feature independently, and another that evaluates the interplay between features. Additionally, we propose two perturbation procedures to evaluate feature contributions. Qualitative and quantitative results demonstrate that our method is able to identify highly discriminant intra-class and inter-class characteristics, as well as predictive behaviors that lead to misclassification by relying on incorrect features.
@article{Fedele_2024,title={Explaining Siamese networks in few-shot learning},volume={113},issn={1573-0565},doi={10.1007/s10994-024-06529-8},number={10},journal={Machine Learning},publisher={Springer Science and Business Media LLC},author={Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino},year={2024},month=apr,pages={7723–7760},openaccess={true}}
Drifting explanations in continual learning
Andrea
Cossu, Francesco
Spinnato, Riccardo
Guidotti, and Davide
Bacciu
Continual Learning (CL) trains models on streams of data, with the aim of learning new information without forgetting previous knowledge. However, many of these models lack interpretability, making it difficult to understand or explain how they make decisions. This lack of interpretability becomes even more challenging given the non-stationary nature of the data streams in CL. Furthermore, CL strategies aimed at mitigating forgetting directly impact the learned representations. We study the behavior of different explanation methods in CL and propose CLEX (ContinuaL EXplanations), an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios, where forgetting is pronounced. We observed that models with similar predictive accuracy do not generate similar explanations. Replay-based strategies, well-known to be some of the most effective ones in class-incremental scenarios, are able to generate explanations that are aligned to the ones of a model trained offline. On the contrary, naive fine-tuning often results in degenerate explanations that drift from the ones of an offline model. Finally, we discovered that even replay strategies do not always operate at best when applied to fully-trained recurrent models. Instead, randomized recurrent models (leveraging on an untrained recurrent component) clearly reduce the drift of the explanations. This discrepancy between fully-trained and randomized recurrent models, previously known only in the context of their predictive continual performance, is more general, including also continual explanations.
@article{COSSU2024127960,title={Drifting explanations in continual learning},journal={Neurocomputing},volume={597},pages={127960},year={2024},issn={0925-2312},doi={https://doi.org/10.1016/j.neucom.2024.127960},url={https://www.sciencedirect.com/science/article/pii/S0925231224007318},author={Cossu, Andrea and Spinnato, Francesco and Guidotti, Riccardo and Bacciu, Davide},keywords={Continual learning, Explainable AI, Lifelong learning, Recurrent neural networks},line={1},openaccess={true}}
Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions
Luca
Longo, Mario
Brcic, Federico
Cabitza, Jaesik
Choi, Roberto
Confalonieri, and
14 more authors
Understanding black box models has become paramount as systems based on opaque Artificial Intelligence (AI) continue to flourish in diverse real-world applications. In response, Explainable AI (XAI) has emerged as a field of research with practical and ethical benefits across various domains. This paper highlights the advancements in XAI and its application in real-world scenarios and addresses the ongoing challenges within XAI, emphasizing the need for broader perspectives and collaborative efforts. We bring together experts from diverse fields to identify open problems, striving to synchronize research agendas and accelerate XAI in practical applications. By fostering collaborative discussion and interdisciplinary cooperation, we aim to propel XAI forward, contributing to its continued success. We aim to develop a comprehensive proposal for advancing XAI. To achieve this goal, we present a manifesto of 28 open problems categorized into nine categories. These challenges encapsulate the complexities and nuances of XAI and offer a road map for future research. For each problem, we provide promising research directions in the hope of harnessing the collective intelligence of interested stakeholders.
@article{LONGO2024102301,title={Explainable Artificial Intelligence (XAI) 2.0: A manifesto of open challenges and interdisciplinary research directions},journal={Information Fusion},volume={106},pages={102301},year={2024},issn={1566-2535},doi={https://doi.org/10.1016/j.inffus.2024.102301},url={https://www.sciencedirect.com/science/article/pii/S1566253524000794},author={Longo, Luca and Brcic, Mario and Cabitza, Federico and Choi, Jaesik and Confalonieri, Roberto and Ser, Javier Del and Guidotti, Riccardo and Hayashi, Yoichi and Herrera, Francisco and Holzinger, Andreas and Jiang, Richard and Khosravi, Hassan and Lecue, Freddy and Malgieri, Gianclaudio and Páez, Andrés and Samek, Wojciech and Schneider, Johannes and Speith, Timo and Stumpf, Simone},keywords={Explainable artificial intelligence, XAI, Interpretability, Manifesto, Open challenges, Interdisciplinarity, Ethical AI, Large language models, Trustworthy AI, Responsible AI, Generative AI, Multi-faceted explanations, Concept-based explanations, Causality, Actionable XAI, Falsifiability},line={1}}
Social Bias Probing: Fairness Benchmarking for Language Models
Marta
Marchiori Manerba, Karolina
Stanczak, Riccardo
Guidotti, and Isabelle
Augenstein
In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Nov 2024
While the impact of social biases in language models has been recognized, prior methods for bias evaluation have been limited to binary association tests on small datasets, limiting our understanding of bias complexities. This paper proposes a novel framework for probing language models for social biases by assessing disparate treatment, which involves treating individuals differently according to their affiliation with a sensitive demographic group. We curate SoFa, a large-scale benchmark designed to address the limitations of existing fairness collections. SoFa expands the analysis beyond the binary comparison of stereotypical versus anti-stereotypical identities to include a diverse range of identities and stereotypes. Comparing our methodology with existing benchmarks, we reveal that biases within language models are more nuanced than acknowledged, indicating a broader scope of encoded biases than previously recognized. Benchmarking LMs on SoFa, we expose how identities expressing different religions lead to the most pronounced disparate treatments across all models. Finally, our findings indicate that real-life adversities faced by various groups such as women and people with disabilities are mirrored in the behavior of these models.
@inproceedings{marchiori-manerba-etal-2024-social,title={Social Bias Probing: Fairness Benchmarking for Language Models},author={Marchiori Manerba, Marta and Stanczak, Karolina and Guidotti, Riccardo and Augenstein, Isabelle},editor={Al-Onaizan, Yaser and Bansal, Mohit and Chen, Yun-Nung},booktitle={Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},month=nov,year={2024},address={Miami, Florida, USA},publisher={Association for Computational Linguistics},url={https://aclanthology.org/2024.emnlp-main.812},pages={14653--14671},line={5},}
Data-Agnostic Pivotal Instances Selection for Decision-Making Models
Alessio
Cascione, Mattia
Setzu, and Riccardo
Guidotti
In Machine Learning and Knowledge Discovery in Databases. Research Track , Nov 2024
As decision-making processes become increasingly complex, machine learning tools have become essential resources for tackling business and social issues. However, many methodologies rely on complex models that experts and everyday users cannot really interpret or understand. This is why constructing interpretable models is crucial. Humans typically make decisions by comparing the case at hand with a few exemplary and representative cases imprinted in their minds. Our objective is to design an approach that can select such exemplary cases, which we call pivots, to build an interpretable predictive model. To this aim, we propose a hierarchical and interpretable pivot selection model inspired by Decision Trees, and based on the similarity between pivots and input instances. Such a model can be used both as a pivot selection method, and as a standalone predictive model. By design, our proposal can be applied to any data type, as we can exploit pre-trained networks for data transformation. Through experiments on various datasets of tabular data, texts, images, and time series, we have demonstrated the superiority of our proposal compared to naive alternatives and state-of-the-art instance selectors, while minimizing the model complexity, i.e., the number of pivots identified.
@inproceedings{10.1007/978-3-031-70341-6_22,author={Cascione, Alessio and Setzu, Mattia and Guidotti, Riccardo},editor={Bifet, Albert and Davis, Jesse and Krilavi{\v{c}}ius, Tomas and Kull, Meelis and Ntoutsi, Eirini and {\v{Z}}liobait{\.{e}}, Indr{\.{e}}},title={Data-Agnostic Pivotal Instances Selection for Decision-Making Models},booktitle={Machine Learning and Knowledge Discovery in Databases. Research Track},year={2024},publisher={Springer Nature Switzerland},address={Cham},pages={367--386},isbn={978-3-031-70341-6},doi={10.1007/978-3-031-70341-6_22},openaccess={false},line={1}}
Causality-Aware Local Interpretable Model-Agnostic Explanations
A main drawback of eXplainable Artificial Intelligence (XAI) approaches is the feature independence assumption, hindering the study of potential variable dependencies. This leads to approximating black box behaviors by analyzing the effects on randomly generated feature values that may rarely occur in the original samples. This paper addresses this issue by integrating causal knowledge in an XAI method to enhance transparency and enable users to assess the quality of the generated explanations. Specifically, we propose a novel extension to a widely used local and model-agnostic explainer, which encodes explicit causal relationships within the data surrounding the instance being explained. Extensive experiments show that our approach overcomes the original method in terms of faithfully replicating the black-box model’s mechanism and the consistency and reliability of the generated explanations.
@inproceedings{10.1007/978-3-031-63800-8_6,author={Cinquini, Martina and Guidotti, Riccardo},editor={Longo, Luca and Lapuschkin, Sebastian and Seifert, Christin},title={Causality-Aware Local Interpretable Model-Agnostic Explanations},booktitle={Explainable Artificial Intelligence},year={2024},publisher={Springer Nature Switzerland},address={Cham},pages={108--124},isbn={978-3-031-63800-8},doi={10.1007/978-3-031-63800-8_6},openaccess={false},line={2},month=jul}
Bridging the Gap in Hybrid Decision-Making Systems
Federico
Mazzoni, Roberto
Pellungrini, and Riccardo
Guidotti
We introduce BRIDGET, a novel human-in-the-loop system for hybrid decision-making, aiding the user to label records from an un-labeled dataset, attempting to “bridge the gap” between the two most popular Hybrid Decision-Making paradigms: those featuring the human in a leading position, and the other with a machine making most of the decisions. BRIDGET understands when either a machine or a human user should be in charge, dynamically switching between two statuses. In the different statuses, BRIDGET still fosters the human-AI interaction, either having a machine learning model assuming skeptical stances towards the user and offering them suggestions, or towards itself and calling the user back. We believe our proposal lays the groundwork for future synergistic systems involving a human and a machine decision-makers.
@misc{mazzoni2024bridginggaphybriddecisionmaking,title={Bridging the Gap in Hybrid Decision-Making Systems},author={Mazzoni, Federico and Pellungrini, Roberto and Guidotti, Riccardo},year={2024},eprint={2409.19415},archiveprefix={arXiv},primaryclass={cs.AI},url={https://arxiv.org/abs/2409.19415},line={1}}
An Interactive Interface for Feature Space Navigation
Eleonora
Cappuccio, Isacco
Beretta, Marta Marchiori
Manerba, and Salvatore
Rinzivillo
In HHAI 2024: Hybrid Human AI Systems for the Social Good - Proceedings
of the Third International Conference on Hybrid Human-Artificial Intelligence,
Malmö, Sweden, 10-14 June 2024 , Jul 2024
In this paper, we present Feature Space Navigator, an interactive interface that allows an exploration of the decision boundary of a model. The proposal aims to overcome the limitations of the techno-solutionist approach to explanations based on factual and counterfactual generation, reaffirming interactivity as a core value in designing the conversation between the model and the user. Starting from an instance, users can explore the feature space by selectively modifying the original instance, on the basis of her own knowledge and experience. The interface visually displays how model predictions react in response to the adjustments introduced by the users, letting them to identify relevant prototypes and counterfactuals. Our proposal leverages the autonomy and control of the users that can explore the behavior of the decision model accordingly with their own knowledge base, reducing the need for a dedicated explanation algorithm.
@inproceedings{CappuccioBMR24,author={Cappuccio, Eleonora and Beretta, Isacco and Manerba, Marta Marchiori and Rinzivillo, Salvatore},editor={Lorig, Fabian and Tucker, Jason and Lindstr{\"{o}}m, Adam Dahlgren and Dignum, Frank and Murukannaiah, Pradeep K. and Theodorou, Andreas and Yolum, Pinar},title={An Interactive Interface for Feature Space Navigation},booktitle={{HHAI} 2024: Hybrid Human {AI} Systems for the Social Good - Proceedings
of the Third International Conference on Hybrid Human-Artificial Intelligence,
Malm{\"{o}}, Sweden, 10-14 June 2024},series={Frontiers in Artificial Intelligence and Applications},volume={386},pages={73--83},publisher={{IOS} Press},year={2024},url={https://doi.org/10.3233/FAIA240184},doi={10.3233/FAIA240184},line={3},openaccess={false},}
A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions
Luca
Pappalardo, Emanuele
Ferragina, Salvatore
Citraro, Giuliano
Cornacchia, Mirco
Nanni, and
9 more authors
@misc{pappalardo2024surveyimpactaibasedrecommenders,title={A survey on the impact of AI-based recommenders on human behaviours: methodologies, outcomes and future directions},author={Pappalardo, Luca and Ferragina, Emanuele and Citraro, Salvatore and Cornacchia, Giuliano and Nanni, Mirco and Rossetti, Giulio and Gezici, Gizem and Giannotti, Fosca and Lalli, Margherita and Gambetta, Daniele and Mauro, Giovanni and Morini, Virginia and Pansanella, Valentina and Pedreschi, Dino},year={2024},eprint={2407.01630},archiveprefix={arXiv},primaryclass={cs.IR},url={https://arxiv.org/abs/2407.01630},month=jun,line={1},key={value}}
Inference through innovation processes tested in the authorship attribution task
Giulio
Tani Raffaelli, Margherita
Lalli, and Francesca
Tria
Urn models for innovation capture fundamental empirical laws shared by several real-world processes. The so-called urn model with triggering includes, as particular cases, the urn representation of the two-parameter Poisson-Dirichlet process and the Dirichlet process, seminal in Bayesian non-parametric inference. In this work, we leverage this connection to introduce a general approach for quantifying closeness between symbolic sequences and test it within the framework of the authorship attribution problem. The method demonstrates high accuracy when compared to other related methods in different scenarios, featuring a substantial gain in computational efficiency and theoretical transparency. Beyond the practical convenience, this work demonstrates how the recently established connection between urn models and non-parametric Bayesian inference can pave the way for designing more efficient inference methods. In particular, the hybrid approach that we propose allows us to relax the exchangeability hypothesis, which can be particularly relevant for systems exhibiting complex correlation patterns and non-stationary dynamics.
@article{tani_raffaelli_inference_2024,title={Inference through innovation processes tested in the authorship attribution task},volume={7},issn={2399-3650},url={https://doi.org/10.1038/s42005-024-01714-6},doi={10.1038/s42005-024-01714-6},number={1},journal={Communications Physics},author={Tani Raffaelli, Giulio and Lalli, Margherita and Tria, Francesca},month=sep,year={2024},pages={300},line={4},openaccess={true},}
DINE: Dimensional Interpretability of Node Embeddings
Simone
Piaggesi, Megha
Khosla, André
Panisson, and Avishek
Anand
IEEE Transactions on Knowledge and Data Engineering, Dec 2024
Conference Name: IEEE Transactions on Knowledge and Data Engineering
Graph representation learning methods, such as node embeddings, are powerful approaches to map nodes into a latent vector space, allowing their use for various graph learning tasks. Despite their success, these techniques are inherently black-boxes and few studies have focused on investigating local explanations of node embeddings for specific instances. Moreover, explaining the overall behavior of unsupervised embedding models remains an unexplored problem, limiting global interpretability and debugging potentials. We address this gap by developing human-understandable explanations for latent space dimensions in node embeddings. Towards that, we first develop new metrics that measure the global interpretability of embeddings based on the marginal contribution of the latent dimensions to predicting graph structure. We say an embedding dimension is more interpretable if it can faithfully map to an understandable sub-structure in the input graph - like community structure. Having observed that standard node embeddings have low interpretability, we then introduce Dine (Dimension-based Interpretable Node Embedding). This novel approach can retrofit existing node embeddings by making them more interpretable without sacrificing their task performance. We conduct extensive experiments on synthetic and real-world graphs and show that we can simultaneously learn highly interpretable node embeddings with effective performance in link prediction and node classification.
@article{piaggesi_dine_2024,title={{DINE}: {Dimensional} {Interpretability} of {Node} {Embeddings}},volume={36},issn={1558-2191},shorttitle={{DINE}},url={https://ieeexplore.ieee.org/document/10591463/authors#authors},doi={10.1109/TKDE.2024.3425460},number={12},urldate={2024-12-02},journal={IEEE Transactions on Knowledge and Data Engineering},author={Piaggesi, Simone and Khosla, Megha and Panisson, André and Anand, Avishek},month=dec,year={2024},note={Conference Name: IEEE Transactions on Knowledge and Data Engineering},keywords={Interpretability, link prediction, Measurement, node embeddings, Proteins, representation learning, Representation learning, Semantics, Social networking (online), Task analysis, Vectors},pages={7986--7997},file={Full Text PDF:/Users/dfadda/Zotero/storage/HSRG489V/Piaggesi et al. - 2024 - DINE Dimensional Interpretability of Node Embeddings.pdf:application/pdf;IEEE Xplore Abstract Record:/Users/dfadda/Zotero/storage/6RUCPYZJ/authors.html:text/html},openaccess={true},line={1}}
Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space
Simone
Piaggesi, Francesco
Bodria, Riccardo
Guidotti, Fosca
Giannotti, and Dino
Pedreschi
Artificial Intelligence decision-making systems have dramatically increased their predictive power in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose CP-ILS, a comprehensive interpretable feature reduction method for tabular data capable of generating Counterfactual and Prototypical post-hoc explanations using an Interpretable Latent Space. CP-ILS optimizes a transparent feature space whose similarity and linearity properties enable the easy extraction of local and global explanations for any pre-trained black-box model, in the form of counterfactual/prototype pairs. We evaluated the effectiveness of the created latent space by showing its capability to preserve pair-wise similarities like well-known dimensionality reduction techniques. Moreover, we assessed the quality of counterfactuals and prototypes generated with CP-ILS against state-of-the-art explainers, demonstrating that our approach obtains more robust, plausible, and accurate explanations than its competitors under most experimental conditions.
@article{10753432,author={Piaggesi, Simone and Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino},journal={IEEE Access},title={Counterfactual and Prototypical Explanations for Tabular Data via Interpretable Latent Space},year={2024},volume={12},number={},month=nov,pages={168983-169000},keywords={Measurement;Dimensionality reduction;Explainable AI;Decision making;Prototypes;Closed box;Optimization methods;Linearity;Feature extraction;Proposals;Learning latent representations;learning linear models;neural networks;counterfactual search;prototype search},doi={10.1109/ACCESS.2024.3496114},line={1},openaccess={true}}
Enhancing Echo State Networks with Gradient-based Explainability Methods
Francesco
Spinnato, Andrea
Cossu, Riccardo
Guidotti, Andrea
Ceni, Claudio
Gallicchio, and
1 more author
Recurrent Neural Networks are effective for analyzing temporal data, such as time series, but they often require costly and time-intensive training. Echo State Networks simplify the training process by using a fixed recurrent layer, the reservoir, and a trainable output layer, the readout. In sequence classification problems, the readout typically receives only the f inal state of the reservoir. However, averaging all states can sometimes be beneficial. In this work, we assess whether a weighted average of hidden states can enhance the Echo State Network performance. To this end, we propose a gradient-based, explainable technique to guide the contribution of each hidden state towards the final prediction. We show that our approach outperforms the naive average, as well as other baselines, in time series classification, particularly on noisy data.
@inproceedings{spinnato_enhancing_2024,address={Bruges (Belgium) and online},title={Enhancing {Echo} {State} {Networks} with {Gradient}-based {Explainability} {Methods}},isbn={978-2-87587-090-2},url={https://www.esann.org/sites/default/files/proceedings/2024/ES2024-78.pdf},doi={10.14428/esann/2024.ES2024-78},language={en},urldate={2024-12-04},booktitle={{ESANN} 2024 proceesdings},publisher={Ciaco - i6doc.com},author={Spinnato, Francesco and Cossu, Andrea and Guidotti, Riccardo and Ceni, Andrea and Gallicchio, Claudio and Bacciu, Davide},year={2024},pages={17--22},}
Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields
Francesco
Spinnato, Riccardo
Guidotti, Anna
Monreale, and Mirco
Nanni
The current trend in the literature on Time Series Classification is to develop increasingly accurate algorithms by combining multiple models in ensemble hybrids, representing time series in complex and expressive feature spaces, and extracting features from different representations of the same time series. As a consequence of this focus on predictive performance, the best time series classifiers are black-box models, which are not understandable from a human standpoint. Even the approaches that are regarded as interpretable, such as shapelet-based ones, rely on randomization to maintain computational efficiency. This poses challenges for interpretability, as the explanation can change from run to run. Given these limitations, we propose the Bag-Of-Receptive-Field (BORF), a fast, interpretable, and deterministic time series transform. Building upon the classical Bag-Of-Patterns, we bridge the gap between convolutional operators and discretization, enhancing the Symbolic Aggregate Approximation (SAX) with dilation and stride, which can more effectively capture temporal patterns at multiple scales. We propose an algorithmic speedup that reduces the time complexity associated with SAX-based classifiers, allowing the extension of the Bag-Of-Patterns to the more flexible Bag-Of-Receptive-Fields, represented as a sparse multivariate tensor. The empirical results from testing our proposal on more than 150 univariate and multivariate classification datasets demonstrate good accuracy and great computational efficiency compared to traditional SAX-based methods and state-of-the-art time series classifiers, while providing easy-to-understand explanations.
@article{10684604,author={Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco},journal={IEEE Access},title={Fast, Interpretable, and Deterministic Time Series Classification With a Bag-of-Receptive-Fields},year={2024},volume={12},month=oct,number={},pages={137893-137912},keywords={Time series analysis;Transforms;Indexes;Training;Time complexity;Standards;Time series analysis;Classification algorithms;Time series;classification;explainable AI;symbolic aggregate approximation},doi={10.1109/ACCESS.2024.3464743},openaccess={true}}
Mapping the landscape of ethical considerations in explainable AI research
Luca
Nannini, Marta
Marchiori Manerba, and Isacco
Beretta
With its potential to contribute to the ethical governance of AI, eXplainable AI (XAI) research frequently asserts its relevance to ethical considerations. Yet, the substantiation of these claims with rigorous ethical analysis and reflection remains largely unexamined. This contribution endeavors to scrutinize the relationship between XAI and ethical considerations. By systematically reviewing research papers mentioning ethical terms in XAI frameworks and tools, we investigate the extent and depth of ethical discussions in scholarly research. We observe a limited and often superficial engagement with ethical theories, with a tendency to acknowledge the importance of ethics, yet treating it as a monolithic and not contextualized concept. Our findings suggest a pressing need for a more nuanced and comprehensive integration of ethics in XAI research and practice. To support this, we propose to critically reconsider transparency and explainability in regards to ethical considerations during XAI systems design while accounting for ethical complexity in practice. As future research directions, we point to the promotion of interdisciplinary collaborations and education, also for underrepresented ethical perspectives. Such ethical grounding can guide the design of ethically robust XAI systems, aligning technical advancements with ethical considerations.
@article{nannini_mapping_2024,title={Mapping the landscape of ethical considerations in explainable {AI} research},volume={26},issn={1572-8439},url={https://doi.org/10.1007/s10676-024-09773-7},doi={10.1007/s10676-024-09773-7},language={en},number={3},urldate={2024-12-05},journal={Ethics and Information Technology},author={Nannini, Luca and Marchiori Manerba, Marta and Beretta, Isacco},month=jun,year={2024},keywords={AI ethics, Bibliometric study, Ethical analysis, Explainable AI (XAI)},pages={44},openaccess={true},line={5},}
Commodity-specific triads in the Dutch inter-industry production network
Marzio
Di Vece, Frank P.
Pijpers, and Diego
Garlaschelli
Triadic motifs are the smallest building blocks of higher-order interactions in complex networks and can be detected as over-occurrences with respect to null models with only pair-wise interactions. Recently, the motif structure of production networks has attracted attention in light of its possible role in the propagation of economic shocks. However, its characterization at the level of individual commodities is still poorly understood. Here we analyze both binary and weighted triadic motifs in the Dutch inter-industry production network disaggregated at the level of 187 commodity groups, which Statistics Netherlands reconstructed from National Accounts registers, surveys and known empirical data. We introduce appropriate null models that filter out node heterogeneity and the strong effects of link reciprocity and find that, while the aggregate network that overlays all products is characterized by a multitude of triadic motifs, most single-product layers feature no significant motif, and roughly 85% of the layers feature only two motifs or less. This result paves the way for identifying a simple ‘triadic fingerprint’ of each commodity and for reconstructing most product-specific networks from partial information in a pairwise fashion by controlling for their reciprocity structure. We discuss how these results can help statistical bureaus identify fine-grained information in structural analyses of interest for policymakers.
@article{di_vece_commodity-specific_2024,title={Commodity-specific triads in the {Dutch} inter-industry production network},volume={14},copyright={2024 The Author(s)},issn={2045-2322},url={https://www.nature.com/articles/s41598-024-53655-3},doi={10.1038/s41598-024-53655-3},language={en},number={1},urldate={2024-12-05},journal={Scientific Reports},author={Di Vece, Marzio and Pijpers, Frank P. and Garlaschelli, Diego},month=feb,year={2024},note={Publisher: Nature Publishing Group},keywords={Complex networks, Information theory and computation, Statistics},pages={3625},line={4},}
2023
Co-design of Human-centered, Explainable AI for Clinical Decision Support
Cecilia
Panigutti, Andrea
Beretta, Daniele
Fadda, Fosca
Giannotti, Dino
Pedreschi, and
2 more authors
eXplainable AI (XAI) involves two intertwined but separate challenges: the development of techniques to extract explanations from black-box AI models and the way such explanations are presented to users, i.e., the explanation user interface. Despite its importance, the second aspect has received limited attention so far in the literature. Effective AI explanation interfaces are fundamental for allowing human decision-makers to take advantage and oversee high-risk AI systems effectively. Following an iterative design approach, we present the first cycle of prototyping-testing-redesigning of an explainable AI technique and its explanation user interface for clinical Decision Support Systems (DSS). We first present an XAI technique that meets the technical requirements of the healthcare domain: sequential, ontology-linked patient data, and multi-label classification tasks. We demonstrate its applicability to explain a clinical DSS, and we design a first prototype of an explanation user interface. Next, we test such a prototype with healthcare providers and collect their feedback with a two-fold outcome: First, we obtain evidence that explanations increase users’ trust in the XAI system, and second, we obtain useful insights on the perceived deficiencies of their interaction with the system, so we can re-design a better, more human-centered explanation interface.
@article{10.1145/3587271,address={New York, NY, USA},articleno={21},author={Panigutti, Cecilia and Beretta, Andrea and Fadda, Daniele and Giannotti, Fosca and Pedreschi, Dino and Perotti, Alan and Rinzivillo, Salvatore},doi={10.1145/3587271},issn={2160-6455},issue_date={December 2023},journal={ACM Trans. Interact. Intell. Syst.},keywords={user study, human-computer interaction, clinical decision support systems, Explainable artificial intelligence},month=dec,number={4},numpages={35},publisher={Association for Computing Machinery},title={Co-design of Human-centered, Explainable AI for Clinical Decision Support},volume={13},year={2023},line={3},openaccess={true},}
The Importance of Time in Causal Algorithmic Recourse
Isacco
Beretta, and Martina
Cinquini
In World Conference on Explainable Artificial Intelligence , Dec 2023
The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inproceedings{beretta2023importance,author={Beretta, Isacco and Cinquini, Martina},booktitle={World Conference on Explainable Artificial Intelligence},line={2},month=dec,organization={Springer},pages={283--298},title={The Importance of Time in Causal Algorithmic Recourse},year={2023},doi={10.1007/978-3-031-44064-9_16}}
Benchmarking and survey of explanation methods for black box models
Francesco
Bodria, Fosca
Giannotti, Riccardo
Guidotti, Francesca
Naretto, Dino
Pedreschi, and
1 more author
The rise of sophisticated black-box machine learning models in Artificial Intelligence systems has prompted the need for explanation methods that reveal how these models work in an understandable way to users and decision makers. Unsurprisingly, the state-of-the-art exhibits currently a plethora of explainers providing many different types of explanations. With the aim of providing a compass for researchers and practitioners, this paper proposes a categorization of explanation methods from the perspective of the type of explanation they return, also considering the different input data formats. The paper accounts for the most representative explainers to date, also discussing similarities and discrepancies of returned explanations through their visual appearance. A companion website to the paper is provided as a continuous update to new explainers as they appear. Moreover, a subset of the most robust and widely adopted explainers, are benchmarked with respect to a repertoire of quantitative metrics.
@article{Bodria_2023,author={Bodria, Francesco and Giannotti, Fosca and Guidotti, Riccardo and Naretto, Francesca and Pedreschi, Dino and Rinzivillo, Salvatore},doi={10.1007/s10618-023-00933-9},issn={1573-756X},journal={Data Mining and Knowledge Discovery},line={1},month=jun,number={5},pages={1719–1778},publisher={Springer Science and Business Media LLC},title={Benchmarking and survey of explanation methods for black box models},volume={37},year={2023},openaccess={true},}
Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations
Eleonora
Cappuccio, Daniele
Fadda, Rosa
Lanzilotti, and Salvatore
Rinzivillo
In Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter , Torino, Italy, Jun 2023
The Human-Computer Interaction (HCI) community has long stressed the need for a more user-centered approach to Explainable Artificial Intelligence (XAI), a research area that aims at defining algorithms and tools to illustrate the predictions of the so-called black-box models. This approach can benefit from the fields of user-interface, user experience, and visual analytics. In this demo, we propose a visual-based tool, "F.I.P.E.R.", that shows interactive explanations combining rules and feature importance.
@inproceedings{10.1145/3605390.3610811,author={Cappuccio, Eleonora and Fadda, Daniele and Lanzilotti, Rosa and Rinzivillo, Salvatore},title={Demo: an Interactive Visualization Combining Rule-Based and Feature Importance Explanations},year={2023},isbn={9798400708060},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3605390.3610811},doi={10.1145/3605390.3610811},booktitle={Proceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter},articleno={52},numpages={4},keywords={User-interface, Explainable AI},location={Torino, Italy},series={CHItaly '23},line={3},}
Handling missing values in local post-hoc explainability
Martina
Cinquini, Fosca
Giannotti, Riccardo
Guidotti, and Andrea
Mattei
In World Conference on Explainable Artificial Intelligence , Oct 2023
Missing data are quite common in real scenarios when using Artificial Intelligence (AI) systems for decision-making with tabular data and effectively handling them poses a significant challenge for such systems. While some machine learning models used by AI systems can tackle this problem, the existing literature lacks post-hoc explainability approaches able to deal with predictors that encounter missing data. In this paper, we extend a widely used local model-agnostic post-hoc explanation approach that enables explainability in the presence of missing values by incorporating state-of-the-art imputation methods within the explanation process. Since our proposal returns explanations in the form of feature importance, the user will be aware also of the importance of a missing value in a given record for a particular prediction. Extensive experiments show the effectiveness of the proposed method with respect to some baseline solutions relying on traditional data imputation.
@inproceedings{cinquini2023handling,author={Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo and Mattei, Andrea},booktitle={World Conference on Explainable Artificial Intelligence},line={2},month=oct,organization={Springer},pages={256--278},title={Handling missing values in local post-hoc explainability},year={2023},}
EXPHLOT: EXplainable Privacy Assessment for Human LOcation Trajectories
Francesca
Naretto, Roberto
Pellungrini, Salvatore
Rinzivillo, and Daniele
Fadda
In Discovery Science - 26th International Conference, DS 2023, Porto,
Portugal, October 9-11, 2023, Proceedings , Oct 2023
Human mobility data play a crucial role in understanding mobility patterns and developing analytical services across various domains such as urban planning, transportation, and public health. However, due to the sensitive nature of this data, accurately identifying privacy risks is essential before deciding to release it to the public. Recent work has proposed the use of machine learning models for predicting privacy risk on raw mobility trajectories and the use of SHAP for risk explanation. However, applying SHAP to mobility data results in explanations that are of limited use both for privacy experts and end-users. In this work, we present a novel version of the EXPERT privacy risk prediction and explanation framework specifically tailored for human mobility data. We leverage state-of-the-art algorithms in time series classification, as ROCKET and INCEPTIONTIME, to improve risk prediction while reducing computation time. Additionally, we address two key issues with SHAP explanation on mobility data: first, we devise an entropy-based mask to efficiently compute SHAP values for privacy risk in mobility data; second, we develop a module for interactive analysis and visualization of SHAP values over a map, empowering users with an intuitive understanding of SHAP values and privacy risk.
@inproceedings{DBLP:conf/dis/NarettoPRF23,author={Naretto, Francesca and Pellungrini, Roberto and Rinzivillo, Salvatore and Fadda, Daniele},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/conf/dis/NarettoPRF23.bib},booktitle={Discovery Science - 26th International Conference, {DS} 2023, Porto,
Portugal, October 9-11, 2023, Proceedings},doi={10.1007/978-3-031-45275-8\_22},editor={Bifet, Albert and Lorena, Ana Carolina and Ribeiro, Rita P. and Gama, Jo{\~{a}}o and Abreu, Pedro H.},line={3,5},month=oct,pages={325--340},publisher={Springer},series={Lecture Notes in Computer Science},timestamp={Wed, 01 Nov 2023 08:59:01 +0100},title={{EXPHLOT:} EXplainable Privacy Assessment for Human LOcation Trajectories},volume={14276},year={2023},}
Declarative Reasoning on Explanations Using Constraint Logic Programming
Laura
State, Salvatore
Ruggieri, and Franco
Turini
Explaining opaque Machine Learning (ML) models is an increasingly relevant problem. Current explanation in AI (XAI) methods suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of abstraction and interactivity with the user. We propose REASONX, an explanation method based on Constraint Logic Programming (CLP). REASONX can provide declarative, interactive explanations for decision trees, which can be the ML models under analysis or global/local surrogate models of any black-box model. Users can express background or common sense knowledge using linear constraints and MILP optimization over features of factual and contrastive instances, and interact with the answer constraints at different levels of abstraction through constraint projection. We present here the architecture of REASONX, which consists of a Python layer, closer to the user, and a CLP layer. REASONX’s core execution engine is a Prolog meta-program with declarative semantics in terms of logic theories.
@article{DBLP:journals/corr/abs-2309-00422,author={State, Laura and Ruggieri, Salvatore and Turini, Franco},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/journals/corr/abs-2309-00422.bib},doi={10.48550/ARXIV.2309.00422},eprint={2309.00422},eprinttype={arXiv},journal={CoRR},line={2},month=sep,timestamp={Mon, 11 Sep 2023 16:01:35 +0200},title={Declarative Reasoning on Explanations Using Constraint Logic Programming},volume={abs/2309.00422},year={2023},}
Modeling Events and Interactions through Temporal Processes – A Survey
Angelica
Liguori, Luciano
Caroprese, Marco
Minici, Bruno
Veloso, Francesco
Spinnato, and
3 more authors
In real-world scenario, many phenomena produce a collection of events that occur in continuous time. Point Processes provide a natural mathematical framework for modeling these sequences of events. In this survey, we investigate probabilistic models for modeling event sequences through temporal processes. We revise the notion of event modeling and provide the mathematical foundations that characterize the literature on the topic. We define an ontology to categorize the existing approaches in terms of three families: simple, marked, and spatio-temporal point processes. For each family, we systematically review the existing approaches based based on deep learning. Finally, we analyze the scenarios where the proposed techniques can be used for addressing prediction and modeling aspects.
@misc{https://doi.org/10.48550/arxiv.2303.06067,author={Liguori, Angelica and Caroprese, Luciano and Minici, Marco and Veloso, Bruno and Spinnato, Francesco and Nanni, Mirco and Manco, Giuseppe and Gama, Joao},copyright={arXiv.org perpetual, non-exclusive license},doi={10.48550/ARXIV.2303.06067},keywords={Machine Learning (cs.LG), Machine Learning (stat.ML), FOS: Computer and information sciences, FOS: Computer and information sciences},line={1},month=mar,publisher={arXiv},title={Modeling Events and Interactions through Temporal Processes -- A Survey},year={2023},}
Geolet: An Interpretable Model for Trajectory Classification
Cristiano
Landi, Francesco
Spinnato, Riccardo
Guidotti, Anna
Monreale, and Mirco
Nanni
The large and diverse availability of mobility data enables the development of predictive models capable of recognizing various types of movements. Through a variety of GPS devices, any moving entity, animal, person, or vehicle can generate spatio-temporal trajectories. This data is used to infer migration patterns, manage traffic in large cities, and monitor the spread and impact of diseases, all critical situations that necessitate a thorough understanding of the underlying problem. Researchers, businesses, and governments use mobility data to make decisions that affect people’s lives in many ways, employing accurate but opaque deep learning models that are difficult to interpret from a human standpoint. To address these limitations, we propose Geolet, a human-interpretable machine-learning model for trajectory classification. We use discriminative sub-trajectories extracted from mobility data to turn trajectories into a simplified representation that can be used as input by any machine learning classifier. We test our approach against state-of-the-art competitors on real-world datasets. Geolet outperforms black-box models in terms of accuracy while being orders of magnitude faster than its interpretable competitors.
@inbook{Landi_2023,author={Landi, Cristiano and Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-031-30047-9_19},isbn={9783031300479},issn={1611-3349},line={1},month=apr,pages={236–248},publisher={Springer Nature Switzerland},title={Geolet: An Interpretable Model for Trajectory Classification},year={2023},openaccess={false},}
Exposing Racial Dialect Bias in Abusive Language Detection: Can Explainability Play a Role?
Biases can arise and be introduced during each phase of a supervised learning pipeline, eventually leading to harm. Within the task of automatic abusive language detection, this matter becomes particularly severe since unintended bias towards sensitive topics such as gender, sexual orientation, or ethnicity can harm underrepresented groups. The role of the datasets used to train these models is crucial to address these challenges. In this contribution, we investigate whether explainability methods can expose racial dialect bias attested within a popular dataset for abusive language detection. Through preliminary experiments, we found that pure explainability techniques cannot effectively uncover biases within the dataset under analysis: the rooted stereotypes are often more implicit and complex to retrieve.
@inbook{Manerba_2023,author={Manerba, Marta Marchiori and Morini, Virginia},booktitle={Machine Learning and Principles and Practice of Knowledge Discovery in Databases},doi={10.1007/978-3-031-23618-1_32},isbn={9783031236181},issn={1865-0937},line={5},month=jan,pages={483–497},publisher={Springer Nature Switzerland},title={Exposing Racial Dialect Bias in Abusive Language Detection: Can Explainability Play a Role?},year={2023},}
Differentiable Causal Discovery with Smooth Acyclic Orientations
Riccardo
Massidda, Francesco
Landolfi, Martina
Cinquini, and Davide
Bacciu
In ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators , Jun 2023
Most differentiable causal discovery approaches constrain or regularize an optimization problem using a continuous relaxation of the acyclicity property. The cost of computing the relaxation is cubic on the number of nodes and thus affects the scalability of such techniques. In this work, we introduce COSMO, the first quadratic and constraint-free continuous optimization scheme. COSMO represents a directed acyclic graph as a priority vector on the nodes and an adjacency matrix. We prove that the priority vector represents a differentiable approximation of the acyclic orientation of the graph, and we demonstrate the existence of an upper bound on the orientation acyclicity. In addition to being asymptotically faster, our empirical analysis highlights how COSMO performs comparably to constrained methods for graph discovery.
@inproceedings{massidda2023differentiable,author={Massidda, Riccardo and Landolfi, Francesco and Cinquini, Martina and Bacciu, Davide},booktitle={ICML 2023 Workshop on Differentiable Almost Everything: Differentiable Relaxations, Algorithms, Operators, and Simulators},line={2},month=jun,title={Differentiable Causal Discovery with Smooth Acyclic Orientations},year={2023},}
Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning
Carlo
Metta, Andrea
Beretta, Riccardo
Guidotti, Yuan
Yin, Patrick
Gallinari, and
2 more authors
International Journal of Data Science and Analytics, Jun 2023
A key issue in critical contexts such as medical diagnosis is the interpretability of the deep learning models adopted in decision-making systems. Research in eXplainable Artificial Intelligence (XAI) is trying to solve this issue. However, often XAI approaches are only tested on generalist classifier and do not represent realistic problems such as those of medical diagnosis. In this paper, we aim at improving the trust and confidence of users towards automatic AI decision systems in the field of medical skin lesion diagnosis by customizing an existing XAI approach for explaining an AI model able to recognize different types of skin lesions. The explanation is generated through the use of synthetic exemplar and counter-exemplar images of skin lesions and our contribution offers the practitioner a way to highlight the crucial traits responsible for the classification decision. A validation survey with domain experts, beginners, and unskilled people shows that the use of explanations improves trust and confidence in the automatic decision system. Also, an analysis of the latent space adopted by the explainer unveils that some of the most frequent skin lesion classes are distinctly separated. This phenomenon may stem from the intrinsic characteristics of each class and may help resolve common misclassifications made by human experts.
@article{Metta_2023,author={Metta, Carlo and Beretta, Andrea and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore and Giannotti, Fosca},doi={10.1007/s41060-023-00401-z},issn={2364-4168},journal={International Journal of Data Science and Analytics},line={1,3,4},month=jun,publisher={Springer Science and Business Media LLC},title={Improving trust and confidence in medical skin lesion diagnosis through explainable deep learning},year={2023},openaccess={true},}
AUC-based Selective Classification
Andrea
Pugnana, and Salvatore
Ruggieri
In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics , 25–27 apr 2023
Selective classification (or classification with a reject option) pairs a classifier with a selection function to determine whether or not a prediction should be accepted. This framework trades off coverage (probability of accepting a prediction) with predictive performance, typically measured by distributive loss functions. In many application scenarios, such as credit scoring, performance is instead measured by ranking metrics, such as the Area Under the ROC Curve (AUC). We propose a model-agnostic approach to associate a selection function to a given probabilistic binary classifier. The approach is specifically targeted at optimizing the AUC. We provide both theoretical justifications and a novel algorithm, called AUCROSS, to achieve such a goal. Experiments show that our method succeeds in trading-off coverage for AUC, improving over existing selective classification methods targeted at optimizing accuracy.
@inproceedings{pmlr-v206-pugnana23a,author={Pugnana, Andrea and Ruggieri, Salvatore},booktitle={Proceedings of The 26th International Conference on Artificial Intelligence and Statistics},editor={Ruiz, Francisco and Dy, Jennifer and van de Meent, Jan-Willem},line={1,2},month={25--27 Apr},pages={2494--2514},publisher={PMLR},series={Proceedings of Machine Learning Research},title={AUC-based Selective Classification},volume={206},year={2023}}
Text to Time Series Representations: Towards Interpretable Predictive Models
Mattia
Poggioli, Francesco
Spinnato, and Riccardo
Guidotti
Time Series Analysis (TSA) and Natural Language Processing (NLP) are two domains of research that have seen a surge of interest in recent years. NLP focuses mainly on enabling computers to manipulate and generate human language, whereas TSA identifies patterns or components in time-dependent data. Given their different purposes, there has been limited exploration of combining them. In this study, we present an approach to convert text into time series to exploit TSA for exploring text properties and to make NLP approaches interpretable for humans. We formalize our Text to Time Series framework as a feature extraction and aggregation process, proposing a set of different conversion alternatives for each step. We experiment with our approach on several textual datasets, showing the conversion approach’s performance and applying it to the field of interpretable time series classification.
@inbook{Poggioli_2023,author={Poggioli, Mattia and Spinnato, Francesco and Guidotti, Riccardo},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-031-45275-8_16},isbn={9783031452758},issn={1611-3349},line={1},month=oct,pages={230–245},publisher={Springer Nature Switzerland},title={Text to Time Series Representations: Towards Interpretable Predictive Models},year={2023},openaccess={false},}
Topics in Selective Classification
Andrea
Pugnana
Proceedings of the AAAI Conference on Artificial Intelligence, Sep 2023
In recent decades, advancements in information technology allowed Artificial Intelligence (AI) systems to predict future outcomes with unprecedented success. This brought the widespread deployment of these methods in many fields, intending to support decision-making. A pressing question is how to make AI systems robust to common challenges in real-life scenarios and trustworthy. In my work, I plan to explore ways to enhance the trustworthiness of AI through the selective classification framework. In this setting, the AI system can refrain from predicting whenever it is not confident enough, allowing it to trade off coverage, i.e. the percentage of instances that receive a prediction, for performance.
@article{Pugnana_2023,author={Pugnana, Andrea},doi={10.1609/aaai.v37i13.26925},journal={Proceedings of the AAAI Conference on Artificial Intelligence},line={1,2},month=sep,number={13},pages={16129-16130},title={Topics in Selective Classification},volume={37},year={2023}}
Understanding Any Time Series Classifier with a Subsequence-based Explainer
Francesco
Spinnato, Riccardo
Guidotti, Anna
Monreale, Mirco
Nanni, Dino
Pedreschi, and
1 more author
ACM Transactions on Knowledge Discovery from Data, Nov 2023
The growing availability of time series data has increased the usage of classifiers for this data type. Unfortunately, state-of-the-art time series classifiers are black-box models and, therefore, not usable in critical domains such as healthcare or finance, where explainability can be a crucial requirement. This paper presents a framework to explain the predictions of any black-box classifier for univariate and multivariate time series. The provided explanation is composed of three parts. First, a saliency map highlighting the most important parts of the time series for the classification. Second, an instance-based explanation exemplifies the black-box’s decision by providing a set of prototypical and counterfactual time series. Third, a factual and counterfactual rule-based explanation, revealing the reasons for the classification through logical conditions based on subsequences that must, or must not, be contained in the time series. Experiments and benchmarks show that the proposed method provides faithful, meaningful, stable, and interpretable explanations.
@article{Spinnato_2023,author={Spinnato, Francesco and Guidotti, Riccardo and Monreale, Anna and Nanni, Mirco and Pedreschi, Dino and Giannotti, Fosca},doi={10.1145/3624480},issn={1556-472X},journal={ACM Transactions on Knowledge Discovery from Data},line={1},month=nov,number={2},pages={1–34},publisher={Association for Computing Machinery (ACM)},title={Understanding Any Time Series Classifier with a Subsequence-based Explainer},volume={18},year={2023},openaccess={true}}
Reason to explain: Interactive contrastive explanations (REASONX)
Laura
State, Salvatore
Ruggieri, and Franco
Turini
Many high-performing machine learning models are not interpretable. As they are increasingly used in decision scenarios that can critically affect individuals, it is necessary to develop tools to better understand their outputs. Popular explanation methods include contrastive explanations. However, they suffer several shortcomings, among others an insufficient incorporation of background knowledge, and a lack of interactivity. While (dialogue-like) interactivity is important to better communicate an explanation, background knowledge has the potential to significantly improve their quality, e.g., by adapting the explanation to the needs of the end-user.
To close this gap, we present REASONX, an explanation tool based on Constraint Logic Programming (CLP). REASONX provides interactive contrastive explanations that can be augmented by background knowledge, and allows to operate under a setting of under-specified information, leading to increased flexibility in the provided explanations. REASONX computes factual and contrastive decision rules, as well as closest contrastive examples. It provides explanations for decision trees, which can be the ML models under analysis, or global/local surrogate models of any ML model.
While the core part of REASONX is built on CLP, we also provide a program layer that allows to compute the explanations via Python, making the tool accessible to a wider audience. We illustrate the capability of REASONX on a synthetic data set, and on a well-developed example in the credit domain. In both cases, we can show how REASONX can be flexibly used and tailored to the needs of the user.
@misc{state2023reason,archiveprefix={arXiv},author={State, Laura and Ruggieri, Salvatore and Turini, Franco},eprint={2305.18143},line={2,3},month=dec,primaryclass={cs.AI},title={Reason to explain: Interactive contrastive explanations (REASONX)},year={2023},}
Effects of Route Randomization on Urban Emissions
Giuliano
Cornacchia, Mirco
Nanni, Dino
Pedreschi, and Luca
Pappalardo
Routing algorithms typically suggest the fastest path or slight variation to reach a user’s desired destination. Although this suggestion at the individual level is undoubtedly advantageous for the user, from a collective point of view, the aggregation of all single suggested paths may result in an increasing impact (e.g., in terms of emissions). In this study, we use SUMO to simulate the effects of incorporating randomness into routing algorithms on emissions, their distribution, and travel time in the urban area of Milan (Italy). Our results reveal that, given the common practice of routing towards the fastest path, a certain level of randomness in routes reduces emissions and travel time. In other words, the stronger the random component in the routes, the more pronounced the benefits upon a certain threshold. Our research provides insight into the potential advantages of considering collective outcomes in routing decisions and highlights the need to explore further the relationship between route randomization and sustainability in urban transportation.
@article{Cornacchia_2023,title={Effects of Route Randomization on Urban Emissions},volume={4},issn={2750-4425},url={http://dx.doi.org/10.52825/scp.v4i.217},doi={10.52825/scp.v4i.217},journal={SUMO Conference Proceedings},publisher={TIB Open Publishing},author={Cornacchia, Giuliano and Nanni, Mirco and Pedreschi, Dino and Pappalardo, Luca},year={2023},month=jun,pages={75--87},line={4},}
Semantic Enrichment of Explanations of AI Models for Healthcare
Luca
Corbucci, Anna
Monreale, Cecilia
Panigutti, Michela
Natilli, Simona
Smiraglio, and
1 more author
Explaining AI-based clinical decision support systems is crucial to enhancing clinician trust in those powerful systems. Unfortunately, current explanations provided by eXplainable Artificial Intelligence techniques are not easily understandable by experts outside of AI. As a consequence, the enrichment of explanations with relevant clinical information concerning the health status of a patient is fundamental to increasing human experts’ ability to assess the reliability of AI decisions. Therefore, in this paper, we propose a methodology to enable clinical reasoning by semantically enriching AI explanations. Starting with a medical AI explanation based only on the input features provided to the algorithm, our methodology leverages medical ontologies and NLP embedding techniques to link relevant information present in the patient’s clinical notes to the original explanation. Our experiments, involving a human expert, highlight promising performance in correctly identifying relevant information about the diseases of the patients.
@inbook{Corbucci_2023b,title={Semantic Enrichment of Explanations of AI Models for Healthcare},isbn={9783031452758},issn={1611-3349},doi={10.1007/978-3-031-45275-8_15},booktitle={Discovery Science},publisher={Springer Nature Switzerland},author={Corbucci, Luca and Monreale, Anna and Panigutti, Cecilia and Natilli, Michela and Smiraglio, Simona and Pedreschi, Dino},year={2023},pages={216--229},month=oct,}
Interpretable Data Partitioning Through Tree-Based Clustering Methods
Riccardo
Guidotti, Cristiano
Landi, Andrea
Beretta, Daniele
Fadda, and Mirco
Nanni
The growing interpretable machine learning research field is mainly focusing on the explanation of supervised approaches. However, also unsupervised approaches might benefit from considering interpretability aspects. While existing clustering methods only provide the assignment of records to clusters without justifying the partitioning, we propose tree-based clustering methods that offer interpretable data partitioning through a shallow decision tree. These decision trees enable easy-to-understand explanations of cluster assignments through short and understandable split conditions. The proposed methods are evaluated through experiments on synthetic and real datasets and proved to be more effective than traditional clustering approaches and interpretable ones in terms of standard evaluation measures and runtime. Finally, a case study involving human participation demonstrates the effectiveness of the interpretable clustering trees returned by the proposed method.
@inbook{Guidotti_2023,title={Interpretable Data Partitioning Through Tree-Based Clustering Methods},isbn={9783031452758},issn={1611-3349},doi={10.1007/978-3-031-45275-8_33},booktitle={Discovery Science},publisher={Springer Nature Switzerland},author={Guidotti, Riccardo and Landi, Cristiano and Beretta, Andrea and Fadda, Daniele and Nanni, Mirco},year={2023},pages={492–507},line={1,3},}
Explaining Black-Boxes in Federated Learning
Luca
Corbucci, Riccardo
Guidotti, and Anna
Monreale
Federated Learning has witnessed increasing popularity in the past few years for its ability to train Machine Learning models in critical contexts, using private data without moving them. Most of the work in the literature proposes algorithms and architectures for training neural networks, which although they present high performance in different predicting tasks and are easy to be learned with a cooperative mechanism, their predictive reasoning is obscure. Therefore, in this paper, we propose a variant of SHAP, one of the most widely used explanation methods, tailored to Horizontal server-based Federated Learning. The basic idea is having the possibility to explain an instance’s prediction performed by the trained Machine Leaning model as an aggregation of the explanations provided by the clients participating in the cooperation. We empirically test our proposal on two different tabular datasets, and we observe interesting and encouraging preliminary results.
@inbook{Corbucci_2023,title={Explaining Black-Boxes in Federated Learning},isbn={9783031440670},issn={1865-0937},doi={10.1007/978-3-031-44067-0_8},booktitle={Explainable Artificial Intelligence},publisher={Springer Nature Switzerland},author={Corbucci, Luca and Guidotti, Riccardo and Monreale, Anna},year={2023},pages={151--163},line={1},month=oct,}
Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence
Sajid
Ali, Tamer
Abuhmed, Shaker
El-Sappagh, Khan
Muhammad, Jose M.
Alonso-Moral, and
5 more authors
Artificial intelligence (AI) is currently being utilized in a wide range of sophisticated applications, but the outcomes of many AI models are challenging to comprehend and trust due to their black-box nature. Usually, it is essential to understand the reasoning behind an AI model’s decision-making. Thus, the need for eXplainable AI (XAI) methods for improving trust in AI models has arisen. XAI has become a popular research subject within the AI field in recent years. Existing survey papers have tackled the concepts of XAI, its general terms, and post-hoc explainability methods but there have not been any reviews that have looked at the assessment methods, available tools, XAI datasets, and other related aspects. Therefore, in this comprehensive study, we provide readers with an overview of the current research and trends in this rapidly emerging area with a case study example. The study starts by explaining the background of XAI, common definitions, and summarizing recently proposed techniques in XAI for supervised machine learning. The review divides XAI techniques into four axes using a hierarchical categorization system: (i) data explainability, (ii) model explainability, (iii) post-hoc explainability, and (iv) assessment of explanations. We also introduce available evaluation metrics as well as open-source packages and datasets with future research directions. Then, the significance of explainability in terms of legal demands, user viewpoints, and application orientation is outlined, termed as XAI concerns. This paper advocates for tailoring explanation content to specific user types. An examination of XAI techniques and evaluation was conducted by looking at 410 critical articles, published between January 2016 and October 2022, in reputed journals and using a wide range of research databases as a source of information. The article is aimed at XAI researchers who are interested in making their AI models more trustworthy, as well as towards researchers from other disciplines who are looking for effective XAI methods to complete tasks with confidence while communicating meaning from data.
@article{ALI2023101805,title={Explainable Artificial Intelligence (XAI): What we know and what is left to attain Trustworthy Artificial Intelligence},journal={Information Fusion},volume={99},pages={101805},year={2023},issn={1566-2535},doi={10.1016/j.inffus.2023.101805},author={Ali, Sajid and Abuhmed, Tamer and El-Sappagh, Shaker and Muhammad, Khan and Alonso-Moral, Jose M. and Confalonieri, Roberto and Guidotti, Riccardo and {Del Ser}, Javier and Díaz-Rodríguez, Natalia and Herrera, Francisco},keywords={Explainable Artificial Intelligence, Interpretable machine learning, Trustworthy AI, AI principles, Post-hoc explainability, XAI assessment, Data Fusion, Deep Learning},line={1}}
A Protocol for Continual Explanation of SHAP
Andrea
Cossu, Francesco
Spinnato, Riccardo
Guidotti, and Davide
Bacciu
Continual Learning trains models on a stream of data, with the aim of learning new information without forgetting previous knowledge. Given the dynamic nature of such environments, explaining the predictions of these models can be challenging. We study the behavior of SHAP values explanations in Continual Learning and propose an evaluation protocol to robustly assess the change of explanations in Class-Incremental scenarios. We observed that, while Replay strategies enforce the stability of SHAP values in feedforward/convolutional models, they are not able to do the same with fully-trained recurrent models. We show that alternative recurrent approaches, like randomized recurrent models, are more effective in keeping the explanations stable over time.
@misc{cossu2023protocolcontinualexplanationshap,title={A Protocol for Continual Explanation of SHAP},author={Cossu, Andrea and Spinnato, Francesco and Guidotti, Riccardo and Bacciu, Davide},year={2023},month=jul,eprint={2306.07218},archiveprefix={arXiv},primaryclass={cs.LG},line={1}}
Trustworthy AI at KDD Lab
Fosca
Giannotti, Riccardo
Guidotti, Anna
Monreale, Luca
Pappalardo, Dino
Pedreschi, and
6 more authors
In Proceedings of the Italia Intelligenza Artificiale - Thematic Workshops
co-located with the 3rd CINI National Lab AIIS Conference on Artificial
Intelligence (Ital IA 2023), Pisa, Italy, May 29-30, 2023 , Jul 2023
Pandemic vaccination campaigns must account for vaccine skepticism as an obstacle to overcome. Using machine learning to identify behavioral and psychological patterns in public survey datasets can provide valuable insights and inform vaccination campaigns based on empirical evidence. However, we argue that the adoption of local and global explanation methodologies can provide additional support to health practitioners by suggesting personalized communication strategies and revealing potential demographic, social, or structural barriers to vaccination requiring systemic changes. In this paper, we first implement a chain classification model for the adoption of the vaccine during the H1N1 influenza outbreak taking seasonal vaccination information into account, and then compare it with a binary classifier for vaccination to better understand the overall patterns in the data. Following that, we derive and compare global explanations using post-hoc methodologies and interpretable-by-design models. Our findings indicate that socio-demographic factors play a distinct role in the H1N1 vaccination as compared to the general vaccination. Nevertheless, medical recommendation and health insurance remain significant factors for both vaccinations. Then, we concentrated on the subpopulation of individuals who did not receive an H1N1 vaccination despite being at risk of developing severe symptoms. In an effort to assist practitioners in providing effective recommendations to patients, we present rules and counterfactuals for the selected instances based on local explanations. Finally, we raise concerns regarding gender and racial disparities in healthcare access by analysing the interaction effects of sensitive attributes on the model’s output.
@inproceedings{10.1007/978-3-031-44067-0_31,author={Punzi, Clara and Maslennikova, Aleksandra and Gezici, Gizem and Pellungrini, Roberto and Giannotti, Fosca},editor={Longo, Luca},title={Explaining Socio-Demographic and Behavioral Patterns of Vaccination Against the Swine Flu (H1N1) Pandemic},booktitle={Explainable Artificial Intelligence},year={2023},month=oct,publisher={Springer Nature Switzerland},address={Cham},pages={621--635},isbn={978-3-031-44067-0},openaccess={true},line={4},}
The Ethical Impact Assessment of Selling Life Insurance to Titanic Passengers
The Artificial Intelligence Act (AIA) is a uniform legal framework to ensure that AI systems within the European Union (EU) are safe and comply with existing law on fundamental rights and constitutional values. The AIA adopts a risk-based approach with the aim of intending to regulate AI systems, especially categorised as high-risk, which have significant harmful impacts on the health, safety and fundamental rights of persons in the Union. The AIA is founded on the Ethics Guidelines of the High-Level Expert Group for Trustworthy AI, which are grounded in fundamental rights and reflect four ethical imperatives in order to ensure ethical and robust AI. While we acknowledge that ethics is not law, we advocate that the analysis of ethical risks can assist us in complying with laws, thereby facilitating the implementation of the AIA requirements. Thus, we first design an AI-driven Decision Support System for individual risk prediction in the insurance domain (categorised as high-risk by the AIA) based on the Titanic case, which is a popular benchmark dataset in machine learning. We then fulfill an ethical impact assessment of the Titanic case study, relying on the four ethical imperatives of respect for human autonomy, prevention of harm, fairness, and explicability, declared by the High-Level Expert Group for Trustworthy AI. In the context of this ethical impact assessment, we also refer to the questions in the ALTAI checklist. Our discussions regarding the ethical impact assessment in the insurance domain demonstrate that ethical principles can intersect but also create tensions (intriguingly, only in this particular context), for which there is no definitive solution. When tensions arise, which may result in unavoidable trade-offs, these trade-offs should be addressed in a rational and methodical manner, paying special attention to the context of the current case study being evaluated
@inproceedings{Gezici2023TheEI,title={The Ethical Impact Assessment of Selling Life Insurance to Titanic Passengers},author={Gezici, Gizem and Mannari, Chiara and Orlandi, Lorenzo},booktitle={HHAI Workshops},year={2023},url={https://api.semanticscholar.org/CorpusID:260971654},line={4},}
The Importance of Time in Causal Algorithmic Recourse
The application of Algorithmic Recourse in decision-making is a promising field that offers practical solutions to reverse unfavorable decisions. However, the inability of these methods to consider potential dependencies among variables poses a significant challenge due to the assumption of feature independence. Recent advancements have incorporated knowledge of causal dependencies, thereby enhancing the quality of the recommended recourse actions. Despite these improvements, the inability to incorporate the temporal dimension remains a significant limitation of these approaches. This is particularly problematic as identifying and addressing the root causes of undesired outcomes requires understanding time-dependent relationships between variables. In this work, we motivate the need to integrate the temporal dimension into causal algorithmic recourse methods to enhance recommendations’ plausibility and reliability. The experimental evaluation highlights the significance of the role of time in this field.
@inproceedings{beretta_importance_2023,address={Cham},title={The {Importance} of {Time} in {Causal} {Algorithmic} {Recourse}},isbn={978-3-031-44064-9},doi={10.1007/978-3-031-44064-9_16},language={en},booktitle={Explainable {Artificial} {Intelligence}},publisher={Springer Nature Switzerland},author={Beretta, Isacco and Cinquini, Martina},editor={Longo, Luca},year={2023},keywords={Algorithmic Recourse, Causality, Consequential Recommendations},pages={283--298},line={2},}
2022
Explaining Black Box with Visual Exploration of Latent Space
Francesco
Bodria, Salvatore
Rinzivillo, Daniele
Fadda, Riccardo
Guidotti, Fosca
Giannotti, and
1 more author
Autoencoders are a powerful yet opaque feature reduction technique, on top of which we propose a novel way for the joint visual exploration of both latent and real space. By interactively exploiting the mapping between latent and real features, it is possible to unveil the meaning of latent features while providing deeper insight into the original variables. To achieve this goal, we exploit and re-adapt existing approaches from eXplainable Artificial Intelligence (XAI) to understand the relationships between the input and latent features. The uncovered relationships between input features and latent ones allow the user to understand the data structure concerning external variables such as the predictions of a classification model. We developed an interactive framework that visually explores the latent space and allows the user to understand the relationships of the input features with model prediction.
@inproceedings{10.2312:evs.20221098,author={Bodria, Francesco and Rinzivillo, Salvatore and Fadda, Daniele and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino},booktitle={EuroVis 2022 - Short Papers},doi={10.2312/evs.20221098},editor={Agus, Marco and Aigner, Wolfgang and Hoellt, Thomas},isbn={978-3-03868-184-7},publisher={The Eurographics Association},title={{Explaining Black Box with Visual Exploration of Latent Space}},year={2022},line={3}}
Interpretable Latent Space to Enable Counterfactual Explanations
Francesco
Bodria, Riccardo
Guidotti, Fosca
Giannotti, and Dino
Pedreschi
Many dimensionality reduction methods have been introduced to map a data space into one with fewer features and enhance machine learning models’ capabilities. This reduced space, called latent space, holds properties that allow researchers to understand the data better and produce better models. This work proposes an interpretable latent space that preserves the similarity of data points and supports a new way of learning a classification model that allows prediction and explanation through counterfactual examples. We demonstrate with extensive experiments the effectiveness of the latent space with respect to different metrics in comparison with several competitors, as well as the quality of the achieved counterfactual explanations.
@inbook{Bodria_2022_Interpretable,author={Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-031-18840-4_37},isbn={9783031188404},issn={1611-3349},line={1},month=dec,pages={525–540},publisher={Springer Nature Switzerland},title={Interpretable Latent Space to Enable Counterfactual Explanations},year={2022}}
Transparent Latent Space Counterfactual Explanations for Tabular Data
Francesco
Bodria, Riccardo
Guidotti, Fosca
Giannotti, and Dino
Pedreschi
In 2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA) , Oct 2022
Artificial Intelligence decision-making systems have dramatically increased their predictive performance in recent years, beating humans in many different specific tasks. However, with increased performance has come an increase in the complexity of the black-box models adopted by the AI systems, making them entirely obscure for the decision process adopted. Explainable AI is a field that seeks to make AI decisions more transparent by producing explanations. In this paper, we propose T-LACE, an approach able to retrieve post-hoc counterfactual explanations for a given pre-trained black-box model. T-LACE exploits the similarity and linearity proprieties of a custom-created transparent latent space to build reliable counterfactual explanations. We tested T-LACE on several tabular datasets and provided qualitative evaluations of the generated explanations in terms of similarity, robustness, and diversity. Comparative analysis against various state-of-the-art counterfactual explanation methods shows the higher effectiveness of our approach.
@inproceedings{Bodria_2022_Transparent,author={Bodria, Francesco and Guidotti, Riccardo and Giannotti, Fosca and Pedreschi, Dino},booktitle={2022 IEEE 9th International Conference on Data Science and Advanced Analytics (DSAA)},doi={10.1109/dsaa54385.2022.10032407},line={1},month=oct,publisher={IEEE},title={Transparent Latent Space Counterfactual Explanations for Tabular Data},year={2022}}
Explaining Siamese Networks in Few-Shot Learning for Audio Data
Andrea
Fedele, Riccardo
Guidotti, and Dino
Pedreschi
In Discovery Science - 25th International Conference, DS 2022, Montpellier,
France, October 10-12, 2022, Proceedings , Dec 2022
Machine learning models are not able to generalize correctly when queried on samples belonging to class distributions that were never seen during training. This is a critical issue, since real world applications might need to quickly adapt without the necessity of re-training. To overcome these limitations, few-shot learning frameworks have been proposed and their applicability has been studied widely for computer vision tasks. Siamese Networks learn pairs similarity in form of a metric that can be easily extended on new unseen classes. Unfortunately, the downside of such systems is the lack of explainability. We propose a method to explain the outcomes of Siamese Networks in the context of few-shot learning for audio data. This objective is pursued through a local perturbation-based approach that evaluates segments-weighted-average contributions to the final outcome considering the interplay between different areas of the audio spectrogram. Qualitative and quantitative results demonstrate that our method is able to show common intra-class characteristics and erroneous reliance on silent sections.
@inproceedings{DBLP:conf/dis/FedeleGP22,author={Fedele, Andrea and Guidotti, Riccardo and Pedreschi, Dino},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/conf/dis/FedeleGP22.bib},booktitle={Discovery Science - 25th International Conference, {DS} 2022, Montpellier,
France, October 10-12, 2022, Proceedings},doi={10.1007/978-3-031-18840-4\_36},editor={Poncelet, Pascal and Ienco, Dino},line={1},month=dec,pages={509--524},publisher={Springer},series={Lecture Notes in Computer Science},timestamp={Sat, 30 Sep 2023 09:38:59 +0200},title={Explaining Siamese Networks in Few-Shot Learning for Audio Data},volume={13601},year={2022}}
Explainable AI for Time Series Classification: A Review, Taxonomy
and Research Directions
Andreas
Theissler, Francesco
Spinnato, Udo
Schlegel, and Riccardo
Guidotti
Time series classification (TSC) is a challenging task in machine learning
and data mining. The increasing complexity of models used for TSC
has led to a growing demand for explainable artificial intelligence
(XAI) methods. This paper provides a comprehensive review of the
state-of-the-art in XAI for TSC. We present a taxonomy of XAI methods
for TSC and discuss their strengths and weaknesses. We also identify
research directions for future work in this area.
@article{DBLP:journals/access/TheisslerSSG22,author={Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo},bibsource={dblp computer science bibliography, https://dblp.org},biburl={https://dblp.org/rec/journals/access/TheisslerSSG22.bib},doi={10.1109/ACCESS.2022.3207765},journal={{IEEE} Access},line={1},month=dec,pages={100700--100724},timestamp={Tue, 18 Oct 2022 22:17:12 +0200},title={Explainable {AI} for Time Series Classification: {A} Review, Taxonomy
and Research Directions},volume={10},year={2022}}
Stable and actionable explanations of black-box models through factual and counterfactual rules
Riccardo
Guidotti, Anna
Monreale, Salvatore
Ruggieri, Francesca
Naretto, Franco
Turini, and
2 more authors
Recent years have witnessed the rise of accurate but obscure classification models that hide the logic of their internal decision processes. Explaining the decision taken by a black-box classifier on a specific input instance is therefore of striking interest. We propose a local rule-based model-agnostic explanation method providing stable and actionable explanations. An explanation consists of a factual logic rule, stating the reasons for the black-box decision, and a set of actionable counterfactual logic rules, proactively suggesting the changes in the instance that lead to a different outcome. Explanations are computed from a decision tree that mimics the behavior of the black-box locally to the instance to explain. The decision tree is obtained through a bagging-like approach that favors stability and fidelity: first, an ensemble of decision trees is learned from neighborhoods of the instance under investigation; then, the ensemble is merged into a single decision tree. Neighbor instances are synthetically generated through a genetic algorithm whose fitness function is driven by the black-box behavior. Experiments show that the proposed method advances the state-of-the-art towards a comprehensive approach that successfully covers stability and actionability of factual and counterfactual explanations.
@article{Guidotti2023Stable,author={Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Naretto, Francesca and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca},journal={Data Mining and Knowledge Discovery},line={1},month=dec,title={Stable and actionable explanations of black-box models through factual and counterfactual rules},year={2022}}
Counterfactual explanations and how to find them: literature review and benchmarking
Interpretable machine learning aims at unveiling the reasons behind predictions returned by uninterpretable classifiers. One of the most valuable types of explanation consists of counterfactuals. A counterfactual explanation reveals what should have been different in an instance to observe a diverse outcome. For instance, a bank customer asks for a loan that is rejected. The counterfactual explanation consists of what should have been different for the customer in order to have the loan accepted. Recently, there has been an explosion of proposals for counterfactual explainers. The aim of this work is to survey the most recent explainers returning counterfactual explanations. We categorize explainers based on the approach adopted to return the counterfactuals, and we label them according to characteristics of the method and properties of the counterfactuals returned. In addition, we visually compare the explanations, and we report quantitative benchmarking assessing minimality, actionability, stability, diversity, discriminative power, and running time. The results make evident that the current state of the art does not provide a counterfactual explainer able to guarantee all these properties simultaneously.
@article{Guidotti_2022,author={Guidotti, Riccardo},doi={10.1007/s10618-022-00831-6},issn={1573-756X},journal={Data Mining and Knowledge Discovery},line={1},month=apr,publisher={Springer Science and Business Media LLC},title={Counterfactual explanations and how to find them: literature review and benchmarking},year={2022}}
Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers
Carlo
Metta, Riccardo
Guidotti, Yuan
Yin, Patrick
Gallinari, and Salvatore
Rinzivillo
Explainable AI consists in developing models allowing interaction between decision systems and humans by making the decisions understandable. We propose a case study for skin lesion diagnosis showing how it is possible to provide explanations of the decisions of deep neural network trained to label skin lesions.
@inbook{Metta_2022,author={Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore},booktitle={Frontiers in Artificial Intelligence and Applications},doi={10.3233/faia220209},issn={1879-8314},line={1,3},month=sep,publisher={IOS Press},title={Exemplars and Counterexemplars Explanations for Skin Lesion Classifiers},year={2022}}
Evaluating the Privacy Exposure of Interpretable Global Explainers
Francesca
Naretto, Anna
Monreale, and Fosca
Giannotti
In 2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2022
In recent years we are witnessing the diffusion of AI systems based on powerful Machine Learning models which find application in many critical contexts such as medicine, financial market and credit scoring. In such a context it is particularly important to design Trustworthy AI systems while guaranteeing transparency, with respect to their decision reasoning and privacy protection. Although many works in the literature addressed the lack of transparency and the risk of privacy exposure of Machine Learning models, the privacy risks of explainers have not been appropriately studied. This paper presents a methodology for evaluating the privacy exposure raised by interpretable global explainers able to imitate the original black-box classifier. Our methodology exploits the well-known Membership Inference Attack. The experimental results highlight that global explainers based on interpretable trees lead to an increase in privacy exposure.
@inproceedings{Naretto_2022,author={Naretto, Francesca and Monreale, Anna and Giannotti, Fosca},booktitle={2022 IEEE 4th International Conference on Cognitive Machine Intelligence (CogMI)},doi={10.1109/cogmi56440.2022.00012},line={5},month=dec,publisher={IEEE},title={Evaluating the Privacy Exposure of Interpretable Global Explainers},year={2022}}
Privacy Risk of Global Explainers
Francesca
Naretto, Anna
Monreale, and Fosca
Giannotti
@inbook{Naretto_2022_Privacy,author={Naretto, Francesca and Monreale, Anna and Giannotti, Fosca},booktitle={Frontiers in Artificial Intelligence and Applications},doi={10.3233/faia220206},issn={1879-8314},line={5},month=sep,publisher={IOS Press},title={Privacy Risk of Global Explainers},year={2022}}
Methods and tools for causal discovery and causal inference
Ana Rita
Nogueira, Andrea
Pugnana, Salvatore
Ruggieri, Dino
Pedreschi, and João
Gama
WIREs Data Mining and Knowledge Discovery, Jan 2022
Causality is a complex concept, which roots its developments across several fields, such as statistics, economics, epidemiology, computer science, and philosophy. In recent years, the study of causal relationships has become a crucial part of the Artificial Intelligence community, as causality can be a key tool for overcoming some limitations of correlation-based Machine Learning systems. Causality research can generally be divided into two main branches, that is, causal discovery and causal inference. The former focuses on obtaining causal knowledge directly from observational data. The latter aims to estimate the impact deriving from a change of a certain variable over an outcome of interest. This article aims at covering several methodologies that have been developed for both tasks. This survey does not only focus on theoretical aspects. But also provides a practical toolkit for interested researchers and practitioners, including software, datasets, and running examples.
@article{Nogueira_2022,author={Nogueira, Ana Rita and Pugnana, Andrea and Ruggieri, Salvatore and Pedreschi, Dino and Gama, João},doi={10.1002/widm.1449},issn={1942-4795},journal={WIREs Data Mining and Knowledge Discovery},line={1},month=jan,number={2},publisher={Wiley},title={Methods and tools for causal discovery and causal inference},volume={12},year={2022}}
Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems
Cecilia
Panigutti, Andrea
Beretta, Fosca
Giannotti, and Dino
Pedreschi
In CHI Conference on Human Factors in Computing Systems , Apr 2022
The field of eXplainable Artificial Intelligence (XAI) focuses on providing explanations for AI systems’ decisions. XAI applications to AI-based Clinical Decision Support Systems (DSS) should increase trust in the DSS by allowing clinicians to investigate the reasons behind its suggestions. In this paper, we present the results of a user study on the impact of advice from a clinical DSS on healthcare providers’ judgment in two different cases: the case where the clinical DSS explains its suggestion and the case it does not. We examined the weight of advice, the behavioral intention to use the system, and the perceptions with quantitative and qualitative measures. Our results indicate a more significant impact of advice when an explanation for the DSS decision is provided. Additionally, through the open-ended questions, we provide some insights on how to improve the explanations in the diagnosis forecasts for healthcare assistants, nurses, and doctors.
@inproceedings{Panigutti_2022,author={Panigutti, Cecilia and Beretta, Andrea and Giannotti, Fosca and Pedreschi, Dino},booktitle={CHI Conference on Human Factors in Computing Systems},collection={CHI ’22},doi={10.1145/3491102.3502104},line={1,3,4},month=apr,publisher={ACM},series={CHI ’22},title={Understanding the impact of explanations on advice-taking: a user study for AI-based clinical Decision Support Systems},year={2022}}
Explaining Crash Predictions on Multivariate Time Series Data
Francesco
Spinnato, Riccardo
Guidotti, Mirco
Nanni, Daniele
Maccagnola, Giulia
Paciello, and
1 more author
In Assicurazioni Generali, an automatic decision-making model is used to check real-time multivariate time series and alert if a car crash happened. In such a way, a Generali operator can call the customer to provide first assistance. The high sensitivity of the model used, combined with the fact that the model is not interpretable, might cause the operator to call customers even though a car crash did not happen but only due to a harsh deviation or the fact that the road is bumpy. Our goal is to tackle the problem of interpretability for car crash prediction and propose an eXplainable Artificial Intelligence (XAI) workflow that allows gaining insights regarding the logic behind the deep learning predictive model adopted by Generali. We reach our goal by building an interpretable alternative to the current obscure model that also reduces the training data usage and the prediction time.
@inbook{Spinnato_2022,author={Spinnato, Francesco and Guidotti, Riccardo and Nanni, Mirco and Maccagnola, Daniele and Paciello, Giulia and Farina, Antonio Bencini},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-031-18840-4_39},isbn={9783031188404},issn={1611-3349},line={1},month=dec,pages={556–566},publisher={Springer Nature Switzerland},title={Explaining Crash Predictions on Multivariate Time Series Data},year={2022}}
Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions
Andreas
Theissler, Francesco
Spinnato, Udo
Schlegel, and Riccardo
Guidotti
Time series data is increasingly used in a wide range of fields, and it is often relied on in crucial applications and high-stakes decision-making. For instance, sensors generate time series data to recognize different types of anomalies through automatic decision-making systems. Typically, these systems are realized with machine learning models that achieve top-tier performance on time series classification tasks. Unfortunately, the logic behind their prediction is opaque and hard to understand from a human standpoint. Recently, we observed a consistent increase in the development of explanation methods for time series classification justifying the need to structure and review the field. In this work, we (a) present the first extensive literature review on Explainable AI (XAI) for time series classification, (b) categorize the research field through a taxonomy subdividing the methods into time points-based, subsequences-based and instance-based, and (c) identify open research directions regarding the type of explanations and the evaluation of explanations and interpretability.
@article{Theissler_2022,author={Theissler, Andreas and Spinnato, Francesco and Schlegel, Udo and Guidotti, Riccardo},doi={10.1109/access.2022.3207765},issn={2169-3536},journal={IEEE Access},line={1},month=dec,pages={100700–100724},publisher={Institute of Electrical and Electronics Engineers (IEEE)},title={Explainable AI for Time Series Classification: A Review, Taxonomy and Research Directions},volume={10},year={2022}}
Understanding peace through the world news
Vasiliki
Voukelatou, Ioanna
Miliou, Fosca
Giannotti, and Luca
Pappalardo
Peace is a principal dimension of well-being and is the way out of inequity and violence. Thus, its measurement has drawn the attention of researchers, policymakers, and peacekeepers. During the last years, novel digital data streams have drastically changed the research in this field. The current study exploits information extracted from a new digital database called Global Data on Events, Location, and Tone (GDELT) to capture peace through the Global Peace Index (GPI). Applying predictive machine learning models, we demonstrate that news media attention from GDELT can be used as a proxy for measuring GPI at a monthly level. Additionally, we use explainable AI techniques to obtain the most important variables that drive the predictions. This analysis highlights each country’s profile and provides explanations for the predictions, and particularly for the errors and the events that drive these errors. We believe that digital data exploited by researchers, policymakers, and peacekeepers, with data science tools as powerful as machine learning, could contribute to maximizing the societal benefits and minimizing the risks to peace.
@article{Voukelatou_2022,author={Voukelatou, Vasiliki and Miliou, Ioanna and Giannotti, Fosca and Pappalardo, Luca},doi={10.1140/epjds/s13688-022-00315-z},issn={2193-1127},journal={EPJ Data Science},line={1},month=jan,number={4},publisher={Springer Science and Business Media LLC},title={Understanding peace through the world news},volume={11},year={2022}}
A Modularized Framework for Explaining Black Box Classifiers for Text Data
Mahtab
Sarvmaili, Riccardo
Guidotti, Anna
Monreale, Amilcar
Soares, Zahra
Sadeghi, and
3 more authors
Proceedings of the Canadian Conference on Artificial Intelligence, May 2022
@article{Sarvmaili2022Modularized,author={Sarvmaili, Mahtab and Guidotti, Riccardo and Monreale, Anna and Soares, Amilcar and Sadeghi, Zahra and Giannotti, Fosca and Pedreschi, Dino and Matwin, Stan},journal={Proceedings of the Canadian Conference on Artificial Intelligence},year={2022},month=may,note={https://caiac.pubpub.org/pub/71c292m6},publisher={Canadian Artificial Intelligence Association (CAIAC)},title={A {Modularized} {Framework} for {Explaining} {Black} {Box} {Classifiers} for {Text} {Data}},line={1}}
Investigating Debiasing Effects on Classification and Explainability
Marta
Marchiori Manerba, and Riccardo
Guidotti
In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , Oxford, United Kingdom, Jul 2022
During each stage of a dataset creation and development process, harmful biases can be accidentally introduced, leading to models that perpetuates marginalization and discrimination of minorities, as the role of the data used during the training is critical. We propose an evaluation framework that investigates the impact on classification and explainability of bias mitigation preprocessing techniques used to assess data imbalances concerning minorities’ representativeness and mitigate the skewed distributions discovered. Our evaluation focuses on assessing fairness, explainability and performance metrics. We analyze the behavior of local model-agnostic explainers on the original and mitigated datasets to examine whether the proxy models learned by the explainability techniques to mimic the black-boxes disproportionately rely on sensitive attributes, demonstrating biases rooted in the explainers. We conduct several experiments about known biased datasets to demonstrate our proposal’s novelty and effectiveness for evaluation and bias detection purposes.
@inproceedings{10.1145/3514094.3534170,author={Marchiori Manerba, Marta and Guidotti, Riccardo},title={Investigating Debiasing Effects on Classification and Explainability},year={2022},isbn={9781450392471},publisher={Association for Computing Machinery},address={New York, NY, USA},url={https://doi.org/10.1145/3514094.3534170},doi={10.1145/3514094.3534170},booktitle={Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society},pages={468–478},numpages={11},keywords={xai, ml evaluation, fairness in ml, data equity, bias mitigation, algorithmic bias, algorithmic auditing},location={Oxford, United Kingdom},series={AIES '22},line={2},month=jul}
How routing strategies impact urban emissions
Giuliano
Cornacchia, Matteo
Böhm, Giovanni
Mauro, Mirco
Nanni, Dino
Pedreschi, and
1 more author
In Proceedings of the 30th international conference on advances in geographic information systems , Nov 2022
Navigation apps use routing algorithms to suggest the best path to reach a user’s desired destination. Although undoubtedly useful, navigation apps’ impact on the urban environment (e.g., CO2 emissions and pollution) is still largely unclear. In this work, we design a simulation framework to assess the impact of routing algorithms on carbon dioxide emissions within an urban environment. Using APIs from TomTom and OpenStreetMap, we find that settings in which either all vehicles or none of them follow a navigation app’s suggestion lead to the worst impact in terms of CO2 emissions. In contrast, when just a portion (around half) of vehicles follow these suggestions, and some degree of randomness is added to the remaining vehicles’ paths, we observe a reduction in the overall CO2 emissions over the road network. Our work is a first step towards designing next-generation routing principles that may increase urban well-being while satisfying individual needs.
@inproceedings{cornacchia2022routing,title={How routing strategies impact urban emissions},author={Cornacchia, Giuliano and B{\"o}hm, Matteo and Mauro, Giovanni and Nanni, Mirco and Pedreschi, Dino and Pappalardo, Luca},booktitle={Proceedings of the 30th international conference on advances in geographic information systems},pages={1--4},year={2022},line={4},month=nov,doi={10.1145/3557915.3560977}}
2021
Explainable AI Within the Digital Transformation and Cyber Physical Systems: XAI Methods and Applications
This book presents Explainable Artificial Intelligence (XAI), which aims at producing explainable models that enable human users to understand and appropriately trust the obtained results. The authors discuss the challenges involved in making machine learning-based AI explainable. Firstly, that the explanations must be adapted to different stakeholders (end-users, policy makers, industries, utilities etc.) with different levels of technical knowledge (managers, engineers, technicians, etc.) in different application domains. Secondly, that it is important to develop an evaluation framework and standards in order to measure the effectiveness of the provided explanations at the human and the technical levels. This book gathers research contributions aiming at the development and/or the use of XAI techniques in order to address the aforementioned challenges in different applications such as healthcare, finance, cybersecurity, and document summarization. It allows highlighting the benefits and requirements of using explainable models in different application domains in order to provide guidance to readers to select the most adapted models to their specified problem and conditions. Includes recent developments of the use of Explainable Artificial Intelligence (XAI) in order to address the challenges of digital transition and cyber-physical systems; Provides a textual scientific description of the use of XAI in order to address the challenges of digital transition and cyber-physical systems; Presents examples and case studies in order to increase transparency and understanding of the methodological concepts.
@book{2021,doi={10.1007/978-3-030-76409-8},isbn={9783030764098},line={1,5},month=dec,publisher={Springer International Publishing},title={Explainable AI Within the Digital Transformation and Cyber Physical Systems: XAI Methods and Applications},year={2021}}
Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche
Major technological advances over the past two decades have made it possible to obtain, store and analyze a massive amount of data (also called Big Data) concerning every aspect of reality. The progress we are witnessing in this area transcends purely technical-scientific aspects, going on to pervade the sphere of daily life. Facial recognition in photos on social media (1), the personalization of content and shopping suggestions (2-3), and the management of interactions with voice assistants such as Siri and Alexa (4-5) are some examples of applications of artificial intelligence (AI) methods to analyze the huge volume of available data in real time. In the field of biomedical sciences we are witnessing a similarly rapid increase in the ability to produce and analyze data at multiple levels, with major implications for the diagnosis and treatment of many diseases. Although terms such as precision medicine and precision therapy are increasingly popular, the underlying AI concepts are often ignored by experts in the medical area. This article will discuss the basics of AI, advantages and limitations of the state of the art in medicine (and particularly in diabetology), and future prospects for routine applications.
@article{2021diabetologico,author={Panigutti, Cecilia and Bosi, Emanuele},doi={10.30682/ildia2101f},issn={1720-8335},journal={2021},line={4},month=dec,number={1,4},publisher={Bononia University Press},title={Intelligenza artificiale in ambito diabetologico: prospettive, dalla ricerca di base alle applicazioni cliniche},volume={33},year={2021}}
Deep Learning in Biology and Medicine
Davide
Bacciu, Paulo J G
Lisboa, and Alfredo
Vellido
Biology, medicine and biochemistry have become data-centric fields for which Deep Learning methods are delivering groundbreaking results. Addressing high impact challenges, Deep Learning in Biology and Medicine provides an accessible and organic collection of Deep Learning essays on bioinformatics and medicine. It caters for a wide readership, ranging from machine learning practitioners and data scientists seeking methodological knowledge to address biomedical applications, to life science specialists in search of a gentle reference for advanced data analytics.
With contributions from internationally renowned experts, the book covers foundational methodologies in a wide spectrum of life sciences applications, including electronic health record processing, diagnostic imaging, text processing, as well as omics-data processing. This survey of consolidated problems is complemented by a selection of advanced applications, including cheminformatics and biomedical interaction network analysis. A modern and mindful approach to the use of data-driven methodologies in the life sciences also requires careful consideration of the associated societal, ethical, legal and transparency challenges, which are covered in the concluding chapters of this book.
@book{Bacciu_2021,author={Bacciu, Davide and Lisboa, Paulo J G and Vellido, Alfredo},doi={10.1142/q0322},isbn={9781800610941},line={4},month=jun,publisher={WORLD SCIENTIFIC (EUROPE)},title={Deep Learning in Biology and Medicine},year={2021}}
Deriving a Single Interpretable Model by Merging Tree-Based Classifiers
Valerio
Bonsignori, Riccardo
Guidotti, and Anna
Monreale
Decision tree classifiers have been proved to be among the most interpretable models due to their intuitive structure that illustrates decision processes in form of logical rules. Unfortunately, more complex tree-based classifiers such as oblique trees and random forests overcome the accuracy of decision trees at the cost of becoming non interpretable. In this paper, we propose a method that takes as input any tree-based classifier and returns a single decision tree able to approximate its behavior. Our proposal merges tree-based classifiers by an intensional and extensional approach and applies a post-hoc explanation strategy. Our experiments shows that the retrieved single decision tree is at least as accurate as the original tree-based model, faithful, and more interpretable.
@inbook{Bonsignori_2021,author={Bonsignori, Valerio and Guidotti, Riccardo and Monreale, Anna},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-88942-5_27},isbn={9783030889425},issn={1611-3349},line={1},month=dec,pages={347–357},publisher={Springer International Publishing},title={Deriving a Single Interpretable Model by Merging Tree-Based Classifiers},year={2021}}
Trustworthy AI
Raja
Chatila, Virginia
Dignum, Michael
Fisher, Fosca
Giannotti, Katharina
Morik, and
2 more authors
Modern AI systems have become of widespread use in almost all sectors with a strong impact on our society. However, the very methods on which they rely, based on Machine Learning techniques for processing data to predict outcomes and to make decisions, are opaque, prone to bias and may produce wrong answers. Objective functions optimized in learning systems are not guaranteed to align with the values that motivated their definition. Properties such as transparency, verifiability, explainability, security, technical robustness and safety, are key to build operational governance frameworks, so that to make AI systems justifiably trustworthy and to align their development and use with human rights and values.
@inbook{Chatila_2021,author={Chatila, Raja and Dignum, Virginia and Fisher, Michael and Giannotti, Fosca and Morik, Katharina and Russell, Stuart and Yeung, Karen},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-69128-8_2},isbn={9783030691288},issn={1611-3349},line={1,5},month=dec,pages={13–39},publisher={Springer International Publishing},title={Trustworthy AI},year={2021}}
Matrix Profile-Based Interpretable Time Series Classifier
Time series classification (TSC) is a pervasive and transversal problem in various fields ranging from disease diagnosis to anomaly detection in finance. Unfortunately, the most effective models used by Artificial Intelligence (AI) systems for TSC are not interpretable and hide the logic of the decision process, making them unusable in sensitive domains. Recent research is focusing on explanation methods to pair with the obscure classifier to recover this weakness. However, a TSC approach that is transparent by design and is simultaneously efficient and effective is even more preferable. To this aim, we propose an interpretable TSC method based on the patterns, which is possible to extract from the Matrix Profile (MP) of the time series in the training set. A smart design of the classification procedure allows obtaining an efficient and effective transparent classifier modeled as a decision tree that expresses the reasons for the classification as the presence of discriminative subsequences. Quantitative and qualitative experimentation shows that the proposed method overcomes the state-of-the-art interpretable approaches.
@article{Guidotti_2021,author={Guidotti, Riccardo and D’Onofrio, Matteo},doi={10.3389/frai.2021.699448},issn={2624-8212},journal={Frontiers in Artificial Intelligence},line={1},month=oct,publisher={Frontiers Media SA},title={Matrix Profile-Based Interpretable Time Series Classifier},volume={4},year={2021}}
Designing Shapelets for Interpretable Data-Agnostic Classification
Riccardo
Guidotti, and Anna
Monreale
In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , Jul 2021
Time series shapelets are discriminatory subsequences which are representative of a class, and their similarity to a time series can be used for successfully tackling the time series classification problem. The literature shows that Artificial Intelligence (AI) systems adopting classification models based on time series shapelets can be interpretable, more accurate, and significantly fast. Thus, in order to design a data-agnostic and interpretable classification approach, in this paper we first extend the notion of shapelets to different types of data, i.e., images, tabular and textual data. Then, based on this extended notion of shapelets we propose an interpretable data-agnostic classification method. Since the shapelets discovery can be time consuming, especially for data types more complex than time series, we exploit a notion of prototypes for finding candidate shapelets, and reducing both the time required to find a solution and the variance of shapelets. A wide experimentation on datasets of different types shows that the data-agnostic prototype-based shapelets returned by the proposed method empower an interpretable classification which is also fast, accurate, and stable. In addition, we show and we prove that shapelets can be at the basis of explainable AI methods.
@inproceedings{Guidotti_2021_Designing,author={Guidotti, Riccardo and Monreale, Anna},booktitle={Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society},collection={AIES ’21},doi={10.1145/3461702.3462553},line={1},month=jul,publisher={ACM},series={AIES ’21},title={Designing Shapelets for Interpretable Data-Agnostic Classification},year={2021}}
Evaluating local explanation methods on ground truth
Evaluating local explanation methods is a difficult task due to the lack of a shared and universally accepted definition of explanation. In the literature, one of the most common ways to assess the performance of an explanation method is to measure the fidelity of the explanation with respect to the classification of a black box model adopted by an Artificial Intelligent system for making a decision. However, this kind of evaluation only measures the degree of adherence of the local explainer in reproducing the behavior of the black box classifier with respect to the final decision. Therefore, the explanation provided by the local explainer could be different in the content even though it leads to the same decision of the AI system. In this paper, we propose an approach that allows to measure to which extent the explanations returned by local explanation methods are correct with respect to a synthetic ground truth explanation. Indeed, the proposed methodology enables the generation of synthetic transparent classifiers for which the reason for the decision taken, i.e., a synthetic ground truth explanation, is available by design. Experimental results show how the proposed approach allows to easily evaluate local explanations on the ground truth and to characterize the quality of local explanation methods.
@article{Guidotti_2021_Evaluating,author={Guidotti, Riccardo},doi={10.1016/j.artint.2020.103428},issn={0004-3702},journal={Artificial Intelligence},line={1},month=feb,pages={103428},publisher={Elsevier BV},title={Evaluating local explanation methods on ground truth},volume={291},year={2021}}
FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems
Marta Marchiori
Manerba, and Riccardo
Guidotti
In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021
At every stage of a supervised learning process, harmful biases can arise and be inadvertently introduced, ultimately leading to marginalization, discrimination, and abuse towards minorities. This phenomenon becomes particularly impactful in the sensitive real-world context of abusive language detection systems, where non-discrimination is difficult to assess. In addition, given the opaqueness of their internal behavior, the dynamics leading a model to a certain decision are often not clear nor accountable, and significant problems of trust could emerge. A robust value-oriented evaluation of models’ fairness is therefore necessary. In this paper, we present FairShades, a model-agnostic approach for auditing the outcomes of abusive language detection systems. Combining explainability and fairness evaluation, FairShades can identify unintended biases and sensitive categories towards which models are most discriminative. This objective is pursued through the auditing of meaningful counterfactuals generated within CheckList framework. We conduct several experiments on BERT-based models to demonstrate our proposal’s novelty and effectiveness for unmasking biases.
@inproceedings{Manerba_2021,author={Manerba, Marta Marchiori and Guidotti, Riccardo},booktitle={2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)},doi={10.1109/cogmi52975.2021.00014},line={5},month=dec,publisher={IEEE},title={FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems},year={2021}}
Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling
Carlo
Metta, Riccardo
Guidotti, Yuan
Yin, Patrick
Gallinari, and Salvatore
Rinzivillo
In 2021 IEEE Symposium on Computers and Communications (ISCC) , Sep 2021
Explainable AI consists in developing mechanisms allowing for an interaction between decision systems and humans by making the decisions of the formers understandable. This is particularly important in sensitive contexts like in the medical domain. We propose a use case study, for skin lesion diagnosis, illustrating how it is possible to provide the practitioner with explanations on the decisions of a state of the art deep neural network classifier trained to characterize skin lesions from examples. Our framework consists of a trained classifier onto which an explanation module operates. The latter is able to offer the practitioner exemplars and counterexemplars for the classification diagnosis thus allowing the physician to interact with the automatic diagnosis system. The exemplars are generated via an adversarial autoencoder. We illustrate the behavior of the system on representative examples.
@inproceedings{Metta_2021,author={Metta, Carlo and Guidotti, Riccardo and Yin, Yuan and Gallinari, Patrick and Rinzivillo, Salvatore},booktitle={2021 IEEE Symposium on Computers and Communications (ISCC)},doi={10.1109/iscc53001.2021.9631485},line={1},month=sep,publisher={IEEE},title={Exemplars and Counterexemplars Explanations for Image Classifiers, Targeting Skin Lesion Labeling},year={2021}}
FairLens: Auditing black-box clinical decision support systems
Cecilia
Panigutti, Alan
Perotti, André
Panisson, Paolo
Bajardi, and Dino
Pedreschi
Highlights: We present a pipeline to detect and explain potential fairness issues in Clinical DSS. We study and compare different multi-label classification disparity measures. We explore ICD9 bias in MIMIC-IV, an openly available ICU benchmark dataset
@article{Panigutti_2021,author={Panigutti, Cecilia and Perotti, Alan and Panisson, André and Bajardi, Paolo and Pedreschi, Dino},doi={10.1016/j.ipm.2021.102657},issn={0306-4573},journal={Information Processing & Management},line={1,4},month=sep,number={5},pages={102657},publisher={Elsevier BV},title={FairLens: Auditing black-box clinical decision support systems},volume={58},year={2021}}
Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals
The biomedical field is characterized by an ever-increasing production of sequential data, which often come in the form of biosignals capturing the time-evolution of physiological processes, such as blood pressure and brain activity. This has motivated a large body of research dealing with the development of machine learning techniques for the predictive analysis of such biosignals. Unfortunately, in high-stakes decision making, such as clinical diagnosis, the opacity of machine learning models becomes a crucial aspect to be addressed in order to increase the trust and adoption of AI technology. In this paper, we propose a model agnostic explanation method, based on occlusion, that enables the learning of the input’s influence on the model predictions. We specifically target problems involving the predictive analysis of time-series data and the models that are typically used to deal with data of such nature, i.e., recurrent neural networks. Our approach is able to provide two different kinds of explanations: one suitable for technical experts, who need to verify the quality and correctness of machine learning models, and one suited to physicians, who need to understand the rationale underlying the prediction to make aware decisions. A wide experimentation on different physiological data demonstrates the effectiveness of our approach both in classification and regression tasks.
@article{Resta_2021,author={Resta, Michele and Monreale, Anna and Bacciu, Davide},doi={10.3390/e23081064},issn={1099-4300},journal={Entropy},line={1},month=aug,number={8},pages={1064},publisher={MDPI AG},title={Occlusion-Based Explanations in Deep Recurrent Models for Biomedical Signals},volume={23},year={2021}}
TRIPLEx: Triple Extraction for Explanation
Mattia
Setzu, Anna
Monreale, and Pasquale
Minervini
In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021
Transformer-based models are used to solve a va-riety of Natural Language Processing tasks. Still, these models are opaque and poorly understandable for their users. Current approaches to explainability focus on token importance, in which the explanation consists of a set of tokens relevant to the prediction, and natural language explanations, in which the explanation is a generated piece of text. The latter are usually learned by design with models traind end-to-end to provide a prediction and an explanation, or rely on powerful external text generators to do the heavy lifting for them. In this paper we present TRIPLEX, an explainability algorithm for Transformer-based models fine-tuned on Natural Language Inference, Semantic Text Similarity, or Text Classification tasks. TRIPLEX explains Transformers-based models by extracting a set of facts from the input data, subsuming it by abstraction, and generating a set of weighted triples as explanation.
@inproceedings{Setzu_2021,author={Setzu, Mattia and Monreale, Anna and Minervini, Pasquale},booktitle={2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)},doi={10.1109/cogmi52975.2021.00015},line={1,2},month=dec,publisher={IEEE},title={TRIPLEx: Triple Extraction for Explanation},year={2021}}
GLocalX - From Local to Global Explanations of Black Box AI Models
Mattia
Setzu, Riccardo
Guidotti, Anna
Monreale, Franco
Turini, Dino
Pedreschi, and
1 more author
Artificial Intelligence (AI) has come to prominence as one of the major components of our society, with applications in most aspects of our lives. In this field, complex and highly nonlinear machine learning models such as ensemble models, deep neural networks, and Support Vector Machines have consistently shown remarkable accuracy in solving complex tasks. Although accurate, AI models often are “black boxes” which we are not able to understand. Relying on these models has a multifaceted impact and raises significant concerns about their transparency. Applications in sensitive and critical domains are a strong motivational factor in trying to understand the behavior of black boxes. We propose to address this issue by providing an interpretable layer on top of black box models by aggregating “local” explanations. We present GLocalX, a “local-first” model agnostic explanation method. Starting from local explanations expressed in form of local decision rules, GLocalX iteratively generalizes them into global explanations by hierarchically aggregating them. Our goal is to learn accurate yet simple interpretable models to emulate the given black box, and, if possible, replace it entirely. We validate GLocalX in a set of experiments in standard and constrained settings with limited or no access to either data or local explanations. Experiments show that GLocalX is able to accurately emulate several models with simple and small models, reaching state-of-the-art performance against natively global solutions. Our findings show how it is often possible to achieve a high level of both accuracy and comprehensibility of classification models, even in complex domains with high-dimensional data, without necessarily trading one property for the other. This is a key requirement for a trustworthy AI, necessary for adoption in high-stakes decision making applications.
@article{Setzu_2021_GLocalX,author={Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco and Pedreschi, Dino and Giannotti, Fosca},doi={10.1016/j.artint.2021.103457},issn={0004-3702},journal={Artificial Intelligence},line={1,2},month=may,pages={103457},publisher={Elsevier BV},title={GLocalX - From Local to Global Explanations of Black Box AI Models},volume={294},year={2021}}
Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery
Martina
Cinquini, Fosca
Giannotti, and Riccardo
Guidotti
In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) , Dec 2021
Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, artificial intelligence explanation, etc. In all such contexts, it is important to generate plausible data samples. A common assumption of approaches widely used for data generation is the independence of the features. However, typically, the variables of a dataset de-pend on one another, and these dependencies are not considered in data generation leading to the creation of implausible records. The main problem is that dependencies among variables are typically unknown. In this paper, we design a synthetic dataset generator for tabular data that is able to discover nonlinear causalities among the variables and use them at generation time. State-of-the-art methods for nonlinear causal discovery are typically inefficient. We boost them by restricting the causal discovery among the features appearing in the frequent patterns efficiently retrieved by a pattern mining algorithm. To validate our proposal, we design a framework for generating synthetic datasets with known causalities. Wide experimentation on many synthetic datasets and real datasets with known causalities shows the effectiveness of the proposed method.
@inproceedings{9750370,author={Cinquini, Martina and Giannotti, Fosca and Guidotti, Riccardo},booktitle={ 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI) },title={{ Boosting Synthetic Data Generation with Effective Nonlinear Causal Discovery }},year={2021},volume={},issn={},pages={54-63},keywords={Software testing;Deep learning;Data privacy;Filtering;Neural networks;Learning (artificial intelligence);Generators},doi={10.1109/CogMI52975.2021.00016},url={https://doi.ieeecomputersociety.org/10.1109/CogMI52975.2021.00016},publisher={IEEE Computer Society},address={Los Alamitos, CA, USA},month=dec,line={1}}
2020
Explaining Any Time Series Classifier
Riccardo
Guidotti, Anna
Monreale, Francesco
Spinnato, Dino
Pedreschi, and Fosca
Giannotti
In 2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI) , Oct 2020
We present a method to explain the decisions of black box models for time series classification. The explanation consists of factual and counterfactual shapelet-based rules revealing the reasons for the classification, and of a set of exemplars and counter-exemplars highlighting similarities and differences with the time series under analysis. The proposed method first generates exemplar and counter-exemplar time series in the latent feature space and learns a local latent decision tree classifier. Then, it selects and decodes those respecting the decision rules explaining the decision. Finally, it learns on them a shapelet-tree that reveals the parts of the time series that must, and must not, be contained for getting the returned outcome from the black box. A wide experimentation shows that the proposed method provides faithful, meaningful and interpretable explanations.
@inproceedings{Guidotti_2020,author={Guidotti, Riccardo and Monreale, Anna and Spinnato, Francesco and Pedreschi, Dino and Giannotti, Fosca},booktitle={2020 IEEE Second International Conference on Cognitive Machine Intelligence (CogMI)},doi={10.1109/cogmi50398.2020.00029},line={1},month=oct,publisher={IEEE},title={Explaining Any Time Series Classifier},year={2020}}
Black Box Explanation by Learning Image Exemplars in the Latent Feature Space
Riccardo
Guidotti, Anna
Monreale, Stan
Matwin, and Dino
Pedreschi
We present an approach to explain the decisions of black box models for image classification. While using the black box to label images, our explanation method exploits the latent feature space learned through an adversarial autoencoder. The proposed method first generates exemplar images in the latent feature space and learns a decision tree classifier. Then, it selects and decodes exemplars respecting local decision rules. Finally, it visualizes them in a manner that shows to the user how the exemplars can be modified to either stay within their class, or to become counter-factuals by “morphing” into another class. Since we focus on black box decision systems for image classification, the explanation obtained from the exemplars also provides a saliency map highlighting the areas of the image that contribute to its classification, and areas of the image that push it into another class. We present the results of an experimental evaluation on three datasets and two black box models. Besides providing the most useful and interpretable explanations, we show that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability.
@inbook{Guidotti_2020_Black_Box,author={Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-46150-8_12},isbn={9783030461508},issn={1611-3349},line={1},month=dec,pages={189–205},publisher={Springer International Publishing},title={Black Box Explanation by Learning Image Exemplars in the Latent Feature Space},year={2020}}
Data-Agnostic Local Neighborhood Generation
Riccardo
Guidotti, and Anna
Monreale
In 2020 IEEE International Conference on Data Mining (ICDM) . More Information can be found here
, Nov 2020
Synthetic data generation has been widely adopted in software testing, data privacy, imbalanced learning, machine learning explanation, etc. In such contexts, it is important to generate data samples located within “local” areas surrounding specific instances. Local synthetic data can help the learning phase of predictive models, and it is fundamental for methods explaining the local behavior of obscure classifiers. The contribution of this paper is twofold. First, we introduce a method based on generative operators allowing the synthetic neighborhood generation by applying specific perturbations on a given input instance. The key factor consists in performing a data transformation that makes applicable to any type of data, i.e., data-agnostic. Second, we design a framework for evaluating the goodness of local synthetic neighborhoods exploiting both supervised and unsupervised methodologies. A deep experimentation shows the effectiveness of the proposed method.
@inproceedings{Guidotti_2020_Data-Agnostic,author={Guidotti, Riccardo and Monreale, Anna},booktitle={2020 IEEE International Conference on Data Mining (ICDM)},doi={10.1109/icdm50108.2020.00122},line={1,5},month=nov,publisher={IEEE},title={Data-Agnostic Local Neighborhood Generation},year={2020}}
Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations
Riccardo
Guidotti, Anna
Monreale, Stan
Matwin, and Dino
Pedreschi
Proceedings of the AAAI Conference on Artificial Intelligence, Apr 2020
We present an approach to explain the decisions of black box image classifiers through synthetic exemplar and counter-exemplar learnt in the latent feature space. Our explanation method exploits the latent representations learned through an adversarial autoencoder for generating a synthetic neighborhood of the image for which an explanation is required. A decision tree is trained on a set of images represented in the latent space, and its decision rules are used to generate exemplar images showing how the original image can be modified to stay within its class. Counterfactual rules are used to generate counter-exemplars showing how the original image can “morph” into another class. The explanation also comprehends a saliency map highlighting the areas that contribute to its classification, and areas that push it into another class. A wide and deep experimental evaluation proves that the proposed method outperforms existing explainers in terms of fidelity, relevance, coherence, and stability, besides providing the most useful and interpretable explanations.
@article{Guidotti_2020_Image_Classifiers,author={Guidotti, Riccardo and Monreale, Anna and Matwin, Stan and Pedreschi, Dino},doi={10.1609/aaai.v34i09.7116},issn={2159-5399},journal={Proceedings of the AAAI Conference on Artificial Intelligence},line={1},month=apr,number={09},pages={13665–13668},publisher={Association for the Advancement of Artificial Intelligence (AAAI)},title={Explaining Image Classifiers Generating Exemplars and Counter-Exemplars from Latent Representations},volume={34},year={2020}}
Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars
Orestis
Lampridis, Riccardo
Guidotti, and Salvatore
Ruggieri
We present xspells, a model-agnostic local approach for explaining the decisions of a black box model for sentiment classification of short texts. The explanations provided consist of a set of exemplar sentences and a set of counter-exemplar sentences. The former are examples classified by the black box with the same label as the text to explain. The latter are examples classified with a different label (a form of counter-factuals). Both are close in meaning to the text to explain, and both are meaningful sentences – albeit they are synthetically generated. xspells generates neighbors of the text to explain in a latent space using Variational Autoencoders for encoding text and decoding latent instances. A decision tree is learned from randomly generated neighbors, and used to drive the selection of the exemplars and counter-exemplars. We report experiments on two datasets showing that xspells outperforms the well-known lime method in terms of quality of explanations, fidelity, and usefulness, and that is comparable to it in terms of stability.
@inbook{Lampridis_2020,author={Lampridis, Orestis and Guidotti, Riccardo and Ruggieri, Salvatore},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-61527-7_24},isbn={9783030615277},issn={1611-3349},line={1},month=dec,pages={357–373},publisher={Springer International Publishing},title={Explaining Sentiment Classification with Synthetic Exemplars and Counter-Exemplars},year={2020}}
Predicting and Explaining Privacy Risk Exposure in Mobility Data
Francesca
Naretto, Roberto
Pellungrini, Anna
Monreale, Franco Maria
Nardini, and Mirco
Musolesi
Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.
@inbook{Naretto_2020_Predicting,author={Naretto, Francesca and Pellungrini, Roberto and Monreale, Anna and Nardini, Franco Maria and Musolesi, Mirco},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-61527-7_27},isbn={9783030615277},issn={1611-3349},line={5},month=dec,pages={403–418},publisher={Springer International Publishing},title={Predicting and Explaining Privacy Risk Exposure in Mobility Data},year={2020}}
Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks
Francesca
Naretto, Roberto
Pellungrini, Franco Maria
Nardini, and Fosca
Giannotti
The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.
@inbook{Naretto_2020_Prediction,author={Naretto, Francesca and Pellungrini, Roberto and Nardini, Franco Maria and Giannotti, Fosca},booktitle={Communications in Computer and Information Science},doi={10.1007/978-3-030-65965-3_34},isbn={9783030659653},issn={1865-0937},line={5},month=dec,pages={501–516},publisher={Springer International Publishing},title={Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks},year={2020}}
Doctor XAI: an ontology-based approach to black-box sequential data classification explanations
Cecilia
Panigutti, Alan
Perotti, and Dino
Pedreschi
In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency , Jan 2020
Several recent advancements in Machine Learning involve blackbox models: algorithms that do not provide human-understandable explanations in support of their decisions. This limitation hampers the fairness, accountability and transparency of these models; the field of eXplainable Artificial Intelligence (XAI) tries to solve this problem providing human-understandable explanations for black-box models. However, healthcare datasets (and the related learning tasks) often present peculiar features, such as sequential data, multi-label predictions, and links to structured background knowledge. In this paper, we introduce Doctor XAI, a model-agnostic explainability technique able to deal with multi-labeled, sequential, ontology-linked data. We focus on explaining Doctor AI, a multilabel classifier which takes as input the clinical history of a patient in order to predict the next visit. Furthermore, we show how exploiting the temporal dimension in the data and the domain knowledge encoded in the medical ontology improves the quality of the mined explanations.
@inproceedings{Panigutti_2020,author={Panigutti, Cecilia and Perotti, Alan and Pedreschi, Dino},booktitle={Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency},collection={FAT* ’20},doi={10.1145/3351095.3372855},line={1,4},month=jan,publisher={ACM},series={FAT* ’20},title={Doctor XAI: an ontology-based approach to black-box sequential data classification explanations},year={2020}}
Global Explanations with Local Scoring
Mattia
Setzu, Riccardo
Guidotti, Anna
Monreale, and Franco
Turini
Artificial Intelligence systems often adopt machine learning models encoding complex algorithms with potentially unknown behavior. As the application of these “black box” models grows, it is our responsibility to understand their inner working and formulate them in human-understandable explanations. To this end, we propose a rule-based model-agnostic explanation method that follows a local-to-global schema: it generalizes a global explanation summarizing the decision logic of a black box starting from the local explanations of single predicted instances. We define a scoring system based on a rule relevance score to extract global explanations from a set of local explanations in the form of decision rules. Experiments on several datasets and black boxes show the stability, and low complexity of the global explanations provided by the proposed solution in comparison with baselines and state-of-the-art global explainers.
@inbook{Setzu_2020,author={Setzu, Mattia and Guidotti, Riccardo and Monreale, Anna and Turini, Franco},booktitle={Communications in Computer and Information Science},doi={10.1007/978-3-030-43823-4_14},isbn={9783030438234},issn={1865-0937},line={1,2},month=dec,pages={159–171},publisher={Springer International Publishing},title={Global Explanations with Local Scoring},year={2020}}
2019
Helping Your Docker Images to Spread Based on Explainable Models
Riccardo
Guidotti, Jacopo
Soldani, Davide
Neri, Antonio
Brogi, and Dino
Pedreschi
Docker is on the rise in today’s enterprise IT. It permits shipping applications inside portable containers, which run from so-called Docker images. Docker images are distributed in public registries, which also monitor their popularity. The popularity of an image impacts on its actual usage, and hence on the potential revenues for its developers. In this paper, we present a solution based on interpretable decision tree and regression trees for estimating the popularity of a given Docker image, and for understanding how to improve an image to increase its popularity. The results presented in this work can provide valuable insights to Docker developers, helping them in spreading their images.
@inbook{Guidotti_2019_Helping,author={Guidotti, Riccardo and Soldani, Jacopo and Neri, Davide and Brogi, Antonio and Pedreschi, Dino},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-10997-4_13},isbn={9783030109974},issn={1611-3349},line={1},month=dec,pages={205–221},publisher={Springer International Publishing},title={Helping Your Docker Images to Spread Based on Explainable Models},year={2019}}
Factual and Counterfactual Explanations for Black Box Decision Making
Riccardo
Guidotti, Anna
Monreale, Fosca
Giannotti, Dino
Pedreschi, Salvatore
Ruggieri, and
1 more author
The rise of sophisticated machine learning models has brought accurate but obscure decision systems, which hide their logic, thus undermining transparency, trust, and the adoption of artificial intelligence (AI) in socially sensitive and safety-critical contexts. We introduce a local rule-based explanation method, providing faithful explanations of the decision made by a black box classifier on a specific instance. The proposed method first learns an interpretable, local classifier on a synthetic neighborhood of the instance under investigation, generated by a genetic algorithm. Then, it derives from the interpretable classifier an explanation consisting of a decision rule, explaining the factual reasons of the decision, and a set of counterfactuals, suggesting the changes in the instance features that would lead to a different outcome. Experimental results show that the proposed method outperforms existing approaches in terms of the quality of the explanations and of the accuracy in mimicking the black box.
@article{Guidotti_2019Factual,author={Guidotti, Riccardo and Monreale, Anna and Giannotti, Fosca and Pedreschi, Dino and Ruggieri, Salvatore and Turini, Franco},doi={10.1109/mis.2019.2957223},issn={1941-1294},journal={IEEE Intelligent Systems},line={1},month=nov,number={6},pages={14–23},publisher={Institute of Electrical and Electronics Engineers (IEEE)},title={Factual and Counterfactual Explanations for Black Box Decision Making},volume={34},year={2019}}
Investigating Neighborhood Generation Methods for Explanations of Obscure Image Classifiers
Riccardo
Guidotti, Anna
Monreale, and Leonardo
Cariaggi
Given the wide use of machine learning approaches based on opaque prediction models, understanding the reasons behind decisions of black box decision systems is nowadays a crucial topic. We address the problem of providing meaningful explanations in the widely-applied image classification tasks. In particular, we explore the impact of changing the neighborhood generation function for a local interpretable model-agnostic explanator by proposing four different variants. All the proposed methods are based on a grid-based segmentation of the images, but each of them proposes a different strategy for generating the neighborhood of the image for which an explanation is required. A deep experimentation shows both improvements and weakness of each proposed approach.
@inbook{Guidotti_2019Investigating,author={Guidotti, Riccardo and Monreale, Anna and Cariaggi, Leonardo},booktitle={Lecture Notes in Computer Science},doi={10.1007/978-3-030-16148-4_5},isbn={9783030161484},issn={1611-3349},line={1},month=dec,pages={55–68},publisher={Springer International Publishing},title={Investigating Neighborhood Generation Methods for Explanations of Obscure Image Classifiers},year={2019}}
Explaining Multi-label Black-Box Classifiers for Health Applications
Cecilia
Panigutti, Riccardo
Guidotti, Anna
Monreale, and Dino
Pedreschi
Today the state-of-the-art performance in classification is achieved by the so-called “black boxes”, i.e. decision-making systems whose internal logic is obscure. Such models could revolutionize the health-care system, however their deployment in real-world diagnosis decision support systems is subject to several risks and limitations due to the lack of transparency. The typical classification problem in health-care requires a multi-label approach since the possible labels are not mutually exclusive, e.g. diagnoses. We propose MARLENA, a model-agnostic method which explains multi-label black box decisions. MARLENA explains an individual decision in three steps. First, it generates a synthetic neighborhood around the instance to be explained using a strategy suitable for multi-label decisions. It then learns a decision tree on such neighborhood and finally derives from it a decision rule that explains the black box decision. Our experiments show that MARLENA performs well in terms of mimicking the black box behavior while gaining at the same time a notable amount of interpretability through compact decision rules, i.e. rules with limited length.
@inbook{Panigutti_2019,author={Panigutti, Cecilia and Guidotti, Riccardo and Monreale, Anna and Pedreschi, Dino},booktitle={Studies in Computational Intelligence},doi={10.1007/978-3-030-24409-5_9},isbn={9783030244095},issn={1860-9503},line={1,4},month=aug,pages={97–110},publisher={Springer International Publishing},title={Explaining Multi-label Black-Box Classifiers for Health Applications},year={2019}}
Meaningful Explanations of Black Box AI Decision Systems
Dino
Pedreschi, Fosca
Giannotti, Riccardo
Guidotti, Anna
Monreale, Salvatore
Ruggieri, and
1 more author
Proceedings of the AAAI Conference on Artificial Intelligence, Jul 2019
Black box AI systems for automated decision making, often based on machine learning over (big) data, map a user’s features into a class or a score without exposing the reasons why. This is problematic not only for lack of transparency, but also for possible biases inherited by the algorithms from human prejudices and collection artifacts hidden in the training data, which may lead to unfair or wrong decisions. We focus on the urgent open challenge of how to construct meaningful explanations of opaque AI/ML systems, introducing the local-toglobal framework for black box explanation, articulated along three lines: (i) the language for expressing explanations in terms of logic rules, with statistical and causal interpretation; (ii) the inference of local explanations for revealing the decision rationale for a specific case, by auditing the black box in the vicinity of the target instance; (iii), the bottom-up generalization of many local explanations into simple global ones, with algorithms that optimize for quality and comprehensibility. We argue that the local-first approach opens the door to a wide variety of alternative solutions along different dimensions: a variety of data sources (relational, text, images, etc.), a variety of learning problems (multi-label classification, regression, scoring, ranking), a variety of languages for expressing meaningful explanations, a variety of means to audit a black box.
@article{Pedreschi_2019,author={Pedreschi, Dino and Giannotti, Fosca and Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco},doi={10.1609/aaai.v33i01.33019780},issn={2159-5399},journal={Proceedings of the AAAI Conference on Artificial Intelligence},line={1},month=jul,number={01},pages={9780–9784},publisher={Association for the Advancement of Artificial Intelligence (AAAI)},title={Meaningful Explanations of Black Box AI Decision Systems},volume={33},year={2019}}
2018
A Survey of Methods for Explaining Black Box Models
Riccardo
Guidotti, Anna
Monreale, Salvatore
Ruggieri, Franco
Turini, Fosca
Giannotti, and
1 more author
In recent years, many accurate decision support systems have been constructed as black boxes, that is as systems that hide their internal logic to the user. This lack of explanation constitutes both a practical and an ethical issue. The literature reports many approaches aimed at overcoming this crucial weakness, sometimes at the cost of sacrificing accuracy for interpretability. The applications in which black box decision systems can be used are various, and each approach is typically developed to provide a solution for a specific problem and, as a consequence, it explicitly or implicitly delineates its own definition of interpretability and explanation. The aim of this article is to provide a classification of the main problems addressed in the literature with respect to the notion of explanation and the type of black box system. Given a problem definition, a black box type, and a desired explanation, this survey should help the researcher to find the proposals more useful for his own work. The proposed classification of approaches to open black box models should also be useful for putting the many research open questions in perspective.
@article{Guidotti_2018,author={Guidotti, Riccardo and Monreale, Anna and Ruggieri, Salvatore and Turini, Franco and Giannotti, Fosca and Pedreschi, Dino},doi={10.1145/3236009},issn={1557-7341},journal={ACM Computing Surveys},line={1},month=aug,number={5},pages={1–42},publisher={Association for Computing Machinery (ACM)},title={A Survey of Methods for Explaining Black Box Models},volume={51},year={2018}}