Ethical/legal framework for explanation

5

LINE

Explainability is one of the most important ethical and legal values identified by EU regulations (GDPR, AI act). However, when we design a trustworthy AI System it is important to take into consideration also the other ethical and legal values such as privacy, fairness, safety, robustness, etc. This requires analyzing the interplay among the different values and understanding whether they are in contrast or not.

In this direction, the work of Line 5 has started the investigation of the interplay between explainability and privacy, and explainability and fairness under different viewpoints. In particular, the research of this line tries to answer the following questions:

  1. Can explanation methods, introducing a level of transparency in the whole decision process, might jeopardize individual privacy of people represented in the training data?
  2. Can explanation methods help in understanding the reason for possible ethical risks associated with the use of AI systems (e.g., privacy violations and biased behaviors)?
  3. Can explanation methods be fundamental for discovering other ethical issues like unfair behavior of AI systems?
Starting from these questions, we have scientifically contributed with the following studies. Concerning the ability of explanation methods to explain some ethical risks such as privacy and biases, in [NPM2020,NPN2020] we propose EXPERT, an EXplainable Privacy ExposuRe predicTion framework. It is a tool that exploits Explainability as a tool for increasing privacy user awareness. We applied EXPERT on the privacy risk prediction and explanation of both tabular data [NPM2020] and sequential data [NPN2020]. In the first setting we considered the mobility context where for each user we have the historical movements (a spatio-temporal trajectory). In this context, EXPERT using the privacy risk exposure module extracts from human mobility data an individual mobility profile describing the mobility behavior of any user. Second, for each user it simulates a privacy attack and quantifies the associated privacy risk. Third, it uses the mobility profiles of the users with their associated privacy risks to train a ML model. For a new user, along with the prediction of risk, EXPERT also provides an explanation of the predicted risk generated by the risk explanation module. EXPERT exploits two state-of-the-art explanation techniques, i.e., SHAP and LORE [GMG2019]. In the second setting [NPN2020] the prediction phase is not based on a mobility profile but on the trajectory itself and given the absence of features we used only SHAP as an explanation method. In [PPB2021] we also present FairLens, a framework able to detect and explain potential bias issues in Clinical Decision Support Systems (DSS). FairLens allows testing the clinical DSS before its deployment, i.e., before handling it to final decision-makers such as physicians and nurses. FairLens takes bias analysis a step further by explaining the reasons behind the poor model performance on specific groups; in particular, it uses GlocalX to explain which elements in the patients’ clinical histories are influencing the misclassification.

Concerning the use of explainability as a means for discovering unfair behaviors, in [MG2021], we propose FairShades, a model-agnostic approach for auditing the outcomes of abusive language detection systems. FairShades combines explainability and fairness evaluation within a proactive pipeline to identify unintended biases and sensitive categories toward which the black box model under assessment is most discriminative. It is a task-specific approach for abusive language detection: it can be used to test the fairness of any abusive language detection system working on any textual dataset. However, its ideal application is on sentences containing protected identities, i.e., expressions referring to nationality, gender, etc., as the primary scope is to uncover biases and not explain the reasons for the prediction.

Publications

6.

[MG2022]
Investigating Debiasing Effects on Classification and Explainability
Marta Marchiori Manerba, Guidotti Riccardo (2022) - Conference on AI, Ethics, and Society (AIES 2022). In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society (AIES'22)

Abstract

During each stage of a dataset creation and development process, harmful biases can be accidentally introduced, leading to models that perpetuates marginalization and discrimination of minorities, as the role of the data used during the training is critical. We propose an evaluation framework that investigates the impact on classification and explainability of bias mitigation preprocessing techniques used to assess data imbalances concerning minorities' representativeness and mitigate the skewed distributions discovered. Our evaluation focuses on assessing fairness, explainability and performance metrics. We analyze the behavior of local model-agnostic explainers on the original and mitigated datasets to examine whether the proxy models learned by the explainability techniques to mimic the black-boxes disproportionately rely on sensitive attributes, demonstrating biases rooted in the explainers. We conduct several experiments about known biased datasets to demonstrate our proposal’s novelty and effectiveness for evaluation and bias detection purposes.

More information

Research Line 1▪5

12.

[CDF2021]
Trustworthy AI
Chatila Raja, Dignum Virginia, Fisher Michael, Giannotti Fosca, Morik Katharina, Russell Stuart, Yeung Karen (2022) - Reflections on Artificial Intelligence for Humanity. In Lecture Notes in Computer Science,

Abstract

Modern AI systems have become of widespread use in almost all sectors with a strong impact on our society. However, the very methods on which they rely, based on Machine Learning techniques for processing data to predict outcomes and to make decisions, are opaque, prone to bias and may produce wrong answers. Objective functions optimized in learning systems are not guaranteed to align with the values that motivated their definition. Properties such as transparency, verifiability, explainability, security, technical robustness and safety, are key to build operational governance frameworks, so that to make AI systems justifiably trustworthy and to align their development and use with human rights and values.

14.

[MG2021]
FairShades: Fairness Auditing via Explainability in Abusive Language Detection Systems
Marchiori Manerba Marta, Guidotti Riccardo (2021) - Third Conference on Cognitive Machine Intelligence (COGMI) 2021. In 2021 IEEE Third International Conference on Cognitive Machine Intelligence (CogMI)

Abstract

At every stage of a supervised learning process, harmful biases can arise and be inadvertently introduced, ultimately leading to marginalization, discrimination, and abuse towards minorities. This phenomenon becomes particularly impactful in the sensitive real-world context of abusive language detection systems, where non-discrimination is difficult to assess. In addition, given the opaqueness of their internal behavior, the dynamics leading a model to a certain decision are often not clear nor accountable, and significant problems of trust could emerge. A robust value-oriented evaluation of models' fairness is therefore necessary. In this paper, we present FairShades, a model-agnostic approach for auditing the outcomes of abusive language detection systems. Combining explainability and fairness evaluation, FairShades can identify unintended biases and sensitive categories towards which models are most discriminative. This objective is pursued through the auditing of meaningful counterfactuals generated within CheckList framework. We conduct several experiments on BERT-based models to demonstrate our proposal's novelty and effectiveness for unmasking biases.

More information

External Link

Research Line 1▪5

16.

[GMP2021]
immagine
Explainable AI Within the Digital Transformation and Cyber Physical Systems: XAI Methods and Applications
Guidotti Riccardo, Monreale Anna, Pedreschi Dino, Giannotti Fosca (2021) - Explainable AI Within the Digital Transformation and Cyber Physical Systems (pp. 9-31)

Abstract

This book presents Explainable Artificial Intelligence (XAI), which aims at producing explainable models that enable human users to understand and appropriately trust the obtained results. The authors discuss the challenges involved in making machine learning-based AI explainable. Firstly, that the explanations must be adapted to different stakeholders (end-users, policy makers, industries, utilities etc.) with different levels of technical knowledge (managers, engineers, technicians, etc.) in different application domains. Secondly, that it is important to develop an evaluation framework and standards in order to measure the effectiveness of the provided explanations at the human and the technical levels. This book gathers research contributions aiming at the development and/or the use of XAI techniques in order to address the aforementioned challenges in different applications such as healthcare, finance, cybersecurity, and document summarization. It allows highlighting the benefits and requirements of using explainable models in different application domains in order to provide guidance to readers to select the most adapted models to their specified problem and conditions. Includes recent developments of the use of Explainable Artificial Intelligence (XAI) in order to address the challenges of digital transition and cyber-physical systems; Provides a textual scientific description of the use of XAI in order to address the challenges of digital transition and cyber-physical systems; Presents examples and case studies in order to increase transparency and understanding of the methodological concepts.

More information

External Link

Research Line 1▪5

27.

[NPN2020]
Prediction and Explanation of Privacy Risk on Mobility Data with Neural Networks
Naretto Francesca, Pellungrini Roberto, Nardini Franco Maria, Giannotti Fosca (2021) - ECML PKDD 2020 Workshops. In ECML PKDD 2020 Workshops

Abstract

The analysis of privacy risk for mobility data is a fundamental part of any privacy-aware process based on such data. Mobility data are highly sensitive. Therefore, the correct identification of the privacy risk before releasing the data to the public is of utmost importance. However, existing privacy risk assessment frameworks have high computational complexity. To tackle these issues, some recent work proposed a solution based on classification approaches to predict privacy risk using mobility features extracted from the data. In this paper, we propose an improvement of this approach by applying long short-term memory (LSTM) neural networks to predict the privacy risk directly from original mobility data. We empirically evaluate privacy risk on real data by applying our LSTM-based approach. Results show that our proposed method based on a LSTM network is effective in predicting the privacy risk with results in terms of F1 of up to 0.91. Moreover, to explain the predictions of our model, we employ a state-of-the-art explanation algorithm, Shap. We explore the resulting explanation, showing how it is possible to provide effective predictions while explaining them to the end-user.

More information

External Link

Research Line 4▪5

28.

[NPM2020]
Predicting and Explaining Privacy Risk Exposure in Mobility Data
Naretto Francesca, Pellungrini Roberto, Monreale Anna, Nardini Franco Maria, Musolesi Mirco (2021) - Discovery Science. In Discovery Science Conference

Abstract

Mobility data is a proxy of different social dynamics and its analysis enables a wide range of user services. Unfortunately, mobility data are very sensitive because the sharing of people’s whereabouts may arise serious privacy concerns. Existing frameworks for privacy risk assessment provide tools to identify and measure privacy risks, but they often (i) have high computational complexity; and (ii) are not able to provide users with a justification of the reported risks. In this paper, we propose expert, a new framework for the prediction and explanation of privacy risk on mobility data. We empirically evaluate privacy risk on real data, simulating a privacy attack with a state-of-the-art privacy risk assessment framework. We then extract individual mobility profiles from the data for predicting their risk. We compare the performance of several machine learning algorithms in order to identify the best approach for our task. Finally, we show how it is possible to explain privacy risk prediction on real data, using two algorithms: Shap, a feature importance-based method and Lore, a rule-based method. Overall, expert is able to provide a user with the privacy risk and an explanation of the risk itself. The experiments show excellent performance for the prediction task.

More information

External Link

Research Line 4▪5

42.

[M2020]
Rischi etico-legali dell’Intelligenza Artificiale
Monreale Anna (2020) - DPCE Online, [S.l.], v. 44, n. 3. In DPCE Online, [S.l.], v. 44, n. 3, oct. 2020. ISSN 2037-6677

Abstract

nan

External Link

Research Line 5

Researchers working on this line

Riccardo Guidotti
Riccardo
Guidotti

Assitant Professor

University of Pisa


R. line 1 ▪ 3 ▪ 4 ▪ 5

Franco Turini
Franco
Turini

Full Professor

University of Pisa


R. line 1 ▪ 2 ▪ 5

Salvo Rinzivillo
Salvo
Rinzivillo

Researcher

ISTI - CNR Pisa


R. line 1 ▪ 3 ▪ 4 ▪ 5

Andrea Beretta
Andrea
Beretta

Researcher

ISTI - CNR Pisa


R. line 1 ▪ 4 ▪ 5

Anna Monreale
Anna
Monreale

Associate Professor

University of Pisa


R. line 1 ▪ 4 ▪ 5

Cecilia Panigutti
Cecilia
Panigutti

Phd Student

Scuola Normale


R. line 1 ▪ 4 ▪ 5

Roberto Pellungrini
Roberto
Pellungrini

Researcher

University of Pisa


R. line 5

Francesca Naretto
Francesca
Naretto

Phd Student

Scuola Normale


R. line 1 ▪ 3 ▪ 4 ▪ 5

Francesca Pratesi
Francesca
Pratesi

Researcher

ISTI - CNR Pisa


R. line 5

Marta Marchiori Manerba
Marta
Marchiori Manerba

Phd Student

University of Pisa


R. line 1 ▪ 2 ▪ 5