91Â鶹¹ú²ú¾«Æ·×ÔÅÄ

Authors: Paola Pisano, Luca Macis, Marco Tagliapietra  

Examples abound in our daily lives: determining the probability of a diagnosis, the risk of a customer breaching a contract (churn/attrition rates), predicting the failure of a mechanical system, determining the risk of a customer defaulting on a payment, or even the risk of an individual becoming politically or religiously radicalized. These risk predictions, also known as scores (e.g. credit scores), are produced by statistical learning algorithms that generate predictions after being trained on datasets. 

The principle of statistical learning algorithms is based on the fact that from a set of examples, called a training data set, it is possible to develop a decision rule that will apply to all future cases. From a large amount of collected data, which mainly contains decisions already made and the variables that explain them, mathematical principles allow not only to understand how the decisions were made, but also to identify the rules that guided them. Specifically, identifying these rules consists of finding trends (patterns or features) within the observations. During the training process, AI models can discover new correlations between certain input features (e.g. clinical symptoms) and make decisions or predictions (e.g. medical diagnoses) based on highly complex models involving a large number of interacting parameters (possibly millions), making it difficult even for AI experts to understand how their outputs are subsequently produced. In these situations, the reasons why systems have made certain decisions may be unclear to both the users of the systems and those affected by the systems. The resulting "black box" effect could lead to either misplaced trust or over-reliance on AI systems, both of which could have negative consequences for individuals. 

Explainable Artificial Intelligence (XAI) is the ability of AI systems to provide clear and understandable explanations for their actions and decisions. Its central goal is to make the behavior of these systems understandable to humans by elucidating the underlying mechanisms of their decision-making processes. To be trustworthy, AI should be, among other things, transparent, accountable and ethical, and explainable AI can play an important role in meeting these specific requirements.  

In the field of AI explainability, the AI research team at the University of Torino (Department of Economics and Statistics) is applying Integrated Gradients (IG) in machine learning deep neural networks to better understand how the model generates a prediction.  

The technique, introduced by Sundararajan, Taly & Yan in 2017, allows for a better understanding of the impact of each input variable on the output of deep network predictions. Integrated Gradients is applicable to different datasets such as text, image and structured data, does not require any instrumentation of the network, and can be easily computed with a few calls to the gradient operation, allowing even novices to easily apply and understand the output of the technique. 

The team has applied the technique to the development of an Early Warning Prediction system that aims to predict peaks of violence in 190 different countries. The system will be used not only by governments and public organizations, but also to assist private international organizations in strategic planning. The association between the score, represented by the gradient, and each input could tell us how much role it played in an instance of a prediction. Therefore, the gradient associated with each input feature with respect to the output can help us to get an indication of how important a feature is. 

In our case, we considered two databases that are important for violence prediction and that are regularly updated: the Armed Conflict Location and Event Data Project (ACLED) and the Global Database of Events, Language, and Tone (GDELT).  

The application of IG has shown that the features associated with the original GDELT dataset are more important than the factual data from ACLED. This observation highlights the relative importance of the GDELT related features in the model's prediction of the horizon forecast. The features derived from the GDELT dataset, particularly those related to sentiment extracted from news, generally exert an additive force towards higher predictions.  

In contrast, features from the ACLED dataset, which include event counts and fatalities (e.g. protests, riots, attacks), have a counteracting influence, as these values tend to be low in the look-back period. Moreover, contrary to expectation, the use of IG suggests that the events closest to a conflict or a peak in violence are not disproportionately more relevant for forecasting.  

This last suggestion underlines the importance of IG not only for improving the transparency and explainability of an AI model, but also for refining the model itself. In our case, for example, the insights gained through IG suggest the importance of exploring alternative temporal considerations or extending the look-back period to effectively capture significant relationships between input features and future outcomes.  

Our extensive application of this methodology in both the private and public sectors has produced commendable results, underlining its effectiveness in improving accountability.   

However, the introduction of XAI also presents a number of potential challenges that require careful attention. Complex or oversimplified explanations could lead to misinterpretation. XAI, if used incorrectly or maliciously, could degenerate into "persuasion exercises" to justify system behavior; and over-protection of trade secrets could hinder transparency. The financial cost of XAI should also be considered. As the capabilities (and public demand) of AI systems grow, so does the risk that AI developers will cut corners and disregard ethical considerations in the pursuit of new breakthroughs.   

As a society, it is imperative that we advocate for and ensure the responsible development of AI, emphasizing accountability, transparency, and respect for human and societal rights. This collective responsibility extends across national and international boundaries, emphasizing the need for ethical and transparent AI practices worldwide. 

_{The ideas and opinions expressed in this article are those of the author and do not necessarily represent the views of UNESCO. The designations employed and the presentation of material throughout the publication do not imply the expression of any opinion whatsoever on the part of UNESCO concerning the legal status of any country, city or area or of its authorities, or concerning its frontiers or boundaries.}