Deep learning’s progress over the last decade has heightened interest in the subject of artificial intelligence. However, the growing popularity of AI has exposed some of the field’s fundamental issues, such as the “black box problem,” which is the challenge of understanding how complicated machine learning algorithms make choices.

Because of the growing focus on black-box, a corpus of research on explainable AI has emerged. And much of the work done in the field includes creating approaches that attempt to explain a machine learning algorithm’s conclusion without cracking open the black box.

Two types of Machine Learning black boxes

The black-box problem, like many other aspects of artificial intelligence, is fraught with ambiguity. There are two sorts of black-box AI systems:

  • functionalities that are too complex for any human to grasp
  • proprietary functions.

Deep neural networks, the architecture used in dl algorithms, are an example of the first type of black-box AI. DNNs are made up of layers upon layers of linked variables, which are adjusted as the network is trained on a large number of instances. As neural networks expand in size, it becomes increasingly difficult to track how their millions (and even billions) of parameters interact to produce choices. Even if AI engineers have access to such parameters, they will be unable to accurately dissect the neural network’s judgments.

The second form of black-box AI, proprietary algorithms, refers to firms who conceal the specifics of their AI systems for a variety of reasons, including intellectual property protection or preventing criminal actors from manipulating the system. In this instance, the individuals who built the AI system may be aware of its inner logic, but the people who utilize it are not. Every day, we engage with a variety of black-box AI systems, such as Facebook news, Amazon system for recommendation and Google Search’s ranking algorithm. The most hazardous ones, though, are those used to decide prison sentences, credit ratings, and hospital treatment decisions.

Accuracy and Interpretability

According to a widely held opinion in the AI field, there is a tradeoff between accuracy and interpretability: Deep neural networks, for example, give flexibility and accuracy that other types of machine learning algorithms lack at the price of being uninterpretable.

However, this is very dependent on the issue domain, the type of data provided, and the intended solutions.

In certain situations, the interpretability given by a simpler machine learning model outweighs the minimal performance obtained by using a black-box AI system.

This is especially true in crucial fields like medicine, where clinicians must understand the reasoning behind an AI-generated choice and apply their own thoughts and opinions to it.

Part of the problem arises from a culture that has infected the AI community in the aftermath of deep learning’s growth in popularity. Many academics are leaning toward the “more is better” approach, in which it is hoped that larger deep learning models with more layers and parameters trained on larger data sets would result in artificial intelligence advances. This has resulted in the widespread use of deep learning in fields where interpretable AI approaches may produce similarly accurate results.

Problems with black box AI systems

Many firms conceal the features of their AI systems for financial reasons, such as maintaining a competitive advantage. However, although this business model increases the profit of the firm building the AI system, it does nothing to reduce the pain and damage it does to the end-user, such as a prisoner receiving an overly long sentence or a needy individual being denied a loan.

This tendency is particularly concerning in industries like banking, health care, and criminal justice. There is already a large amount of work and study on algorithmic bias and AI systems that discriminate against specific populations.

However, since algorithms are kept behind closed doors and only accessible to their creators, there is limited chance for an unbiased inquiry into their inner workings, and most researchers must rely on faulty black-box explanation techniques that connect inputs to outputs.

Another common justification given by tech firms for defending black-box AI systems is to keep hostile actors from reverse-engineering and exploiting their algorithms.

This is a method that is being used in various areas of software engineering. For example, in security, open-source and transparency are rapidly replacing the “security by obscurity” ethos, in which corporations assume that concealing the specifics of their program would keep them secure.

There’s no reason why the AI community shouldn’t use the same approach.