We are living in a time where algorithms increasingly aid human decision-making. I’d like to share a couple of studies that provide unique insights into human-AI interaction, exploring how we trust or mistrust machines and our biases towards algorithms.
Berkeley J. Dietvorst, Joseph P. Simmons, and Cade Massey, a research team from the University of Pennsylvania, conducted five experiments to explore whether errors made by algorithms decrease people’s confidence in them, making them less likely to choose algorithmic decisions over those made by an inexperienced individual.
In the studies, participants were asked to make forecasts about certain scenarios, such as predicting MBA student performance or ranking states by number of airline passengers. Participants were then randomly assigned to various conditions where they either saw the forecasts made by an AI model, made their own forecasts, saw both, or neither.
The primary measure of interest was the choice participants made regarding whose predictions they would financially bet on – the AI’s forecasts or those made by a human (either their own or another participant’s). In essence, participants’ choices reflected their trust levels: tying their potential earnings to the AI’s predictions indicated greater trust in the AI, while basing their potential earnings on a human’s forecasts suggested more confidence in human judgment.
In this study, the model used was a regression algorithm. It was fed historical data, which it used to analyze patterns and make future predictions. For example, the algorithm could be given past data about a student’s GPA and extracurricular activities to predict their future grades.
The results were staggering: across the five studies, the algorithm made 15-29% fewer errors on average than human forecasters in the MBA prediction task. In the airline passenger ranking task, the algorithm made 90-97% fewer errors than humans. On average, the algorithm earned $0.17 to $0.68 more in bonus money than humans across the studies. The algorithm’s forecasts were much more highly correlated with true outcomes than humans’ forecasts.
Even though these algorithms consistently outperformed human forecasts, participants were less likely to trust the algorithm over a human, especially after seeing it make some mistakes. This observation led to the term ‘algorithm aversion,’ which describes a phenomenon where an algorithm’s mistakes significantly reduce confidence in it, favoring a human who may be less accurate but is perceived as more trustworthy.
While the costs of making a wrong prediction for MBA grades isn’t life threatening, let’s shift our attention to the medical field where it can be. Another study by an MIT economics professor examined how radiologists interact with AI when making diagnostic assessments from chest x-rays. In this study, radiologists diagnosed 324 retrospective patient cases under different conditions: x-ray only, x-ray with clinical history, x-ray with AI prediction, and x-ray with both.
The AI in this scenario proved more accurate than about two-thirds of the radiologists, yet the average diagnostic accuracy of the radiologist did not improve while using AI predictions. This was in stark contrast when the radiologists were provided with clinical history alongside the x-rays, where their accuracy improved. This suggests that radiologists have additional contextual information that the AI does not possess and gave them an advantage.
Interestingly, when provided access to AI predictions, radiologists did update their diagnosis, but not fully. Their diagnosis process took longer with AI assistance, which could undermine the potential benefits of having AI as a tool to improve their work. It was hard for them to trust right away. The study suggests that radiologists tended to underweight AI predictions.
During my years building and researching AI products, even conversation with customers, I’ve often noticed individuals in various fields, including the music industry, displaying bias against algorithmic predictions. I believe this hesitation stems from a lack of clear understanding about what AI is and how it was trained. Today we use tools to explain why an algorithm made specific prediction, but at first they weren’t intuitive to grasp.
There are many reasons we may distrust algorithms such as fear of losing control, the opaque nature of algorithms, or overconfidence in human abilities. Also, concerns about accountability when things go wrong also play a significant role in this aversion. The unclear line of responsibility when an algorithm makes a mistake can lead to discomfort and mistrust.
These studies serve as a reminder of our inherent biases and the powerful influence they can have, even in the face of clear algorithmic accuracy compared to humans. As we continue to delve into the potential of AI, it’s critical that we also focus on understanding and addressing these biases.
My takeaway for us working in music and AI is that we live in a hyper-competitive industry. Any personalized information available to give us an advantage, without harm to society, is worth using.
Unbias is building an AI product with marketing capabilities. Join our waitlist here to get early access.
