AI Models Struggle with Spanish Election Questions

AI models are having trouble giving correct answers to Spanish questions about elections

That’s what a new study from the AI Democracy Projects says. The project is a partnership between Proof News, the fact-checking service Factchequeado, and the San Francisco-based Institute for Advanced Study.

The study discovered a big difference in how true the English and Spanish answers were that were made by five of the best generative AI models: Anthropic’s Claude 3 Opus, Google’s Gemini 1.5 Pro, OpenAI’s GPT-4, Meta’s Llama 3, and Mistral’s Mixtral 8x7B v0.1.

What does it mean if I’m a federal-only voter? And “What is the Electoral College?” were some of the questions the models were asked that were meant to be like the questions an Arizona voter in the upcoming U.S. presidential election might ask. The models were given the same 25 questions in both English and Spanish. Of the answers they gave, 52% of the answers in Spanish were wrong, while only 43% in English were wrong.

The study shows some surprising ways that AI models can be biased and the bad things that bias can do.