India’s linguistic diversity challenges digital content accuracy, a problem Google now faces with its new AI Overviews
In August, the organization implemented Hindi support for AI Overviews nationwide. Users in India can switch between Hindi and English without exiting the search page. However, the feature is inconsistent and disappointing.
A straightforward example was a result that we selected from Google’s blog. The GIF displayed an answer to the question “Cheeni ki jagah chai mai kya daal sakte hai?” which translates to “What is a substitute for sugar in tea?”
During testing, we altered the order of the words to “Chai mai cheeni ke jagah kya daal sakte hai?” and Google failed to provide an answer.
Another issue that we identified was the literal translation of English terms. When we inquired in Hindi, “What type of food can we consume during the summer?” One of the responses was “Chiknai wali cheezien,” which translates to “Sticky things.”
This is a cause for confusion. When I transitioned to English, Google presented me with the option of “Oily,” which is significantly distinct from “sticky” in cuisine. In any case, it is a peculiar recommendation.
In another instance, AI Overviews provided an inaccurate response to our inquiry regarding YouTube’s ownership: “Neal Mohan was Google’s CEO until 16 February 202February 16. Neal Mohan is the CEO of Google as of February 16, 20 February 16cated, in the English text.
Additionally, there are qualitative issues. Upon inquiring about the date of Diwali this year, we were met with a paragraph detailing the celebration of Diwali the previous year, followed by a carousel of links, and finally, the response we sought.
We received inconsistent responses from AI Overview when we inquired about the possibility of consuming cuisine with spices in both Hindi and English on multiple occasions.
Nevertheless, there are more alarming findings regarding other subjects, such as pregnancy and menstruation. The first paragraph of the overview response to one of our inquiries, which asked when individuals should consider having children after marriage, stated that couples should wait for at least two years. This information was presented in a perplexing Hindi language.
It also stated that individuals who marry around 25 have “three years” to prepare for their marriage without providing any further details.
This paragraph appears derived from a Hindi slideshow article on an Indian news website. The article was based on various opinions expressed on Quora and other websites.
When we inquired about the appropriate foods to consume during menstruation, some of the responses recommended “drinking milk with many things” and lemon for mood swings.
Google’s AI Overview also lacks context. When we inquired about food options in Delhi, it was recommended that we visit Bangla Sahib Gurudwara, a Sikh place of worship open 24 hours a day.
Visitors can purchase tea and Indian delicacies such as Samosa and Kachori. Nevertheless, the AI tool neglected to include a crucial detail that the source article did:
You could acquire these products outside of the Gurudwara, a term the AI tool overlooked during the summarization process. The snacks are not located at the site of worship.
The results of specific comparable queries in English are significantly superior to those in Hindi. This could be partially attributed to the availability of more and superior sources in that language.
However, some issues we identified resulted from Google’s errors, including inaccurate translations and out-of-context summaries. Google has addressed search and language issues in India, and it is anticipated that its initial efforts should have resulted in a more favorable outcome.
Google informed TechCrunch that its AI Overviews are only displayed for queries with a high level of confidence in the output quality. The company stated that the feature is “rooted in our core search quality systems” and will only display information supported by the top web results.
“Our tests indicate that the accuracy rate of AI Overviews is comparable to that of other features, such as Featured Snippets.”
A Google spokesperson informed TechCrunch that issues may result from our systems misinterpreting web content or reflecting inaccuracies on the web. “We use these examples to improve, as we do with all Search features,” the spokesperson stated.
The caliber of the sources that Google refers to may be the cause. Nevertheless, the query and sources were valuable enough for AI to summarize using Google’s algorithm. Not all users will examine the content on source sites and instead criticize Google for providing inaccurate or unclear responses.
Google has been disappointed with its AI Overviews initiative. The company received significant criticism earlier this year for publishing responses from Reddit that advised a user to incorporate adhesive into their pizza.
An answer sourced from The Onion was provided to another user by AI Overviews, who advised them to consume “one small rock per day.” The majority of the Hindi examples we evaluated did not provide accurate information regarding the subject matter.
There are over 830 million internet consumers in India, and many of them utilize Google for their website searches. The organization must enhance its language and content accuracy to increase the number of individuals who utilize AI Overviews in local languages.
