Last month, ProPublica published an investigation into America First Refining, a Texas oil refinery startup secretly funded by Donald Trump Jr. The investigation, led by reporter Justin Elliott, uncovered a complex story involving the Trump administration’s tariff policies, sanctioned Russian oil, and an Indian billionaire family’s private zoo. Central to the story was John Calce, the CEO of America First Refining, described as an obscure serial entrepreneur who struggled for years to secure funding for his refinery project.

Shortly before publication, the team examined another company incorporated by Calce, Brownsville Energy Storage Terminals. The company’s website claimed, “Our network connects the world’s most vital energy markets with speed, safety, and precision bulk oil storage.” When Elliott searched for an award the company claimed to have won, Google’s AI Overview stated, “Recent notable recipients include Brownsville Energy Storage Terminals, recognized for their rapid expansion in the independent oil and terminal operations sector.”

However, America First issued a letter demanding the immediate removal of unauthorized references to its office address from the Brownsville Energy Storage Terminals’ website, asserting no affiliation or connection between the two companies.

In April, The New York Times reported an analysis finding Google’s AI Overviews accurate about 9 out of 10 times, but noted this still resulted in tens of millions of erroneous answers every hour due to the vast volume of searches. Google acknowledged that AI Overviews can make mistakes, though a spokesperson criticized the study’s methodology.

Further concerns arise from the ease with which AI Overviews can be influenced. A recent headline in 404 Media stated, “It Is Trivially Easy to Use Reddit to Manipulate AI Search, Research Suggests,” underscoring vulnerabilities in AI-generated content.

This episode nearly caused a seasoned reporter to spiral amid conflicting and potentially manipulated AI information, illustrating the challenges journalists face when relying on AI tools for fact-checking and research.

Sources