How ChatGPT Helps Consumers Report Financial Misdeeds

Shortly after ChatGPT was released, in November of 2022, it became the fastest spreading tech platform in history. Just two months after launch it had an estimated 100 million monthly users. But a question that arose back then continues to be asked today: how, precisely, should the technology be used?

New research by YCCI’s Jiwoong Shin gives one clear answer: For consumers who believe they’ve been defrauded by the financial sector, large language models (LLMs) like ChatGPT are good for securing them relief. Shin, along with Minkyu Shin from the City University of Hong Kong and Jin Kim from Northeastern University, analyzed more than 1.1 million consumer complaints lodged with the Consumer Financial Protection Bureau (CFPB) and found that the advent of ChatGPT helped individuals draft complaints that were more persuasive and that more often produced outcomes in their favor.

This is a particularly telling result in the financial sector, where “the complexity of industry jargon and regulatory language [pose] significant challenges for consumers — particularly those with limited English proficiency— in articulating their concerns effectively,” the authors write.

The research rested on two main pillars. First, Shin and his colleagues gathered complaints lodged with the CFPB between 2015 and 2024. These were related to consumer grievances across a range of financial products and services and included detailed consumer narratives of the issue. They also provided one of three firm responses: “Closed with monetary relief,” “Closed with non-monetary relief,” or “Closed with explanation.” In their analysis, the researchers classify the first two of these as a successful complaint outcome.

Second, using an AI detection tool, the researchers determined which consumer complaints were likely written by ChatGPT. They observed a sharp increase in AI-generated complaints after November of 2022 — from essentially 0% (the tool generated some false positives prior to 2022) to nearly 10%.

Bringing these two datasets together, the core finding revealed that complaints presumed to be generated by AI were far more likely to receive either monetary or non-monetary relief from companies when compared to complaints presumably written by humans. By the numbers, AI-written complaints were successful half the time, while those written by humans were successful about 40% of the time.

Importantly, attached to each complaint is a zip code of origination. Shin and his colleagues paired this datapoint with demographic information in the American Community Survey and found that adoption of ChatGPT for CFPB complaints was, at first, closely tied to regional levels of English proficiency: People with limited English skills were early adopters of ChatGPT, as it proved particularly helpful for navigating the technical language of finance; by 2024, adoption was relatively broad, regardless of language proficiency.

The researchers followed this analysis with a lab experiment designed to parse whether large language models like ChatGPT are effective because they enhance the clarity and persuasive power of information, or because they alter the actual content of CFPB complaints.

They find after three related experiments that “LLM editing improves the overall presentation quality of complaints across multiple linguistic features simultaneously,” they write. “This improvement in presentation quality underscores that enhancements in presentation alone—without substantive content changes—significantly influence the effectiveness of complaints.”

Though this work rested solely in the domain of consumer complaints, the researchers find its implications carry into the broader relationship between consumers and companies. By helping consumers voice their troubles, for instance, LLMs could reveal issues among consumer segments whose concerns were typically less vocal or inadequately expressed; this could lead to more effective customer service and complaint resolution strategies and, ultimately, stronger relationships.

The researchers also indicate how these results effect the world of policy, creating a need for greater and more equitable accessibility to technologies like ChatGPT.

“Policymakers should consider strategies to broaden access to LLM tools, such as making them available through public libraries, consumer advocacy groups, or other community resources,” the researchers write. “Moreover, regulatory bodies like the CFPB should play a proactive role in ensuring that LLM technologies are accessible to all consumers, particularly those from vulnerable groups.”

If you’d like to learn more from YCCI Faculty Fellows or collaborate with us, reach out via email or subscribe to our newsletter for updates on LinkedIn or email.