Should you trust a chatbot for credit card advice? New study finds AI leans on a handful of sites — and often picks the priciest cards

The study, conducted by New York-based AI communications firm 5W and titled the Credit Cards AI Visibility Index, tested out prompts that consumers might use when researching which credit card is best suited to their money goals. And, in response to thousands of queries, it found that the most-utilized AI platforms drew from a very high concentration of websites — and often recommended some of the most expensive options as a result.

Blatant provider advertising isn’t vastly skewing recommendations — but other forms of marketing might be

As the company noted, credit cards are the most aggressively promoted financial products in the country, soaking up $20 billion of bank marketing budgets each year. However, the investigation found that “when American consumers ask AI engines which card to apply for, the answer is not supplied by the issuers” — so, answers were not heavily influenced by direct advertising — the same small pool of resources was used again and again to inform the suggestions the chatbots provided.

The bulk of citations in AI responses (60-65%) came from a small collection of sites, including The Points Guy, NerdWallet, Bankrate, CardRatings, WalletHub and Forbes Advisor.

While ChatGPT and others treated these sites as “default sources of truth” on the subject, 5W says, it’s not because they are inherently the most reliable, but because of factors like the volume of posts they have about credit cards, how often they publish new posts and the style their posts take (such as comparing different credit cards side by side).

The study also points out that many of these sites earn commission when readers sign up for a credit card, and they might choose to highlight more expensive cards as a result. Most sites that engage in affiliate marketing, including Moneywise, have policies that state commissions do not affect cards reviews and rankings.

AI very likely to mention premium-fee cards and cite social media content

The data also showed that AI pointed to the priciest consumer cards more often than low or no-fee options; premium cards with fees over $400 per year were mentioned 5.7 times more than the alternatives.

Also interesting is that the second-largest bulk of citations came from the social media platform Reddit — an authentic source for testimonies, but not exactly an expert, nonpartisan or balanced one.

Further down the list of relevance, review aggregator sites comprised 8-12% of AI citations, while credit card issuers themselves made up 6%. Trusted regulatory bodies like Consumer Reports and the Federal Trade Commission made up only 2-3% of citations.

While the research is meant to help credit card companies better allocate their advertising efforts for such products, the results are also very informative for the consumer who assumes their favorite AI LLM can provide them with a truthful, well-informed and objective reply.