Newer AI models overlooked more payment fraud in Coinbase’s benchmark
Compiled by KHAO Editorial — aggregated from 1 source. See llms.txt for citation guidance.
◌ Single Source
Coinbase reported Oct. 7 that newer versions of three major AI model families caught fewer fraudulent payments and a smaller share of fraud value in a historical test of payment screening for its Onramp service, despite an unchanged decision policy.
Key facts
- Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol)
- Separately, production measurements put median end-to-end LLM-request latency at 0.683 seconds versus 1.515 seconds for Opus 4.5, a 55% relative reduction
- The company’s evaluation replayed 16,140 transactions across 7,293 users, including 813 confirmed fraudulent transactions
- Its precision rose 11.5 percentage points, but recall fell 20.7 points and dollar-weighted recall fell 21.8 points
Summary
01 Coinbase’s historical replay found lower fraud-case and fraud-value detection across three model upgrades under a fixed decision policy. 02 GPT’s precision improved while fraud coverage declined, showing why one better screening metric can conceal a weaker overall result. 03 Separate custom-model and production-latency tests showed gains; neither measured customer losses from the tested model upgrades. The company’s evaluation replayed 16,140 transactions across 7,293 users, including 813 confirmed fraudulent transactions.