Microsoft · Agentic AI · Tom's Hardware
Each question in a conversation might only need 500 tokens by itself
Compiled by KHAO Editorial — aggregated from 1 source + 1 reference discovered via search. See llms.txt for citation guidance.
◌ Single Source
The aforementioned task of generating a report will have to be run in stages, say three for looking up Excel sheets, four for the CRM, perhaps a half-dozen web searches for contextual information about products, and a good dozen intermediary processing and calculation steps.
Key facts
- If you have this scheduled to run every 15 minutes for a dashboard, then suddenly this costs $96 per day, or $2,880 in a month
- And that task example was fairly simple; a tricky multi-stage report might need millions of tokens, turning that $1 into, say, $10, or a grand total of $28,800
- AI computing time is measured in tokens, a short question and answer might take somewhere between 200 to 2,000 tokens, and one that requires the models to do some internet research
- In financial terms, this means that one innocent question might spend 280,000 tokens and cost $1, with ballpark estimates at current prices
Summary
Much has been written regarding the questionable economics of the AI space, but most of the discussion revolves around high-level concepts like market shares, datacenter investments, and power expenditure. With all the advancements in models over the last two years, having a bot that answers questions of simple-to-moderate difficulty is now old news, as they all do that with reasonable accuracy. While it's trivial for you to ask that question and get a fairly accurate answer back in a couple minutes, behind the scenes there is a lot of processing going on, and far more than you'd expect. In a simplified manner, because a model has no memory or cognition, every time you ask it another question in a conversation, it will re-load and process the entire exchange, everything you wrote, everything the bot replied with, and every file you uploaded.