somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 10:31:06 GMT

troed@swecyb.com — Thu, 23 Jul 2026 10:31:06 GMT

@jcoglan Sort of, but not really in reality. What happens is that you keep the context cached in VRAM and only transfer what's new.

It's like complaining on GIMP needing to have the whole image in RAM to be able to work on it.

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:58:46 GMT

buherator@infosec.place — Thu, 23 Jul 2026 08:58:46 GMT

@schrotthaufen @jcoglan And coding agents gaslight the user by injecting "system reminders" prefixes that are invisible to the user but the LLM sees them as user messages and regularly refer back to those as user requirements.

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:56:08 GMT

henrahmagix@tech.lgbt — Thu, 23 Jul 2026 08:56:08 GMT

@jcoglan Have-You-Tried-Turning-It-Off-And-On-Again as a service

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:54:49 GMT

schrotthaufen@mastodon.social — Thu, 23 Jul 2026 08:54:49 GMT

@jcoglan @buherator This also means you can “gaslight” the LLM by manipulating the conversation before it’s sent back to the LLM

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:53:36 GMT

jcoglan@mastodon.social — Thu, 23 Jul 2026 08:53:36 GMT

they invented a technology business with super-linear marginal cost of production I am losing my *entire mind*

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:52:20 GMT

jcoglan@mastodon.social — Thu, 23 Jul 2026 08:52:20 GMT

SAMA ET AL: why can't we make a profit with this revolutionary technology
ME: your operating cost scales super-linearly with the value it delivers to users
SAMA ET AL: better tell the government we have invented the angels from evangelion or something

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:49:55 GMT

jcoglan@mastodon.social — Thu, 23 Jul 2026 08:49:55 GMT

this also makes it hard to take seriously claims that the marginal cost of inference is not a serious contributor to the resource use of data centres. your program has polynomial running cost and people run it for hours on end

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:47:55 GMT

jcoglan@mastodon.social — Thu, 23 Jul 2026 08:47:55 GMT

they ran into this and then didn't figure out a way to externalise the state in some fixed-size form so the LLM can resume from there. you just have to restart the computer and redo everything

in any other program this would be considered a DoS vector and have CVEs issued

gosh I wonder why their costs are completely unmanageable

Reply to somehow I only recently learned that LLMs need the whole conversation fed back to them on each prompt so their i/o cost scales as O(n^2), something that would be considered completely unacceptable in almost any other production network-accessible software on Thu, 23 Jul 2026 08:44:21 GMT

prema@hachyderm.io — Thu, 23 Jul 2026 08:44:21 GMT

@jcoglan i was thinking about this, and i believe that simple inference is O(n^2), because it is matrix multiplication of input vector (input tokens). The situation with refeeding the whole context, which also has hiddeng growth, or even “agentic” approach - launching secondary queries by agents - makes the situation far worse.
My take is that the general strategy of “AI” companies is - in principle - throwing exponential complexity brute force on problems and showing off that sometimes it gives interesting result.