the precise timeline of how OpenAI fucked over the RAM market

[billsaysthis@curmudgeon.cafe]

@ariadne @davidgerard “Google publishes TurboQuant, a compression algorithm that reduces AI memory requirements by 6x with zero accuracy loss.”

This algorithm is somehow only applicable to AI??

[ariadne@social.treehouse.systems]

@BillSaysThis @davidgerard yes, it is possible to create domain-specific compression algorithms that are better than general ones.

[evoscale@c.im]

@davidgerard Since he's obviously alt Man, is he pro Caveman? Cuz we be headin' back there with his ilk in power.

[vorsos@beige.party]

@ariadne @BillSaysThis @davidgerard Really? I’ve been using pngcrush for audio files.

[demofox@mastodon.gamedev.place]

@BillSaysThis @ariadne @davidgerard if so, it's because they were doing something stupid and this fixes that IMO.

[davidgerard@circumstances.run]

@demofox @BillSaysThis @ariadne yeah I'd be slightly interested in the details, but also only slightly because (a) if it were applicable anywhere else we'd all know about it (b) we're far enough up and along the S curve i can see 6x the memory giving only a slight improvement. Maybe plain ML can benefit a lot, I dunno.

[phl@mastodon.social]

@davidgerard Fuck, and I say this without any reservation whatsoever, Sam Altman.

[gnuplusknoppers@troet.cafe]

@Vorsos @ariadne @BillSaysThis @davidgerard so?

[fritzadalis@infosec.exchange]

@davidgerard @ariadne
So glad memory is cheap again now. Wait, what?

[jnkrtech@social.treehouse.systems]

@davidgerard @demofox @BillSaysThis @ariadne my question is just whether this will make RAM less expensive. I’m guessing “no”, because that would be a good thing, and it seems increasingly likely that we can’t have those.

[ariadne@social.treehouse.systems]

@davidgerard @demofox @BillSaysThis @jnkrtech it did not

[jnkrtech@social.treehouse.systems]

@ariadne @davidgerard @demofox @BillSaysThis 🥲

[rogerb@mastodon.scot]

@davidgerard
Assuming he got a fixed price as part of the deal...
he can now sell them on and make a tidy profit, hence boosting OpenAI's numbers for the next investment round and/or going public.

[abucci@buc.ci]

@davidgerard@circumstances.run @demofox@mastodon.gamedev.place @BillSaysThis@curmudgeon.cafe @ariadne@treehouse.systems A couple points, bearing in mind that this is the first time I'm encountering TurboQuant and might be misspeaking:

• This is perhaps neither here nor there, but the X account making the originally-quoted post is https://www.aibyaakash.com , "AI by Aakash" (this is linked later in the same thread). The person seems fully AI-pilled and has several AI-themed substacks
• TurboQuant, or at least the QJL bit, sounds suspiciously like Locality-Sensitive Hashing. That's a well-known technique, and it can definitely do impressive things. When I tried my hand at startups I made heavy use of it (see https://bucci.onl/notes/Legit-tech ). In my use case I could get something like a 1,000-fold compression with acceptable accuracy loss. Basically LSH can be used to turn a long vector of floats into a comparatively short bitstring without losing too much of the geometrical information in the float vectors. Even one bit packs a ton of information
• The general problem of vector search that this method aims to address is an old one, and rotating or compressing the vectors is nothing new. In old school linear algebra things like diagonalization or SVD do this, for instance. I don't know if that's what they're doing but it's a general class of technique and a straightforward thing to try
• Vector quantization is, of course, also quite old. You experience it every time you listen to an MP3.

So, it's possible this is a characteristic Google move of taking existing science, ramming it through their engineering machine, and suggesting novelty with a clever title, headline, and/or new name. Which is not to suggest it's a bad piece of engineering. I couldn't say. However, it's possible this is a Google rebrand, and the questions raised in this thread, like "wouldn't we already know about this? wouldn't it be applied outside of AI?" are answered by: yes, we did already know about this and yes, it has already been applied outside of AI. Oh and yes, it'd be quite silly if nobody thought to try these old school techniques in the latest incarnation of LLM-based AI before 2026.

[reflex@retrogaming.social]

@davidgerard @demofox @BillSaysThis @ariadne UFD Tech discussed it the other day and it only applies to a very specific aspect of AI resulting in a tiny overall shrink in memory consumption that's being used to load slightly larger models. And it started being used middle of last year, meaning it's already baked in.

[qgustavor@urusai.social]

@Vorsos @ariadne @BillSaysThis @davidgerard Reminds me of when I took a bunch of manga PNG, converted then to BMP and compressed all back using 7z and the resulting file was smaller than compressing the original PNGs using 7z

[cppguy@infosec.space]

@Vorsos

I can't tell if you're serious, but Ariadne is right. Simple example: Flac will losslessly compress audio better than zip or gzip will. That's why it was invented.

@ariadne @BillSaysThis @davidgerard

[katlin@mastodon.social]

@CppGuy @Vorsos @ariadne @BillSaysThis @davidgerard

Interestingly enough, Chinchilla 70B was trained mostly on text and beat domain-specific compressors PNG and FLAC in one experiment.

https://arxiv.org/abs/2309.10668

Not saying you are wrong. I assume that newer, domain-specific algorithms would still outperform the general Chinchilla algorithm, and there can be practical downsides if they involve large memory requirements, even if they result in more efficient compression.

[djgummikuh@mastodon.social]

@davidgerard I wonder though, if the demand side is collapsing this quickly, why isn't the price following? "Analysts expect elevated prices until 2028" are they lying? Trying to protect their investment? Or is there more at play than Altmans eccentrism?