We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
Those benchmarks are easily gamed in my opinion. I take them woth a huge grain of salt.
I am currently building an integrated CAD + Aero + CAM software package for rapid UAV design iteration, and the open models are all not very good at it. They make a ton of mistakes and I have to end up redoing a lot. And that's using very detailed specs and TDDs provided as context to the models. Opus/Astra/Argon/Gemini Flash 3.8/Sol does a significantly better job in my experience.
Most of the look of the thing is about the built in tools and skills and harness settings, not the quality of the code. those things can be replicated locally as well.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
Why 2 5090s instead of just 1 6000? That's 64GB VRAM vs 96GB VRAM. Perfomance is theoretically worse on the 6000 vs 2x 5090s, but given the overhead of tensor parallelism and how slow PCIE is, I imagine it's pretty close for less than half the power draw and heat.
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I use the Qwen models a lot in my private life and they're not quite on par with the SOTA models, but they've definitely gotten to the point where they're usable. Particularly if you invest a lot in harness engineering, they become very usable. It'll be even better when Qwen4 comes out, and less harness engineering will be needed for the same results, and an equivalent amount of harness engineering tuned for the updated models weaknesses will produce results much closer to current SOTA. I was also pretty stunned to see Unsloth's IQ1_M quants of Deepseek v4.1 Flash 1017 held up too. Even that has become runnable locally. It's truly a new era.
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
I replaced my 11 year old PC earlier this year, with a new 5070 on discount and only 16GB RAM. I thought it was expensive. Now prices surged like 40% since then and I regreted not getting more RAM.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
I see them at a little less than 400€ in the second hand market. So expensive for their age but not hard to replace if you were willing to pay 600-700 for it in 2020.
That's an 8GB card. For gaming (DLSS, etc.), you can still get newer 8GB cards like the 3070 which represent solid upgrades over the 1070 Ti on the used market for under $300. Still not great since similar cards were available at similar prices new a couple years ago, but at least not trading at multiples of MSRP like higher-end cards.
Even 8GB cards might start going up. I recently bought a cheap one so I can auxiliary models on it (think smaller LLMs for memory consolidation / decision models etc). Locally, multi-gpu is pretty cool. You can essentially have one GPU drive the others and do increasingly powerful things when you combine them.
You should bite the bullet now and buy a decent card on the secondhand market, before more people figure out what they have and double, even triple, the price.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
I think the discrete GPUs as a concept is coming to an end. Computer graphics is shifting more and more to using local AI-like things like DLSS that allows to hard-render simpler geometry and shadows and then apply AI-realism filters in real-time.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
How would DLSS, which runs on GPU hardware — and specifically, as I understand it, GPU hardware that accelerates AI inference — eliminate the GPU?
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
DLSS does not require any specific packaging configuration. There's no reason DLSS can't run on DGX Spark or GPU cores of the Apple's M-chips aside from specific Nvidia software optimizations that can surely be worked out if needed.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
DLSS still needs discrete hardware to do that (and keeps requiring new hardware on each generation). Not to mention the types of RAM are not the same and have entirely different tradeoffs - using GDDR for a CPU would tank performance due to an order of magnitude worse latency, using DDR for a GPU would make games a slideshow due to an order of magnitude worse bandwidth. And you'd have to redesign the entire concept of how CPUs and motherboards work to get there, while making AMD and NVIDIA agree to have their chips be behest to the bridges of shitty motherboard and consumer RAM manufacturers even more than they already are.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
> DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
Anyone has an idea when will it unclog even a little bit? When is the next NVIDIA lineup comming out? Are they going to release any middle of the road products like strong AMD Insights?
I just did a cost breakdown of owning a GPU vs paying for tokens.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
If you’re returning a product you used because you wanted to pretend it was a rental at the seller’s expense, bluntly, you’re as much the problem as the seller.
Everyone in the heirachy above you is an exploitative business psychopath. Might as well exploit their calculated greed as they attempt to exploit your empathy.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
I don't understand, your subscription models can also use Playwright, you don't even need to setup an mcp, they'll automatically download playwright-cli and get going.
Yes, but even the smaller models, when used within a harness and orchestrator, can be accurate enough for real work, especially when combined or validated by other smaller models.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
Yeah I did something similar but more extensive, because not every model is same plus depend on your options how much you want to degrade model from fp8. Some models is just slow on local hardware and not practical really.
- rate at which tokens are generated
- quality of generated tokens
- ethics of abusing return policies
- long-term viability of abusing return policies
I remember when the top GPU was 550€ and the overclocked, custom-cased, water-cooling ready version cost 660€ tops. Any GPU price out there even $800 for 5060Ti from january 2025 is not only out of any sane band, but look as if we're in a ultra-recession, yet other markets aren't affected nearly as extreme.
I bought two 2TB SSDs from TeamGroup last year, about $100/ssd. I put them side by side in a raid setup for my home server. (I wanted raid, and they were less than half the price of Samsung, etc.). I'm on a budget, so I went with the cheap ones and raid reduduncy vs one expensive one with no reduduncy.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
You're comparing running a heavily quantized 10-30B model to running a multi trillion one. Being half the price (unless you expect to only need AI 3 months in your life) is absolutely not a good deal, here.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
You're comparing apples to oranges. 109 million tokens is ~nothing, easily a day's worth of casual use. Compare price per token and cloud providers are probably much closer. But even that isn't a fair comparison between cloud AI providers and abusing retailer return policies just yet, because usable local models even remotely comparable to the frontier don't fit in 16gb of vram.
Yeah, and Qwen 3B that you run on 16GB of VRAM and 10K context length surely is comparable to whatever subscription you're comparing to, that also gives access to 3B parameter models? You might want to take actual use cases into account before you try this scam in real life.
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
Thankfully, the environment we are all forced to live in, doesn't care what we think about it, and adapting to it accurately at minimal expense is the difference between surviving and thriving.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
> What scam are you speaking of ? The return window ?
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Firstly you are assuming $0.048$/hr for GPU, which is highly misleading and you can easily sell it for double(vast.ai is $0.12) as you are forgetting lot of cost like CPU, cooling etc.
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.
We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models.
Local usage for end-users is unmetered and token speeds are very fast and keep improving every few weeks due to optimizations being done in the community.
Data remains local which makes EU compliance checkboxes way easier since data never touches anything related to the CLOUD act.
> The performance of local AI for coding, given correct setup & harness, is very good and on par for 95% of use cases compared to closed models
While I don't know your specific use cases, when interpreted generally, this seems trivially disproven by putting eg. websites made by Opus 5.5 and the local model side by side.
I have merely a laptop 5090 (24 GB) and I can run Qwen3.8 Flash IQ3_XXS at 110-140 tok/s on Strata, 128K context. If Artificial Analysis' benchmarks mean anything, it sits somewhere around GPT 5.6 Terra/Sol. I use both that and Codex/Sol 6.1 most days of the week, for different projects. I don't feel like I'm particularly compromised when working locally - I very rarely think "ah man I wish I could just point Codex at this". The biggest problem is speed - it needs more reasoning time and I can only work single-threaded.
Those benchmarks are easily gamed in my opinion. I take them woth a huge grain of salt.
I am currently building an integrated CAD + Aero + CAM software package for rapid UAV design iteration, and the open models are all not very good at it. They make a ton of mistakes and I have to end up redoing a lot. And that's using very detailed specs and TDDs provided as context to the models. Opus/Astra/Argon/Gemini Flash 3.8/Sol does a significantly better job in my experience.
Have you tried basalt? I understand that it allows multiple concurrent requests
Imagine thinking that the look of a website is the right metric to judge the performance of AI models.
If your job is to produce websites that look good it's an excellent way to judge performance of AI models _for your workflow_.
What's the problem?
Most of the look of the thing is about the built in tools and skills and harness settings, not the quality of the code. those things can be replicated locally as well.
The problem is you can change the prompt a little and get correct results on a local AI too.
Frontend has been solved for years.
I don't think changing the prompt alone is going to do it, but iteratively providing feedback can get you there.
With GPT-5.4 or GPT-5.5, or of course also by writing the code by hand if we want to go all the way, it was well possible to build nice frontends to our liking.
I still don't want to go back when I can do it better and faster with Opus 5.5.
It is one metric to judge them. And if you do frontend stuff, it's probably a fairly relevant one.
While I don't doubt your efficacy, the cost of those machines today, especially considering the high cost of electricity in Europe, would be uneconomical versus buying subscriptions. The reason to do this setup is data sovereignty, and there are plenty of cases where this is required.
Why 2 5090s instead of just 1 6000? That's 64GB VRAM vs 96GB VRAM. Perfomance is theoretically worse on the 6000 vs 2x 5090s, but given the overhead of tensor parallelism and how slow PCIE is, I imagine it's pretty close for less than half the power draw and heat.
>> given correct setup & harness
I am geninuely interested what stack you are using here. Could you please elaborate?
We are continuously improving the workflow and things improve almost on a weekly basis. Current setup is custom Gitlab project structure + OpenCode + Mattpocock/skills followed by a well defined align/plan/build/review/learn engineering process.
It feels like things changed in the past few months. We are now truly able to do what we need to do without Claude etc, where this was not feasible earlier this year.
I use the Qwen models a lot in my private life and they're not quite on par with the SOTA models, but they've definitely gotten to the point where they're usable. Particularly if you invest a lot in harness engineering, they become very usable. It'll be even better when Qwen4 comes out, and less harness engineering will be needed for the same results, and an equivalent amount of harness engineering tuned for the updated models weaknesses will produce results much closer to current SOTA. I was also pretty stunned to see Unsloth's IQ1_M quants of Deepseek v4.1 Flash 1017 held up too. Even that has become runnable locally. It's truly a new era.
https://unsloth.ai/docs/models/deepseek-v4.1-flash
> We configured 6 machines each having 2x5090 for local Qwen3.8 in the past months. Prices are very high.
Who is "we"?
As someone whose gaming PC is now over six years old, I am very nervous that my GPU is going to crap out on me, because I won't be able to afford to replace it.
I replaced my 11 year old PC earlier this year, with a new 5070 on discount and only 16GB RAM. I thought it was expensive. Now prices surged like 40% since then and I regreted not getting more RAM.
<8GB gpus and below can be found at decent price. While the price are higher than they used to be a year ago for older / lower end levels it is not crazier than the price during the covid/crypto miner period for example which is probably when you built that pc.
My current GPU is a 10 GB RTX 3080. To replace it with something that isn't a downgrade, I would have to shell out at least $1,000 CAD.
I see them at a little less than 400€ in the second hand market. So expensive for their age but not hard to replace if you were willing to pay 600-700 for it in 2020.
Same boat here. I'm stuck with my 1070ti.
That's an 8GB card. For gaming (DLSS, etc.), you can still get newer 8GB cards like the 3070 which represent solid upgrades over the 1070 Ti on the used market for under $300. Still not great since similar cards were available at similar prices new a couple years ago, but at least not trading at multiples of MSRP like higher-end cards.
Even 8GB cards might start going up. I recently bought a cheap one so I can auxiliary models on it (think smaller LLMs for memory consolidation / decision models etc). Locally, multi-gpu is pretty cool. You can essentially have one GPU drive the others and do increasingly powerful things when you combine them.
You should bite the bullet now and buy a decent card on the secondhand market, before more people figure out what they have and double, even triple, the price.
The 5090 is a firecracker of a GPU. RAM of course is a part of this; but it's also just... a really good card. I bought it ostensibly for gaming and light AI, but now I use it for Qwen 3.8 27B running in ninfer and in (shameless plug) dreamcoder.ai it absolutely flies. I get ~2-3k per second prefill (important for large contexts), 256k context and ~150 tok/s.
I have 30 years of coding experience. That tiny model is no Opus 5.5, but make no mistake: it's a very strong coder in the hands of a capable developer.
Wrote a couple of days ago: I decided to check out what a top gaming PC would cost. $12.5k! Absolutely nuts. I have gamer friends that make bank, the types that have always bought the latest flagship parts at launch, and even they slowing down, because they feel the prices are extortionate.
We're back to early 90s prices.
EDIT: Yes, obviously you don't need a RTX5090 to play games. But it seems to me like the top GPU market is now exclusively for the local LLM users.
I lament not splurging for the 5090 back when I built my machine, I figured it would only increase in value (even used) but it was hard to swallow the price at release which was already expensive and way above MSRP.
Moore Threads Lushan GPU will be out by the end of this year with 64GB RAM.
Will this be available in the US? It looks interesting.
I think the discrete GPUs as a concept is coming to an end. Computer graphics is shifting more and more to using local AI-like things like DLSS that allows to hard-render simpler geometry and shadows and then apply AI-realism filters in real-time.
It's harder and harder to justify gaming PCs having separate pools of GPU-only memory and "everything else" RAM.
I think unified memory is the obvious next step for personal computing and vendors will be coming up with solutions that enable more modular approach to building enthusiast grade upgradable personal computers.
How would DLSS, which runs on GPU hardware — and specifically, as I understand it, GPU hardware that accelerates AI inference — eliminate the GPU?
And if a system is intended to be modular and upgradable, what would combining the GPU and CPU into a single module contribute to this goal?
Now an architecture that separates the memory controller from the CPU and gives pluggable CPUs and GPUs (and storage and network controllers, for that matter) equal access to a single pool of RAM in a secure and high-performance way, that would certainly be interesting, and not only or even primarily for gaming.
DLSS does not require any specific packaging configuration. There's no reason DLSS can't run on DGX Spark or GPU cores of the Apple's M-chips aside from specific Nvidia software optimizations that can surely be worked out if needed.
I imagine the PCs for the future would maybe have motherboard with two sockets: a normal CPU one and a GPU one, both having access to some high bandwidth unified memory that.
Right now the unified memory needs to be pretty much soldered super close to the chip to ensure the bandwidth and energy efficiency, but hopefully there will be some solution to circumvent this limitation in the future.
Putting CPU and GPU cores into the same physical die works for your average consumer machines, but hardcore professionals or enthusiasts would always strive for more custom configurations than existing packages can provide, like more GPU cores.
So future PC go back to having GPU on motherboard ?
I think so, yeah. Clearly having separate RAM and GPU memory where memory becoming the main productivity bottle neck is not practical or sustainable.
DLSS still needs discrete hardware to do that (and keeps requiring new hardware on each generation). Not to mention the types of RAM are not the same and have entirely different tradeoffs - using GDDR for a CPU would tank performance due to an order of magnitude worse latency, using DDR for a GPU would make games a slideshow due to an order of magnitude worse bandwidth. And you'd have to redesign the entire concept of how CPUs and motherboards work to get there, while making AMD and NVIDIA agree to have their chips be behest to the bridges of shitty motherboard and consumer RAM manufacturers even more than they already are.
Integrated systems have come a long way for highly optimized mobile and laptops designs with specific hardware in mind. But even if everything went perfectly - DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
Then again, maybe AGI will come and hardware will fundamentally revolutionize overnight.
> DLSS 5 on a 5090 halves the frame rate and requires a 5090. And it can only do a single style - gritty, high contrast lighting photorealism.
The technology is very new and currently in its infancy. I do believe it's the future of computer graphics though, where instead of doing "real" photorealistic geometry and lighting, software can render a very simplified scene and have a good enough photo-realistic filter applied to it so that most people wouldn't be able to tell the difference.
Obviously, I believe it'll be flexible enough to do things other than photo-realism in the future.
> In 10 years will new GPUs primarily be AIPUs? Sure, probably, but they'll still be discrete and they'll still have significant raster and ray tracing pipelines to provide the basis of rendering.
I think we have different definitions of "discrete" here. In my mind AIPUs are the opposite of discrete. I might be wrong here, but I always interpreted this term as a standalone GPU unit with its own onboard memory — a thing that currently fits into a PCIE slot.
I guess I understood "on its way out" to be a shorter timeframe than you intended, sorry!
it is funny how all this is only driven by ram. From a 5070ti with 16GB of ram at less than 1500€ to a 5090ti with 32GB of ram at 12500€ you get nearly a factor of 10x with the rtx pro at 24GB of ram sitting below 4000€.
Anyone has an idea when will it unclog even a little bit? When is the next NVIDIA lineup comming out? Are they going to release any middle of the road products like strong AMD Insights?
When HBM starts to grow on trees or begins to yield like DRAM.
[flagged]
I just did a cost breakdown of owning a GPU vs paying for tokens.
An RTX 5060 Ti 16GB card costs about $800 on Amazon right now. Running 18.4 hours a day ( off peak electric rate schedule ) it can process approximately 109.3 million tokens per month. The card has a 90 day return window. Credit card interest + electricity puts the cost at $80 total out of pocket, to use this card for 3 months and return it.
Paying an AI provider came out to $1,639.50 for the same 3 months of 18.4 hour a day usage.
You know its a bubble when ... $80 vs $1,639.50
I understand that if you're a major corporation, or a researcher that needs 100+ gpus for a large project, unless you have access to your own supercomputer, the economics will be different.
This was for "individual coder at home" scenarios.
tl;dr ... im exploiting jeff bezos, and math is racist. too funny.
I guess people have never heard of "options" before.
I optioned a video card for 90 days. My bad.
That comparison is nonsensical, it exploits a vulnerability in the retail system in order to not pay for the hardware at all.
On the other hand, if you do include the cost of the hardware it doesn't seem change the overall conclusion.
I'm not exploiting vulnerabilities, im calculating "risk" on the transaction.
You're trading goodwill for greed.
They told me I could return it.
They gave me a 90 day evaluation window.
They did this because it benefits them.
I paid 35% above retail for the card.
And I'm the greedy one ? really ?
Someone needs redpilled on how the monkey planet really works, which was the entire point of this post.
We can choose to spend months crying about card prices and waiting for christmas, or ...
We can choose to analyse the market, calculate risk for our projects, and move forward confidently after doing so.
Or, we can stand on the sidelines and throw rocks at the marathon contestants because they pooped on our lawn.
Choose wisely.
You're a dishonest person who seized a mechanism that improves our consumer experience for your personal gain.
Yes
Edit: I do like that you came back to edit a whole bunch more into your reasoning. So I'll do the same.
I like that you describe yourself as someone who shits on other people's lawns.
I fully believe that you randomly shit on other people's lawns but this marathon you've imagined yourself in is pure delusion.
Thank you for the kind words.
I wish you much success in life.
[flagged]
If you’re returning a product you used because you wanted to pretend it was a rental at the seller’s expense, bluntly, you’re as much the problem as the seller.
Thank you for contributing to current year.
Everyone in the heirachy above you is an exploitative business psychopath. Might as well exploit their calculated greed as they attempt to exploit your empathy.
Yes, that does continue the cycle.
Seems like a great way to help the species get a bit lower still.
Calculating the financial risk of a transaction before committing to it is a problem to you ?
MBA school calls it responsible corporate stewardship.
No, as I said, it’s committing said transaction and then reversing it at the expense of the other party that I take issue with.
As for MBA school, don’t really give a shit how it is colorfully worded to attempt to excuse it.
That card will only run a tiny model though, no?
Exactly. A lot of stupid tokens has applications, but it's not a plug-in replacement for a subscription for most.
... yet.
In fact I'm running qwen3.8-flash-next with a Playwright-mcp docker and it is far better for scraping than my ChatGPT or Claude subscription, for the simple reason that sites aren't lying to Playwright.
I don't understand, your subscription models can also use Playwright, you don't even need to setup an mcp, they'll automatically download playwright-cli and get going.
Sites don't lie to Claude operated Playwright either.
Not to invalidate your point, just saying it's a bit of a straw man.
Sure, but that's not what the subscription uses.
You can just tell it to use playwright instead of the browser skill. This has nothing do with the subscription but with how tool discovery works.
Yes, but even the smaller models, when used within a harness and orchestrator, can be accurate enough for real work, especially when combined or validated by other smaller models.
The 5060 wasn't the fastest/$, it just provided the most VRAM / $ as of yesterday.
Im installing it in a box with a 4070 Ti to run 2 different models simultaneously. Coder + MoE via paperclip or multica.
I see it at $770/GPU, or $48/1GB vram
That probably is the cheapest per 1 GB vram right now. However, the R9700 a few months ago was only $1350/32GB or $42/1GB VRAM. And everyone knew it, because the card is now $1800 or so with supply issues.
I think the market discovered the reasonable min is 32GB. That runs Qwen3.8 Q6 with 100k+ context all in vram. 24GB does not.
Depending on RAM - it can run a big one veeeeeeeery slowly
> to use this card for 3 months and return it.
You're counting on most people being willing to abuse a return policy like that. Also, the 3 month return window must've activated 3 days ago. These prices didn't start just then.
> to use this card for 3 months and return it.
You can also keep it and return a bag of flour weighing the same.
Or even go to people houses and rob them at gun point to steal their hardware.
Yeah I did something similar but more extensive, because not every model is same plus depend on your options how much you want to degrade model from fp8. Some models is just slow on local hardware and not practical really.
localaicost.com
To be fair we don't know which direction that arbitrage will collapse to.
Missing from this analysis:
> it can process approximately 109.3 million tokens per month.
quality determined by model(s) and model size(s), not card size.
no return policy has been abused, you imagined this.
the plan was for 1 card today.
RIF.
I remember when the top GPU was 550€ and the overclocked, custom-cased, water-cooling ready version cost 660€ tops. Any GPU price out there even $800 for 5060Ti from january 2025 is not only out of any sane band, but look as if we're in a ultra-recession, yet other markets aren't affected nearly as extreme.
I bought two 2TB SSDs from TeamGroup last year, about $100/ssd. I put them side by side in a raid setup for my home server. (I wanted raid, and they were less than half the price of Samsung, etc.). I'm on a budget, so I went with the cheap ones and raid reduduncy vs one expensive one with no reduduncy.
The exact same ones are now pushing $300/ssd and Samsung's (which was $300 18 months ago) is now $500.
The prices are insane.
You're comparing running a heavily quantized 10-30B model to running a multi trillion one. Being half the price (unless you expect to only need AI 3 months in your life) is absolutely not a good deal, here.
What it does buy, though, is privacy and freedom (nobody can decide to "change the terms" to milk you more).
You are exploiting a loophole. Calculate the differential if you use one of those "don't buy electronics, just rent them!" services.
This is such a boneheaded comparison.
It’s akin to saying “the new iPhone costs $2.5k, it weighs 250 g, I can get a piece of scrap aluminium for $30, checkmate liberals.”
And imagine if you just stole electricity and GPU. It would be $0 vs 1639.50... The bubble becomes even more obvious!
You're comparing apples to oranges. 109 million tokens is ~nothing, easily a day's worth of casual use. Compare price per token and cloud providers are probably much closer. But even that isn't a fair comparison between cloud AI providers and abusing retailer return policies just yet, because usable local models even remotely comparable to the frontier don't fit in 16gb of vram.
Yeah, and Qwen 3B that you run on 16GB of VRAM and 10K context length surely is comparable to whatever subscription you're comparing to, that also gives access to 3B parameter models? You might want to take actual use cases into account before you try this scam in real life.
I too did a cost breakdown of buying my GPU for ~10K EUR more than a year ago. It came out "Invaluable" as I use it for analyzing my diary, financials and other private things I cannot send to random 3rd parties.
But even the best of models you can run with 96GB VRAM really doesn't come close to the hosted SOTA models, sadly.
> before you try this scam in real life.
What scam are you speaking of ? The return window ?
I don't plan on returning it, unless the economics of upgrading in 89 days favor doing so.
The point was to calculate "my risk" on the transaction.
As for the results of the calculation, I didn't invent math or risk.
For $80 total "risk", I can get 10+ programmers worth of work for 3 months. A much better gamble than paying per token.
I am seeing nobody liking what you said, but you are not wrong.
Thankfully, the environment we are all forced to live in, doesn't care what we think about it, and adapting to it accurately at minimal expense is the difference between surviving and thriving.
What people don't like is the environment, and they just decided to stone my post to death for adapting to it, instead of doing the same.
People are just being people.
I'm a much easier target than that Sam guy, who locked up 40% of the planets GPU's using purchase orders, with no obligation to follow through.
He's an untouchable elite. I'm much easier to mug.
We are all opportunists.
The entire thinking behind the post, was spawned by the article the other day, about the "video game construction kit" guy, bragging about spending $3.1 million a month on tokens in an article, with $0 revenue.
> What scam are you speaking of ? The return window ?
Your calculation seems to have depended on "Buy this hardware with the explicit intention of returning it" process, which is a scam. If you buy hardware you do so with the intent of keeping it, otherwise you find renters, if you don't like scams.
So if you don't plan to return it, wouldn't the cost at minimum be the cost of the GPU??? Why count on returning it if you don't plan to do so?
Not sure that Kantian ethics supports the population mass buying Amazon cards for 90 day periods and then returning them en-masse.
This seems more like a sociopathic "hack" than an actual comment on the economy of selfhost vs inference-as-a-service.
Firstly you are assuming $0.048$/hr for GPU, which is highly misleading and you can easily sell it for double(vast.ai is $0.12) as you are forgetting lot of cost like CPU, cooling etc.
Also even if you can somehow fit a few 100 billion parameter model(like gpt 6 luna or deepseek flash) in 16GB, and magically assume you can run at 200 tokens/sec according to your calculation, the blended cost of it would be around $100, not $1639.