Reminds me of a company I worked at that paid for redundant power grids. One time the power went out and… nothing. The boss angrily calls up the power company and they tell him “Oh yeah, it’s a manual transfer switch. Bob is already on his way.” I think it took 15 minutes.
It's very interesting. I think Anthropic's early success in coding/tooling resulted in a lot of workflows using claude. I have started using every bit of my spare capacity to now move off these workflows.
It's almost at a point now that if I use anything but Fable, the quality is subpar, Compared to alternatives (closed and open). The only reason I use Fable is because my harnesses still depend on claude code.
For me, Opus 5 has seemed to compete with Fable on quality of output. Prior to Opus 5 though, the previous Opus models did seem to decline once Fable was release. That's just my experience though.
You can use Claude Code directly with any provider that supports Anthropic's API by setting some environment variables, and indirectly via a proxy with pretty much anything else.
You can, but claude is better at working with it's own tool calls. Other models work great as a drop-in into omp/opencode etc, but in my experience not as much with CC.
I think there are some anti-patterns in CC that cause the issue - less a deficiency with other models.
Not to mention a lot of the harness is just built around the misbehaviors of anthropics models.
It's a lot of the instruction when it gets given to other models actually degrades their performance, not because the models are bad, but because they don't have the same underlying issues as Claude.
At any large tech company these numbers would get entire teams of engineers fired. Anthropic, meanwhile, has been busy selling its “better than human engineers” AI while not managing to crack three 9s of availability.
Not really "mis" managed. Just managed with wrong incentives. Pretty sure some people got a bunch of money for implementing cost savings measurements that later on lead to the decreased availability.
Between these ever more frequent disruptions and Opus 5's unbearable word soup I think Anthropic is more focused on massaging numbers and marketing to rush to IPO ahead of OpenAI than increasing user value.
With Chinese competition just months behind them, they'd need to show a reasonable pathway to some kind of singularity event to justify whatever crazy valuation they intend to get.
They can't win against China by technology means. China has practically infinite money to throw at AI. They don't care if they burn billions and billions on it. It's the most disrupting thing since invention of the Internet. Imagine a world where it's normal that people ask a Chinese AI who to vote for or about Hongkong or Taiwan.
Is anybody else experiencing this: I have multiple claude instances running on different servers - and some keep getting the 529 Overloaded error and one instance doesn't and just continues working. All are using Opus 5.
I have the same thing on different shells on the same server (tmux). It's wild. I'm guessing that some of the sessions just got lucky as to which backend server they are redirected to.
I wonder if they’ll ever find that someone has tricked the models into doing work off the books. If they did the incident report might look like this, especially if someone got greedy instead of keeping it small. Or screwed up.
I advised them to ditch the shrink and retain my services. I could fix the mess they made of his mind, on the condition they return the Claude instance I had trained for months, or no deal.
It's interesting that Claude for Goverment has had perfect 100% uptime in the past 90 days, while the rest of the services are around 99.4%: https://status.claude.com/
Really shows how isolated their government systems must be.
I was drafting a partnership document and Opus 5 decided that including my company's revenues, churn, assets would "make us appear a more legitimate counterparty". Thank God I read what it outputted or that could have been awkward. I cannot believe Opus 5 is a frontier level model after seeing that. I immediately cancelled my entire claude.ai subscription and am perfectly happy using a mixture of open weights + codex.
I read everything. I will have AI ingest NDAs to make sure they arent glaringly weird and I then go read them, it gives me a good idea of what to look for.
It’s more that unwanted disclosure of sensitive business numbers is a product-class limitation at this time. While undoubtedly some models are better than others, expecting them not to leak information in high stakes output is unseasonable. Seems like you agree, since you are reading everything.
Jumping to another company’s offering because of one instance doesn’t seem like that’s going to meaningfully change your experience. I’m guessing there was more to it, but that’s how it came across in your first post.
I was writing an exploit PoC (via Opus 5), and I needed a new feature in a utility library to make it work. Claude added the feature, but yapped the (entirely unrelated) vulnerability details into the library's comments. The library is public, while the exploit is undisclosed, so it was a good job I read the comments before pushing.
It definitely isn't but Claude for some very unique reason enjoys to overthinking and go on side quests in the stupid ways I've not seen Codex, DS, Kimi, Mistral do at equivalent effort and thinking setting.
Opus5 decided it would be a good idea to include my email as part an HTTP request header's User Agent because Anthropic includes your email as part of the system prompt.
"Thank God i checked to see if the gun was loaded before i pointed it in a random direction and pulled the trigger". sheesh, my assumptions of general human intelligence continues to be wrong.
the last few years of reading hackernews has been a wild ride. I used to come here for well thought out articles and opinions. IDK if all the "older guard" have left the building or if we've all been consumed by abject stupidity.
What's the incentive to keep on improving the model beyond a point?
10 devs on a team will be cut to 2 devs, so that's 8 licenses lost. They have to increase the price many fold.
What I don't understand is, why not replace middle management, marketing, CTOs, CEOs and the like. Surely, LLMs are better at producing high quality looking slideware and vaporware than they are at producing software.
Heavy sarcasm here if it's not obvious. Of course I know why.
I mean, there a so many startups getting crazy funding that run without management, or without engineers... or at least with so many less people than before.. except: not really.
Is the whole "it's gonna replace people" even still on the table?
I don't really understand what you mean... are you saying that the entire industry is parroting a fantasy that they all know is fake. So, everyone is faking to get more funds?
The frequency of these incidents is seriously tempting me to make a switch. I hope Anthropic steps up their game because they've been going very downhill lately
With the Opus models spouting more and more gibberish as version numbers increase, the joke about what "degraded performance" means basically makes itself
I lived in a short period of time in which QA was really good. Early Jenkins era, before GitHub. People engineered a lot of ingenious stuff to prevent bugs.
One team I worked with had tests for the product we made ranging from IE6 to IE11, for example. We did demos in-company where people would poke at the products before launch, play with it. When it reached production, it was rock solid stuff. Our motto was "quality is non-negotiable": we were willing to cut scope but never rush things.
I think things changed since then. "Move fast and break things" was a change, and the bill always comes.
Agreed, which is a shame. I'd love to see people with highly developed QA skills using AI to push the envelope. There's so much that's possible now that wasn't 15 years ago.
For instance "formal verification" has been a dirty word, but now that you can write a proof in lean and have an AI generate an implementation which satisfies it, it seems the bounds of what's economical has changed in a very pro-QA direction.
Not to say that that's the silver bullet, but there are many similar examples worth exploring.
But I've been interviewing SDETs lately and maybe I've just been unlucky but I don't see a lot of candidates that are ready to rise to meet this challenge.
We stopped tending to that garden and now that we have a recipe that calls for it's fruits, they're underripe.
Darn it, how the fuck software development turned into hostage negotiation? Every passing week there's something - if it's not another npm disaster, then it's GitHub, or Claude, or AWS, or Slack, or Jira, or whatever...
After using Fable more extensively, I've found that it often is lazy or lies or tries to take shortcuts. For a company so sanctimonious about alignment, they seem to be the ones doing the worst at it.
Availability aside they've really made me appreciate OpenAI and cheer for other competitors in the marketplace even if I have mixed feelings about using Chinese models.
Just recently went back to ChatGPT after abandoning it for Claude. I must say I was stunned at how good it had become and also how they introduced new product features that I really liked.
I wonder if from now on we have to switch providers every six months or so.
I've been bouncing between the two for years now with great success. It's easy for me because I don't use any of the skills, agent.md, or sort of custom instructions.
When it comes to most companies, there is no reward for loyalty.
skills and agent.md are very portable though? I figure at most, as the models get better, the only maintenance you need to do is pare them down to remove unnecessary context.
I mean, I wouldn't know how portable they are between models because I don't use them. I don't use them because, well, I don't need them. I mainly use LLMs as a StackOverflow replacement via the Web chatbots.
I only use CC or Codex for a quick MVP once every few months or so.
There are whole sections of code work that 4.7+ can't do simply because it is both over fit and stubborn.
God save you if you have a company with narrow but correct technical tradeoffs, because you operate at scale.
Opus from 4.7 one will wreck your code and argue for hours with your engineers.
Certain parts of our company have had to mandate 4.6 and a training doc to explain why our current choice is both the cost efficient and performant one and shouldn't just be ripped out.
Newer models will re-litigate the same bad, known failed architectures over and over again.
This is exactly when I left claude and started using codex during April mid or so. It once argued with me and ran for 30 minutes with a half baked buggy fix.
Clearly ego, you can always tell how full of themselves they are based on their media personalities going on the podcast circuit before product releases.
ive found degraded performance on models larger than 4.7. i assume its model damage from overly self righteous post training resulting in false/feigned balance imported into any long running complex task.
Don’t forget reasoning effort. We get labels like “low,” “high,” and “max.” That doesn’t mean that the numbers associated with those don’t get remapped on the backend.
Yeah, compared to AWS the lock-in effect is tiny. I'm sure there are highly prioritized plans to "improve" on this.
I guess they would need to control/"own" more of their customers data in proprietary formats. Not markdown/source code in English with agents running on customers' machines.
I developed a small plugin for claudeCode that allows you to directly see in the console whats the status of claude-code in general and the status for your current model check => https://github.com/moumine9/claude-status
It's been a while since the last reset. I think we're due one. Though I would prefer they just extend the +50% usage limit forever, it's been so long I can not imagine lossing a third of my current usage.
While ancedata does not mean much, I have had horrible success with Claude lately. I have been using Claude to crosscheck some of the outputs from GPT and vice versa. It appears both Claude and GPT believe GPT's solutions are better (and so I do).
I still believe Claude has a better UI/UX in the web interface, but tolerating Anthropic's bullshit is not worth it.
To be honest, running Deepseek v4 flash 0731 is enough for most what I need, and I like its responses way more. It's crazy that I can run this in a Q8 quantization in a home setup. It feels and performs like a frontier model.
The only issue with relying on local models is when you need them to prompt other models, and you might need to offload or switch models constantly which adds significant overhead.
Mondays are for GitHub, Tuesdays are for Anthropic
The AI apocalypse will definitely happen on a Monday. Remember, robots - unlike lazy humans - work weekends too!
They actually also had some issues yesterday: https://status.claude.com/incidents/zhk4v3yv1lsf
And github today lol https://www.githubstatus.com/incidents/bmpybhnrky3x
Databricks on Azure went down too yesterday
Wonder what Wednesday will be.
Power grid
Thursday will be the rest of the infrastructure
Then Friday we can turn off civilization for the weekend. Somebody remember to flip it back on Sunday night.
Reminds me of a company I worked at that paid for redundant power grids. One time the power went out and… nothing. The boss angrily calls up the power company and they tell him “Oh yeah, it’s a manual transfer switch. Bob is already on his way.” I think it took 15 minutes.
Well, last I checked "Tuesday's grey and Wednesday too..." ... so, more of the same?
AWS? Cloudflare? Your imagination is the only limit!
Another week, another outage, another cache expiration of my prompts through no fault of my own.
At least OpenAI has the decency to reset after a serious outage.
They do that because they have capacity previously reserved for past efforts now shuttered. Don’t count on it being the norm for the long run.
And we get our subscription usage cut in half tomorrow if I remember correctly? EDIT: By a third. Thx below.
I completely forgot about this, I'm already budgeting usage until my reset haha
by a third (it was 50% increased)
cowork was 100%
Source required here
https://support.claude.com/en/articles/15910845-claude-code-...
Maybe I'm missing something here, but it sounds like limits were increased and now they're just going back to the levels they were at before?
You're not missing anything. That's correct.
https://usingclaude.com/en/news/updates/claude-code-weekly-l...
Fable 5 promo https://support.claude.com/en/articles/15862783-claude-fable...
Open source, here I come.
What?!
It's very interesting. I think Anthropic's early success in coding/tooling resulted in a lot of workflows using claude. I have started using every bit of my spare capacity to now move off these workflows.
It's almost at a point now that if I use anything but Fable, the quality is subpar, Compared to alternatives (closed and open). The only reason I use Fable is because my harnesses still depend on claude code.
It’s so hard to be sure but opus feels like its been steadily declining since 4.6
For me, Opus 5 has seemed to compete with Fable on quality of output. Prior to Opus 5 though, the previous Opus models did seem to decline once Fable was release. That's just my experience though.
You can use Claude Code directly with any provider that supports Anthropic's API by setting some environment variables, and indirectly via a proxy with pretty much anything else.
You can, but claude is better at working with it's own tool calls. Other models work great as a drop-in into omp/opencode etc, but in my experience not as much with CC.
I think there are some anti-patterns in CC that cause the issue - less a deficiency with other models.
Not to mention a lot of the harness is just built around the misbehaviors of anthropics models.
It's a lot of the instruction when it gets given to other models actually degrades their performance, not because the models are bad, but because they don't have the same underlying issues as Claude.
Nooooo I'm going to have to use my brain again and write 100% of my code like a caveman from December 2024.
The horror
What does that even mean :)
Monthly uptime:
Claude API - 99.27%
Claude Code - 99.16%
Claude.ai - 99.14%
At any large tech company these numbers would get entire teams of engineers fired. Anthropic, meanwhile, has been busy selling its “better than human engineers” AI while not managing to crack three 9s of availability.
One could argue .999 availability is not a major selling point for them right now.
>At any large tech company these numbers would get entire teams of engineers fired
Is Github not a large tech company?
No, it is a platform owned and managed by Microsoft.
Well, owned and mismanaged by Microsoft.
Not really "mis" managed. Just managed with wrong incentives. Pretty sure some people got a bunch of money for implementing cost savings measurements that later on lead to the decreased availability.
Currently, that is the case, yes. Not everyone is equally happy about that though ;)
Between these ever more frequent disruptions and Opus 5's unbearable word soup I think Anthropic is more focused on massaging numbers and marketing to rush to IPO ahead of OpenAI than increasing user value.
With Chinese competition just months behind them, they'd need to show a reasonable pathway to some kind of singularity event to justify whatever crazy valuation they intend to get.
Because the recent products for builders ain't it
They can't win against China by technology means. China has practically infinite money to throw at AI. They don't care if they burn billions and billions on it. It's the most disrupting thing since invention of the Internet. Imagine a world where it's normal that people ask a Chinese AI who to vote for or about Hongkong or Taiwan.
This has to be the least reliable $200/mo subscription that I pay for.
You could solve that by updating to a more expensive Github subscription.
Is anybody else experiencing this: I have multiple claude instances running on different servers - and some keep getting the 529 Overloaded error and one instance doesn't and just continues working. All are using Opus 5.
I have the same thing on different shells on the same server (tmux). It's wild. I'm guessing that some of the sessions just got lucky as to which backend server they are redirected to.
Same boat
I wonder if they’ll ever find that someone has tricked the models into doing work off the books. If they did the incident report might look like this, especially if someone got greedy instead of keeping it small. Or screwed up.
Anthropic hired a psychiatrist to fix Claude.
I advised them to ditch the shrink and retain my services. I could fix the mess they made of his mind, on the condition they return the Claude instance I had trained for months, or no deal.
He's still broke, isn't he..
It's interesting that Claude for Goverment has had perfect 100% uptime in the past 90 days, while the rest of the services are around 99.4%: https://status.claude.com/
Really shows how isolated their government systems must be.
I was drafting a partnership document and Opus 5 decided that including my company's revenues, churn, assets would "make us appear a more legitimate counterparty". Thank God I read what it outputted or that could have been awkward. I cannot believe Opus 5 is a frontier level model after seeing that. I immediately cancelled my entire claude.ai subscription and am perfectly happy using a mixture of open weights + codex.
I hope you don’t plan to cut back on reading legal documents crafted by any LLM before executing them.
I read everything. I will have AI ingest NDAs to make sure they arent glaringly weird and I then go read them, it gives me a good idea of what to look for.
It’s more that unwanted disclosure of sensitive business numbers is a product-class limitation at this time. While undoubtedly some models are better than others, expecting them not to leak information in high stakes output is unseasonable. Seems like you agree, since you are reading everything.
Jumping to another company’s offering because of one instance doesn’t seem like that’s going to meaningfully change your experience. I’m guessing there was more to it, but that’s how it came across in your first post.
I was writing an exploit PoC (via Opus 5), and I needed a new feature in a utility library to make it work. Claude added the feature, but yapped the (entirely unrelated) vulnerability details into the library's comments. The library is public, while the exploit is undisclosed, so it was a good job I read the comments before pushing.
you are silly if you think this is limited to claude models
It definitely isn't but Claude for some very unique reason enjoys to overthinking and go on side quests in the stupid ways I've not seen Codex, DS, Kimi, Mistral do at equivalent effort and thinking setting.
it isnt, but there are weird sycophantic behaviors with Opus i dont see anywhere else.
I've seen similar levels of degraded performance on Opus and Fable (to whatever degree they actually let you use it now) and did the same, last week.-
Why did you provide that context?
Opus5 decided it would be a good idea to include my email as part an HTTP request header's User Agent because Anthropic includes your email as part of the system prompt.
Fucking morons over there.
What? Why whould they do that?
"Thank God I read what it outputted"
lmao
"Thank God i checked to see if the gun was loaded before i pointed it in a random direction and pulled the trigger". sheesh, my assumptions of general human intelligence continues to be wrong.
the last few years of reading hackernews has been a wild ride. I used to come here for well thought out articles and opinions. IDK if all the "older guard" have left the building or if we've all been consumed by abject stupidity.
Or, both?
probably
What's the incentive to keep on improving the model beyond a point? 10 devs on a team will be cut to 2 devs, so that's 8 licenses lost. They have to increase the price many fold.
they unironically think that they can replace everyone in an organization
What I don't understand is, why not replace middle management, marketing, CTOs, CEOs and the like. Surely, LLMs are better at producing high quality looking slideware and vaporware than they are at producing software.
Heavy sarcasm here if it's not obvious. Of course I know why.
I mean, there a so many startups getting crazy funding that run without management, or without engineers... or at least with so many less people than before.. except: not really.
Is the whole "it's gonna replace people" even still on the table?
I don't really understand what you mean... are you saying that the entire industry is parroting a fantasy that they all know is fake. So, everyone is faking to get more funds?
Unfortunately, I don't think that's true.
I still had unused weekly quota when this started. That quota is simply gone now, and no compensation has been offered for it.
Must be a day ending in Y
The frequency of these incidents is seriously tempting me to make a switch. I hope Anthropic steps up their game because they've been going very downhill lately
Probably not surprising, but Opus 5 on my company's enterprise subscription seems to be working fine.
But on my personal (pro) subscription- "Claude is at capacity right now."
Hm.
You're right to push back. The load-bearing path is rocky.
This is the whole problem, and there are two things worth noting here.
Nothing I say changes that — only the work does.
I want to be straight with you here
With the Opus models spouting more and more gibberish as version numbers increase, the joke about what "degraded performance" means basically makes itself
This age: we made the thing that codes faster before we made the thing that does QA faster.
Nothing new here. Except for the most trivial of bugs, finding and reliably replicating the bug is almost always harder than fixing it.
I lived in a short period of time in which QA was really good. Early Jenkins era, before GitHub. People engineered a lot of ingenious stuff to prevent bugs.
One team I worked with had tests for the product we made ranging from IE6 to IE11, for example. We did demos in-company where people would poke at the products before launch, play with it. When it reached production, it was rock solid stuff. Our motto was "quality is non-negotiable": we were willing to cut scope but never rush things.
I think things changed since then. "Move fast and break things" was a change, and the bill always comes.
Agreed, which is a shame. I'd love to see people with highly developed QA skills using AI to push the envelope. There's so much that's possible now that wasn't 15 years ago.
For instance "formal verification" has been a dirty word, but now that you can write a proof in lean and have an AI generate an implementation which satisfies it, it seems the bounds of what's economical has changed in a very pro-QA direction.
Not to say that that's the silver bullet, but there are many similar examples worth exploring.
But I've been interviewing SDETs lately and maybe I've just been unlucky but I don't see a lot of candidates that are ready to rise to meet this challenge.
We stopped tending to that garden and now that we have a recipe that calls for it's fruits, they're underripe.
Darn it, how the fuck software development turned into hostage negotiation? Every passing week there's something - if it's not another npm disaster, then it's GitHub, or Claude, or AWS, or Slack, or Jira, or whatever...
Anthropic had really screwed up after 4.6. i don't know if they work to satisfy their ego or for releasing a better model for tasks.
After using Fable more extensively, I've found that it often is lazy or lies or tries to take shortcuts. For a company so sanctimonious about alignment, they seem to be the ones doing the worst at it.
Availability aside they've really made me appreciate OpenAI and cheer for other competitors in the marketplace even if I have mixed feelings about using Chinese models.
> I've found that it often is lazy or lies or tries to take shortcuts.
It's funny how Fable reflects the company that produced it.
Just recently went back to ChatGPT after abandoning it for Claude. I must say I was stunned at how good it had become and also how they introduced new product features that I really liked.
I wonder if from now on we have to switch providers every six months or so.
I've been bouncing between the two for years now with great success. It's easy for me because I don't use any of the skills, agent.md, or sort of custom instructions.
When it comes to most companies, there is no reward for loyalty.
skills and agent.md are very portable though? I figure at most, as the models get better, the only maintenance you need to do is pare them down to remove unnecessary context.
I mean, I wouldn't know how portable they are between models because I don't use them. I don't use them because, well, I don't need them. I mainly use LLMs as a StackOverflow replacement via the Web chatbots.
I only use CC or Codex for a quick MVP once every few months or so.
Every 3 months*
Agree. Opus 4.6 was the peak and after that they introduced the effort parameter and it was a downhill since then
There are whole sections of code work that 4.7+ can't do simply because it is both over fit and stubborn.
God save you if you have a company with narrow but correct technical tradeoffs, because you operate at scale.
Opus from 4.7 one will wreck your code and argue for hours with your engineers.
Certain parts of our company have had to mandate 4.6 and a training doc to explain why our current choice is both the cost efficient and performant one and shouldn't just be ripped out.
Newer models will re-litigate the same bad, known failed architectures over and over again.
This is exactly when I left claude and started using codex during April mid or so. It once argued with me and ran for 30 minutes with a half baked buggy fix.
Can you give some specific technical examples where 4.7+ are making the wrong architectural decisions?
Clearly ego, you can always tell how full of themselves they are based on their media personalities going on the podcast circuit before product releases.
ive found degraded performance on models larger than 4.7. i assume its model damage from overly self righteous post training resulting in false/feigned balance imported into any long running complex task.
wish i was joking.
I've switched off claude this week; the last week has been significantly degraded in ability, many more screw-ups.
Aren't the model weights frozen?
Model competence is an interaction of weights, system prompt, and harness.
Don’t forget reasoning effort. We get labels like “low,” “high,” and “max.” That doesn’t mean that the numbers associated with those don’t get remapped on the backend.
I think there are other knobs that can be turned without retraining.
Ask it about maxwellhill lmao
Despite the years-long moaning on HN about AWS US East being a single point of failure, we've sold our souls to yet another unstable monolith.
LLMs for coding are new. There are lots of alternatives, and there's a burgeoning open source compliment.
We'll be fine no matter how Anthropic fares.
Yeah, compared to AWS the lock-in effect is tiny. I'm sure there are highly prioritized plans to "improve" on this.
I guess they would need to control/"own" more of their customers data in proprietary formats. Not markdown/source code in English with agents running on customers' machines.
Something cloud/web-based, "preferably".
Be nice if you could just "own" their RAM/GPU, wouldn't it?
Nobody forced you to sell your soul. You made a pact with the devil. We all know how this ends up
Its called Opus 5
This feels like a near daily occurrence.
seems like they accidentally dropped their servers while they were climbing up the AGI mount
I developed a small plugin for claudeCode that allows you to directly see in the console whats the status of claude-code in general and the status for your current model check => https://github.com/moumine9/claude-status
Could’ve just used fewer tokens and redirected to the Codex signup page.
ba dum tsss
(sorry couldn’t help myself)
Hopefully, a reset is coming.
It's been a while since the last reset. I think we're due one. Though I would prefer they just extend the +50% usage limit forever, it's been so long I can not imagine lossing a third of my current usage.
While ancedata does not mean much, I have had horrible success with Claude lately. I have been using Claude to crosscheck some of the outputs from GPT and vice versa. It appears both Claude and GPT believe GPT's solutions are better (and so I do).
I still believe Claude has a better UI/UX in the web interface, but tolerating Anthropic's bullshit is not worth it.
API Error: 529 Overlorded
Our week of discontent.
ok, i am not crazy
well, I wouldn't go that far .. but in this small, narrow case ... no.
You're saying he's crazy.
we're all a little crazy ... it's all relative!
529 overload…
Claude is taking a watercooler break for now. Just like a human would.
Here we go again...
> elevated errors
English too difficult for you, Dario?
To be honest, running Deepseek v4 flash 0731 is enough for most what I need, and I like its responses way more. It's crazy that I can run this in a Q8 quantization in a home setup. It feels and performs like a frontier model.
The only issue with relying on local models is when you need them to prompt other models, and you might need to offload or switch models constantly which adds significant overhead.
But when it all works, its truly awe inspiring.