I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
A small update: I've also managed to port the engine to iOS.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session:
"Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
Does this translate to older generation 16GB GPUs, e.g. RTX 3080 / RX 6950? (Assuming BF16 is available, or possibly even FP16?)
A small update: I've also managed to port the engine to iOS.
I'm testing it on my old iPhone 13 Pro (A15 Bionic, 6GB RAM) with Ling-mini-2.0, a 16B MoE model. The first tests are showing around 13 tokens/sec, which is pretty encouraging for a phone from 2021.
It's still early, and I need to test memory usage, longer generations, and how performance changes as the phone heats up.
I'll keep posting updates here as I run more tests and see how far I can push it.
[flagged]
How about you post a real repo before making any claims?
Repo or it never happened
Cool, I guess.
I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Explain the novelty of this implementation. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
[dead]
I'm running Deepseek 4.1 Flash in 4 gigabytes, but I'm not going to tell you how.
In how many hours per token?
I believe he's joking, he's saying the OP article doesn't actually tell us the how.
double woosh