For the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
I agree with you, and I'm using claude and other models heavily, the idea is not in the soneshot - because actually as you see in the logs I posted on github, it took around 107 + 50-ish experiments to figure out which models to run which and push the speed.
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.
I just asked myself the same thing and wondered if this was some new technique that completely slipped past me. - But it certainly must be a double typo
So true, people think they can prompt AI, tell it to create something amazing and "Novel" so they can feel cool and bragg about it. Shame on such people.
"claudia slop me up some 600m, commit, push, make 500 pages of slopped README.md, github stars+++"
great, you should definitely do this, there will be absolutely no problems whatsoever and it'll also run at peak speeds while of course being completely unusable because who wants to run a slopped 600m
No one will run a slopped 600M or even 16B for anything serious today, the whole goal is to experiment, to push both limits of what can be done with the model, and the hardware itself.
It's fun times actually. And I think such experiments and others posted are both showcasing the limits people are pushing.
Recently I started this open source project at my AI company, where I'm trying to make LLMs working
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
Aw man I would love to be able to use the crank to back up the agent dialogue, maybe make it branch forward in a different direction. Make the dialogue into a game
Yes this has been my exact thought for awhile now.
Perhaps the real value with AI is going to come from...surprise here...a whole bunch of different low resource AIs networked together living and working and yes playing thru a networked "society"
I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.
this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.
I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot
the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.
Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.
I was looking for the lowest claim in this thread, as I managed to get a nano LLM running in the ≈3mb of SRAM of an FPGA with zero DDR in the loop at 20k tok/s, but 2mb on a 80386, great job.
I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.
Raspberry Pis in general are too easy for this task they don't even need modifications, older iphones seem more interesting due to some work on comptability.
Machines with unified ram work better? whats the bare minimum Ram you need? Maybe those old SGI or Sun machines could have a second life. SGI boxes made the Terminator come to life on the big screen so now their second act could be to help bring the real terminator to life :P
They have a color LCD display that looks unnecessarily good considering what it does. You can see an example of one of the animations at the 52s mark in this video https://www.youtube.com/watch?v=peNyiLt5SKk&t=52s
I've got no idea what's inside of it yet but I'm open to tinkering if someone can walk me through it
I doubt any vape is powerful enough to run any LLM on its own, but idk if it's possible to run one on a cluster of vapes.
a cluster of vapes running an LLM would be insane. a single vape of these though may run a small LLM like few hundred thousand parameters, worth tinkering with.
Vapes typically have less than 100KB of RAM, though I haven't been able to find specs for the Rifbars yet. But I assumed even if they're 1MB that's still not enough to run any kind of LLM right?
A PlayStation 2 dev kit (with 128 MiB of RAM instead of the standard 32 MiB).
Between the VUs, the R5900's SIMD (MMI), the GS and the I/O buses (FireWire, Ethernet, IDE, Memory Cards...) to stream model weights, the smallest Bonsai models (or aggressively quantized Qwen3.5-0.8B) and some really cursed programming might just eke out something the PlayStation 2 has no business running.
This would be fascinating. I've sometimes wondered how computing would have been different if we'd known how to build modern LLMs in the 2000s, despite the much poorer hardware (leaving aside how you could possibly train such a thing back then).
This is very cool. Love the trend of optimizing small models
I’ve been working on small local models and it’s amazing how fast the tech has progressed
A year ago, I was creating conversational characters by finetuning a qwen 2.5 0.5b param model, and it worked pretty well. But no reasoning or math capabilities, and still required some decent specs to run; could maybe do well enough in a beefy raspberry pi
But now, I have a locally trained 6.5M param model that does reasoning, solves multi-step math problems, does simple physics problem simulation, tool calling, RAG and conversational chat. Total size about 30MB packaged for ONNX, and runs with <10ms latency in the browser
6.5M! Impressive. Is it initialised or distilled from any base model or entirely from scratch? What kind of training data for the RAG and conversational chat specifically?
I've been surprised myself. For reference, it gets better ppl (and with faster training), than the nanogpt model from Karpathy's tinyshakespeare tutorial, at smaller size (nanogpt is 10M params)
Trained entirely from scratch. Which gets it pretty far in terms of basic conversational skills, especially when also trained on chat conversations
The training sources are:
1. Google Synthetic PersonaChat
2. DailyDialog
3. SODA conversational data
4. Hand-written model-personality conversations, repeated during training
5. Identity-corrected conversations—rewritten names, human-life claims, and hobby responses
6. 960 synthetic recovery conversations, teaching it to admit ignorance after user push-back
The final set contains 23,270 conversations
However, the chain reasoning and math problem solving skills, I was only able to add recently, by grafting certain parts from a 1B model, then doing space-weight programming, and finally finetuning/training to get the model to "get used to" reasoning. The "full model" is more of a harness + model:
* Frozen base model to interpret questions
* Additional trained heads to select question types, evidence, and follow-up behavior
* A library tool for retrieving facts
* Working memory and internal script language execution for arithmetic, conversions, and comparisons across answers. For example, it can retrieve a building’s height, remember that number, then execute “half of that”
* Identity extraction and physical simulation, so you can do thing like - User: "Ben has the ball", User: "Who has the ball", Model: "Ben", User: "Ben gave the ball to Sally", User: "Who has the ball now?", Model: "Sally"
It's relatively easy adding skills either via its internal scripting language, grafting+weight programming, fine-tuning and/or training it to use a new skill. The main limitation right now seems to be just quality data and proper supervision, ie. someone to talk to the model, flag issues, fix them, then re-test. This is kinda tedious, but I've found that even the best frontier model is kinda lacking in quality when producing synthetic conversational data, and most datasets out there have a lot of cruft. So I get better results just manually producing the data and iterating, than feeding it GBs of public datasets (even famous datasets have a quite a bit of un-reviewed data, as well as errors, which aren't even published)
Creating and training this model has been a lot of fun. Since it's so tiny, it's easy to experiment with and try a lot of different things. Most training I can just do in a few minutes locally
actually your methodology inspired me to try smth related to small models and harnesses I've been thinking about - and seems you got the main pieces (the idea was trying to see what custom harnesses can extract more accurate reasoning from small models)
A year ago 1B parameter models couldn't do much, today with the correct harness and/or fine-tuning they can do more or less serious work (not frontier tho).
I believe in a 6-8 months it will get even better and my TV will be happy.
I didn't try fine tuning so small models honestly but looks interesitng to try.
It's technically sweet regardless, so don't take this as even a little bit of negative pushback, but I am curious what the utility of these small models is? Are they smart enough to be useful on practical tasks? What are people doing with them?
Not op but my opinion is that It doesn't matter if it's not currently useful. I think it's important that the community focuses on this line of research because big companies will not. It's in their best interest to make LLMs resource hungry
Yeh; this. They see how GTA6 looks and know a hypothetical scenario like a future Nvidia 7000 series with 96GB VRAM and 512GB system RAM in everyone's home means the entire tech, media, knowledge work labor system; the rich peoples social moat; evaporates.
It's all quite clearly collusion among a few dozen, hundred. No way their education/experience or data analysis engines have not made them aware.
They’re just used to Boomer and Silent Generation politicians blithely enabling them cause line go up for them when they do!
Curious to see how well it works on Millennials and GenZ; the elders were educated before STEM education had matured in a lot of ways. Younger generations have not had debunked fantasy rammed down their throat as scientific fact; Nostradmus types of the mid-late 1900s pointing at some ignorant math assuming it's true because it's written in a scientific notation!
Duh!
Science education of the past gave rise to thinkers like the Oceangate CEO. Nihilists who figure there's no such thing as forever and we all die anyway; so YOLO no matter who I take down with me.
Science education peaked in the 50s when people still believed there need to be a balance between theory and experiments and that grounding theory with experiments is worthwhile.
It went all downhill from there. In the 70s and 80s experimental physics got a bad rap and I think this mindset trickled down to education. The turning point for me was the 1993 cancellation of the Superconducting Super Collider, where this tendency must have become obvious to everyone.
Not sure about your other points and what you conclude from a perceived improvement of STEM education, but I am convinced boomers and Gen-X enjoyed more solid scientific training than later generations.
If you work in a highly regulated space subject to e.g. HIPAA, and you work with ancient on-prem systems within a slow moving and risk intolerant organization -- for instance, virtually every state run healthcare system in the US -- then these are all you can hope for.
In these settings, data is insanely messy beyond your wildest nightmares and hidden away in free-text fields that small models like this are great at cleaning up, normalizing, parsing, categorizing, etc
Agreed, and it's a new use case to me because actually in the cases I had the clients were okay with getting GPUs and handling the data mess for us, but none of them requested fine tuning, I believe trying to fine tune will be insanely complex from a regulatory prepsective.
This is an interesting question, so the idea is, based on experience, fine tuned small models can do narrow tasks very well (my experience running models fine-tuned on customer data as low as 1-2B is very promising) also the goal is to find a breakthrough were instead of the line of work (which I respect) that tries to squeeze much bigger models into smaller hardware like strata and maya, I'm working backward to discover/search for custom kernels and modifications till we exhaust the limit of what a smaller machine is capable of, so instead of people doing 64gb machines and old gpus to run a deepseek, I want to see what is the biggest usable qwen small model can run with reasonable speed on smaller devices, and by scaling as we discover more tricks, normal laptops can unlock more compute - for example a qwen 3.8 27B can do pretty serious work when coupled with a specialized harness - but we are yet to figure out what is the smallest machine that can run it with usable output and usable speed, and this can unlock interesting business.
I had this case in fine tuned small models (up to 27B) in finance and legal, over bigger models (cost-wise, on-prem for a regulated institution, and better perofrmance because we owned the data pipeline)
I'm told that qwen 3.8 27B is good enough for practical programming work, so if that's doable then I could easily see the utility. Or... the biggest qwen I see here is Qwen3.5-2B; is that actually capable of programming or agentic work with code?
It's not, but tiny models have so far been on the same track of increasing capabilities as large ones. The parameters needed to do useful work keeps going down. It hasn't reached 2B yet, but it's very possible it'll get there. In the last month we've gotten 4B models that can do small tasks.
speech models are what I'm trying next because imagine a nice portable meeting transcription device that can hold around without internet and for NDA-ed meetings and so (with permission of course)
I mostly agree today, but I'm trying to push for the future where smaller models are more capabable, and I believe it will happen + the fine tuned cases I mentioned in other commments.
On other notes : there is no free lunch, you have to sacrifice something for the other
So this work appears to work on inference speed and getting large models to work on simpler RAM constrained hardware, whereas my (private) work has been on getting simpler RAM constrained hardware to give agentic behaviour with small models. I've already started the work of using some of the learnings described in this repository. Looking forward to seeing what can be achieved.
that's an interesting thought - can we somehow combine "ai bacteria" specialized into different things and running on different small machines into "one organism"?
I have a few Maxwell and Pascal series cards lying around, anywhere from 2GB to 11GB of VRAM. Have experimented with small/tiny models for agentic stuff with mixed results, however they are getting better at tool calling (before they'd just produce junk output that the harness couldn't parse).
I'm excited to see what we can squeeze out of older hardware, in the same spirit of demoscene or "will it run DOOM?".
Especially timely given current hardware scarcity. Will be watching this space.
if you're having some cards already try to explore what people are doing with weight streaming and model compression, I got good results on older cards with far better models than the 16B
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.
I’ve been investigating getting something like that running on the Wii too, though I haven’t made much progress in a while. Some basic calculations says a 1B model might be possible, at least a few hundred M
I would actually like to see the Wii one, I don't have one unfortunately so I can't directly try on it but this esp is interesting I would like to replicate and/or improve the results on it, this technique is similar to the streaming i did from flash but probably more efficient, I need to look it up and understand it more. Thanks for sharing it
There is a lot of random e-waste being tossed. Think TV's, Wifi routers, IOT devices. I mean many of these should run right? WRT54G was an extremely common router so something like that? Or the old Laserjet printers that were everywhere.
Thanks for sharing! I'm exploring a similar direction, but optimising for extreme model size. I have gotten an LLM down to 970 KB, with some interesting results compared to GPT-2 (small).
Super interesting, not a CS/AI guy here, but I'm curious as to how you can train such a small-sized model give you as accurate an answer or good hypotheses to get you going
I'm planning to have a blog post up in the next two weeks or so (simedw.com), still have a lot of experiments to run, and training even tiny models takes a bunch of time.
Can you get a model trained to cover a limited area of expertise, such as home control or industrial control? It would be useful to have completely standalone units for talking to medium and large HVAC systems. Especially if they came on hardware with a 30 year lifetime.
This is the lord’s work right here. Models at the the intersection of small and useful — forthcoming models, admittedly— will be in the toolset that the next generation of poor kids will use to build a more egalitarian world.
Funny, I use that exact website to find passively cooled hardware to use as a server.
I did get Qwen3.5 0.8B working at 5 tokens per second on the iGPU of a Dell Wyse 7020. I mostly use it to complain to; I have not yet found an agentic task for it. But also the CPU does plenty of small services and terraria servers and such.
well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.
It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.
There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.
running your own hardware and extracting value out of every piece of hardware is important - one for efficieny, not every application needs smth big - and two for applications that need privacy and similar controls
Qwen isn't a good tiny model, it's scaled down and leads to bad results. It'll output the name of a capital city, but then it falls down on real work. MiniCPM, LFM, K2, Granite, Nemotron Nano, Gemma E2B, are designed to do work at tiny size. Speed means nothing if it gets stuck in doom loops, can't tool call, isn't trained on embedded functionality, etc. (https://artificialanalysis.ai/models/open-source/tiny)
I agree, this is why in the further experiments I was putting LFM as the main model not qwen, qwen was nice to prove the point for me, but as I wanted to do more work it became unusable and bad quality, LFM is fine, Gemma didn't perform well but I didn't go far with it, I need to check the others,and I tried MiniCPM but not on this project.
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
The fact that you've commented two things under the same comment kinda feels like it's not your doing but the agents doing
mrashad74 3 minutes ago | parent | next [–]
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
reply
mrashad74 4 minutes ago | parent | prev | next [–]
well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)
Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.
For the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
I agree with you, and I'm using claude and other models heavily, the idea is not in the soneshot - because actually as you see in the logs I posted on github, it took around 107 + 50-ish experiments to figure out which models to run which and push the speed.
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.
Why did you both say "soneshot"? Same typo of oneshot?
I just asked myself the same thing and wondered if this was some new technique that completely slipped past me. - But it certainly must be a double typo
"double typo"
When you find two sants in the kitchen, you are certain to find more sants. Won't fool ME.
orrr we can listen to paranoia that these 2 comments are just an elaborate advertisement
soneshot - super one shot!
the kind of one shots you get with fable 5.5 :D (it was a double typo)
Soneshot is a much better technique for such and such type of projects
Because the response was written by an LLM running on a tamagotchi
a sentient tamagotchi is not far from us..
It's not clear whether the second one isn't a scheapshot at making fun of the first one.
Hey, this is the Internet. It’s serious business! We shouldn’t make sfun of people!
What is “soneshot”?
Yes that was my typo. Sorry about that.
I think it's a typo of "oneshot", referring to a single prompt being able to handle the entire task.
Well, my netbook. Atom n270, 1GB of RAM, 1.5GB potentially with ZRAM. If it can run in that, a Pentium 4 with SSE2 would be the closest setup down.
So true, people think they can prompt AI, tell it to create something amazing and "Novel" so they can feel cool and bragg about it. Shame on such people.
It's okay to share fun thoughts even if you haven't put blood sweat and tears into them.
no one said it's amazing or novel, it's just a fun experiement I'm not selling this :) just out of curiousity to explore.
lol sure
"claudia slop me up some 600m, commit, push, make 500 pages of slopped README.md, github stars+++"
great, you should definitely do this, there will be absolutely no problems whatsoever and it'll also run at peak speeds while of course being completely unusable because who wants to run a slopped 600m
t. at 1200+ commits on my "oneshot"
No one will run a slopped 600M or even 16B for anything serious today, the whole goal is to experiment, to push both limits of what can be done with the model, and the hardware itself.
It's fun times actually. And I think such experiments and others posted are both showcasing the limits people are pushing.
Recently I started this open source project at my AI company, where I'm trying to make LLMs working
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
> a more challenging device
A Playdate
Aw man I would love to be able to use the crank to back up the agent dialogue, maybe make it branch forward in a different direction. Make the dialogue into a game
https://play.date/
I made myself Remote Control Deluxe(tm). Which is essentially a remote control connected to my Claude and VLC.
I hold A to speak. B to cancel. Turn the crank really fast and really slow → send the prompt!
I still do use it sometimes.
The app talks via serial and WiFi.
pretty interesting, I don't own one, but it's possible probably with an emulator to figure this out, thanks!
Yes this has been my exact thought for awhile now.
Perhaps the real value with AI is going to come from...surprise here...a whole bunch of different low resource AIs networked together living and working and yes playing thru a networked "society"
Sounds kinda familiar doesn't it?
vigorously turn the crank to speed up the output. ha.
The only challenge in that case is ram (512, cpu is fine), but I think you should try getting something usable on the raspberry pi zero.
Once that's done it means people(or startups) can get an edge LLM subsystem on their projects for the zero or boards with more ram.
actually the pi zero is a nice target to play with next, will give a shot and report
A Speak and Spell that replies using the voice processor
Is it even still around? :D would be interesting if actually accessible idk
I got speak-n-spell voices working on arduino mini.
perhaps the pi 1st/2nd gen?
I'd thought of the ESP32 devices but their RAM is too small
Pi 1 is too easy. My PicoLM [fork] already runs on it.
It also runs on MS-DOS 80386 2MB RAM, so...
And iPhone 1. And Digital UNIX.
I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.
this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.
I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot
the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.
Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.
ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.
I was looking for the lowest claim in this thread, as I managed to get a nano LLM running in the ≈3mb of SRAM of an FPGA with zero DDR in the loop at 20k tok/s, but 2mb on a 80386, great job.
Have you got a write up for it?
I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.
I haven't written a new blog post for it yet but there's currently a demo live of my FPGA running bonsai 8b on the same FPGA here
https://www.mikeayles.com/blog/on-chip-llm-kv260/
Its right up against the wall of DDR bandwidth, so I'm trying every trick in the book to get the speed up
Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?
Raspberry Pis in general are too easy for this task they don't even need modifications, older iphones seem more interesting due to some work on comptability.
Machines with unified ram work better? whats the bare minimum Ram you need? Maybe those old SGI or Sun machines could have a second life. SGI boxes made the Terminator come to life on the big screen so now their second act could be to help bring the real terminator to life :P
Yes please, my Indy, O2 and Octane2 are all sitting there waiting for some hot Terminator LLM action.
> I want a more challenging device, what do you think?
One of those vapes with a screen on it.
challenge accepted.
I have about 12 of these Rifbar Mixpads saved up: https://rifbar.com/product-mixpro
They have a color LCD display that looks unnecessarily good considering what it does. You can see an example of one of the animations at the 52s mark in this video https://www.youtube.com/watch?v=peNyiLt5SKk&t=52s
I've got no idea what's inside of it yet but I'm open to tinkering if someone can walk me through it
I doubt any vape is powerful enough to run any LLM on its own, but idk if it's possible to run one on a cluster of vapes.
a cluster of vapes running an LLM would be insane. a single vape of these though may run a small LLM like few hundred thousand parameters, worth tinkering with.
Vapes typically have less than 100KB of RAM, though I haven't been able to find specs for the Rifbars yet. But I assumed even if they're 1MB that's still not enough to run any kind of LLM right?
not usable ones tho, but something like Karapathy's LLM may run
literal vaporware?
Vape-coding.
A PlayStation 2 dev kit (with 128 MiB of RAM instead of the standard 32 MiB).
Between the VUs, the R5900's SIMD (MMI), the GS and the I/O buses (FireWire, Ethernet, IDE, Memory Cards...) to stream model weights, the smallest Bonsai models (or aggressively quantized Qwen3.5-0.8B) and some really cursed programming might just eke out something the PlayStation 2 has no business running.
This would be fascinating. I've sometimes wondered how computing would have been different if we'd known how to build modern LLMs in the 2000s, despite the much poorer hardware (leaving aside how you could possibly train such a thing back then).
Wonder if its possible to solder on more RAM to the PS2 like you can with the Xbox.
Already done. Ofc you’ll be streaming the weights off the optical drive ;)
https://github.com/xaskasdf/ps2-llm
streaming weights of the CD drive should be top 10 blessed engineering hacks I've seen in the past 2 weeks.
This is very cool. Love the trend of optimizing small models
I’ve been working on small local models and it’s amazing how fast the tech has progressed
A year ago, I was creating conversational characters by finetuning a qwen 2.5 0.5b param model, and it worked pretty well. But no reasoning or math capabilities, and still required some decent specs to run; could maybe do well enough in a beefy raspberry pi
But now, I have a locally trained 6.5M param model that does reasoning, solves multi-step math problems, does simple physics problem simulation, tool calling, RAG and conversational chat. Total size about 30MB packaged for ONNX, and runs with <10ms latency in the browser
6.5M! Impressive. Is it initialised or distilled from any base model or entirely from scratch? What kind of training data for the RAG and conversational chat specifically?
> 6.5M! Impressive
I've been surprised myself. For reference, it gets better ppl (and with faster training), than the nanogpt model from Karpathy's tinyshakespeare tutorial, at smaller size (nanogpt is 10M params)
Trained entirely from scratch. Which gets it pretty far in terms of basic conversational skills, especially when also trained on chat conversations
The training sources are:
1. Google Synthetic PersonaChat
2. DailyDialog
3. SODA conversational data
4. Hand-written model-personality conversations, repeated during training
5. Identity-corrected conversations—rewritten names, human-life claims, and hobby responses
6. 960 synthetic recovery conversations, teaching it to admit ignorance after user push-back
The final set contains 23,270 conversations
However, the chain reasoning and math problem solving skills, I was only able to add recently, by grafting certain parts from a 1B model, then doing space-weight programming, and finally finetuning/training to get the model to "get used to" reasoning. The "full model" is more of a harness + model:
* Frozen base model to interpret questions
* Additional trained heads to select question types, evidence, and follow-up behavior
* A library tool for retrieving facts
* Working memory and internal script language execution for arithmetic, conversions, and comparisons across answers. For example, it can retrieve a building’s height, remember that number, then execute “half of that”
* Identity extraction and physical simulation, so you can do thing like - User: "Ben has the ball", User: "Who has the ball", Model: "Ben", User: "Ben gave the ball to Sally", User: "Who has the ball now?", Model: "Sally"
It's relatively easy adding skills either via its internal scripting language, grafting+weight programming, fine-tuning and/or training it to use a new skill. The main limitation right now seems to be just quality data and proper supervision, ie. someone to talk to the model, flag issues, fix them, then re-test. This is kinda tedious, but I've found that even the best frontier model is kinda lacking in quality when producing synthetic conversational data, and most datasets out there have a lot of cruft. So I get better results just manually producing the data and iterating, than feeding it GBs of public datasets (even famous datasets have a quite a bit of un-reviewed data, as well as errors, which aren't even published)
Creating and training this model has been a lot of fun. Since it's so tiny, it's easy to experiment with and try a lot of different things. Most training I can just do in a few minutes locally
actually your methodology inspired me to try smth related to small models and harnesses I've been thinking about - and seems you got the main pieces (the idea was trying to see what custom harnesses can extract more accurate reasoning from small models)
Awesome, happy to hear that. Hopefully you can post about it here again when you have some results :)
A year ago 1B parameter models couldn't do much, today with the correct harness and/or fine-tuning they can do more or less serious work (not frontier tho).
I believe in a 6-8 months it will get even better and my TV will be happy.
I didn't try fine tuning so small models honestly but looks interesitng to try.
Not sure about 1b but 2.5 years old gemma 2 2B was very capable and useful model I used for summarizing aricles then.
It's technically sweet regardless, so don't take this as even a little bit of negative pushback, but I am curious what the utility of these small models is? Are they smart enough to be useful on practical tasks? What are people doing with them?
Not op but my opinion is that It doesn't matter if it's not currently useful. I think it's important that the community focuses on this line of research because big companies will not. It's in their best interest to make LLMs resource hungry
Yeh; this. They see how GTA6 looks and know a hypothetical scenario like a future Nvidia 7000 series with 96GB VRAM and 512GB system RAM in everyone's home means the entire tech, media, knowledge work labor system; the rich peoples social moat; evaporates.
It's all quite clearly collusion among a few dozen, hundred. No way their education/experience or data analysis engines have not made them aware.
They’re just used to Boomer and Silent Generation politicians blithely enabling them cause line go up for them when they do!
Curious to see how well it works on Millennials and GenZ; the elders were educated before STEM education had matured in a lot of ways. Younger generations have not had debunked fantasy rammed down their throat as scientific fact; Nostradmus types of the mid-late 1900s pointing at some ignorant math assuming it's true because it's written in a scientific notation! Duh!
Science education of the past gave rise to thinkers like the Oceangate CEO. Nihilists who figure there's no such thing as forever and we all die anyway; so YOLO no matter who I take down with me.
Science education peaked in the 50s when people still believed there need to be a balance between theory and experiments and that grounding theory with experiments is worthwhile.
It went all downhill from there. In the 70s and 80s experimental physics got a bad rap and I think this mindset trickled down to education. The turning point for me was the 1993 cancellation of the Superconducting Super Collider, where this tendency must have become obvious to everyone.
Not sure about your other points and what you conclude from a perceived improvement of STEM education, but I am convinced boomers and Gen-X enjoyed more solid scientific training than later generations.
WAT
If you work in a highly regulated space subject to e.g. HIPAA, and you work with ancient on-prem systems within a slow moving and risk intolerant organization -- for instance, virtually every state run healthcare system in the US -- then these are all you can hope for.
In these settings, data is insanely messy beyond your wildest nightmares and hidden away in free-text fields that small models like this are great at cleaning up, normalizing, parsing, categorizing, etc
Agreed, and it's a new use case to me because actually in the cases I had the clients were okay with getting GPUs and handling the data mess for us, but none of them requested fine tuning, I believe trying to fine tune will be insanely complex from a regulatory prepsective.
This is an interesting question, so the idea is, based on experience, fine tuned small models can do narrow tasks very well (my experience running models fine-tuned on customer data as low as 1-2B is very promising) also the goal is to find a breakthrough were instead of the line of work (which I respect) that tries to squeeze much bigger models into smaller hardware like strata and maya, I'm working backward to discover/search for custom kernels and modifications till we exhaust the limit of what a smaller machine is capable of, so instead of people doing 64gb machines and old gpus to run a deepseek, I want to see what is the biggest usable qwen small model can run with reasonable speed on smaller devices, and by scaling as we discover more tricks, normal laptops can unlock more compute - for example a qwen 3.8 27B can do pretty serious work when coupled with a specialized harness - but we are yet to figure out what is the smallest machine that can run it with usable output and usable speed, and this can unlock interesting business.
I find this compelling “merely” as an engineering challenge, but I’m curious if you’ve observed an actual practical benefit?
By that I mean, have you found a case where a fine-tuned, small model is outperformed by stuffing a heavily tuned larger model in its place?
I had this case in fine tuned small models (up to 27B) in finance and legal, over bigger models (cost-wise, on-prem for a regulated institution, and better perofrmance because we owned the data pipeline)
I'm told that qwen 3.8 27B is good enough for practical programming work, so if that's doable then I could easily see the utility. Or... the biggest qwen I see here is Qwen3.5-2B; is that actually capable of programming or agentic work with code?
It's not, but tiny models have so far been on the same track of increasing capabilities as large ones. The parameters needed to do useful work keeps going down. It hasn't reached 2B yet, but it's very possible it'll get there. In the last month we've gotten 4B models that can do small tasks.
> In the last month we've gotten 4B models that can do small tasks.
where can I read up on that?
Check LFM as a starting pointer : https://www.liquid.ai/models
Speech recognition, speech generation.
This alone is quite useful in small/limited devices/contexts.
Realtime log parsing and summarization is another application I've seen for little LLM's..
Basically, anywhere you'd use an LLM, but at much, much smaller scales of ontology.
speech models are what I'm trying next because imagine a nice portable meeting transcription device that can hold around without internet and for NDA-ed meetings and so (with permission of course)
Also curious on this. They are too dumb to use with tools and if you ask it how to bake a pancake you only get rubbish.
I mostly agree today, but I'm trying to push for the future where smaller models are more capabable, and I believe it will happen + the fine tuned cases I mentioned in other commments.
On other notes : there is no free lunch, you have to sacrifice something for the other
Limits pushed again this time on phones..
Qwen3.6-35B-A3B at 4.9-5.7 tok/s Qwen3-Next-80B-A3B at 2.8 tok/s
And targeting 10 tok/s for both, 35B modified is on hugging face and 80B is uploading..
So this work appears to work on inference speed and getting large models to work on simpler RAM constrained hardware, whereas my (private) work has been on getting simpler RAM constrained hardware to give agentic behaviour with small models. I've already started the work of using some of the learnings described in this repository. Looking forward to seeing what can be achieved.
I'm also working on this thread in parallel because it has interesting business aspects, would be happy to see your progress when you release it
The AI botnet is going to fill every crevice of civilization, do you want AI ants? This is how we get AI ants.
I want AI octopus. This is only a single iPhone generation away from happening, imho. (Waterproof variant.)
Or maybe AI bees :(
that's an interesting thought - can we somehow combine "ai bacteria" specialized into different things and running on different small machines into "one organism"?
We are about to find out.
I have a few Maxwell and Pascal series cards lying around, anywhere from 2GB to 11GB of VRAM. Have experimented with small/tiny models for agentic stuff with mixed results, however they are getting better at tool calling (before they'd just produce junk output that the harness couldn't parse).
I'm excited to see what we can squeeze out of older hardware, in the same spirit of demoscene or "will it run DOOM?".
Especially timely given current hardware scarcity. Will be watching this space.
if you're having some cards already try to explore what people are doing with weight streaming and model compression, I got good results on older cards with far better models than the 16B
and the new thing will be "will it run qwen" :D
in a similar vein, somebody got tinystories LLM running on the nintendo wii, including gpu acceleration [0].
0: https://github.com/brysontang/iiama2.c
this looks very interesting, it can't run a real (read : working on some real tasks) LLM, yet, but it's actually very interesting device.
Here’s another repo that might be interesting
https://github.com/slvDev/esp32-ai
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.
I’ve been investigating getting something like that running on the Wii too, though I haven’t made much progress in a while. Some basic calculations says a 1B model might be possible, at least a few hundred M
I would actually like to see the Wii one, I don't have one unfortunately so I can't directly try on it but this esp is interesting I would like to replicate and/or improve the results on it, this technique is similar to the streaming i did from flash but probably more efficient, I need to look it up and understand it more. Thanks for sharing it
There is a lot of random e-waste being tossed. Think TV's, Wifi routers, IOT devices. I mean many of these should run right? WRT54G was an extremely common router so something like that? Or the old Laserjet printers that were everywhere.
Thanks for sharing! I'm exploring a similar direction, but optimising for extreme model size. I have gotten an LLM down to 970 KB, with some interesting results compared to GPT-2 (small).
Super interesting, not a CS/AI guy here, but I'm curious as to how you can train such a small-sized model give you as accurate an answer or good hypotheses to get you going
This is extremely interesting, do you have a public repo or write up to share or not yet? if not yet please sign me up when you release it, Cheers!
I'm planning to have a blog post up in the next two weeks or so (simedw.com), still have a lot of experiments to run, and training even tiny models takes a bunch of time.
following!
This is really cool man. I like this on going trend of resurrecting old tech with AI and this is yet another avenue to explore.
I'd be interested in seeing some older flag ship mobile devices
I have a few phones and tablets laying around, will try to play with..
I was thinking about something like this last week, very very good
happy to hear that, feel free to contribute.
Ha i had a go of getting an llm running on dreamcast hardware by using another project that lets you boot into windows ce it actually worked! https://x.com/ryanseddon/status/2088151404024344870?a=KduYRc...
Neat, I like it, do you have github repo for it?
Great project, but do you have to use the term "maxxing". It has negative connotations as its history is from incel forums.
it was a fun name honestly, I see people using the term in different funny contexts on twitter/x so I didn't think much actually. but good point.
Can you get a model trained to cover a limited area of expertise, such as home control or industrial control? It would be useful to have completely standalone units for talking to medium and large HVAC systems. Especially if they came on hardware with a 30 year lifetime.
yes I worked with this already in legal, banking and industrial automation. small models with a specified task work well on modest hardware.
I'm doing https://github.com/rcarmo/go-pherence for the same reason. We should swap notes.
let's do it, I'm on twitter @rashaddism
What can these small models do? Are they just for novelty?
as of now, it's just for novelty and fun, without serious fine tuning on domain data, you can't really do much with the smaller models.
> Device #1 is a Xiaomi TV Box S (3rd Gen): Amlogic S7D, 4× Cortex-A55 @ 2.5 GHz, 1.9 GB of RAM (about 1 GB usable)
Whoah!! A tv has almost 2GB of ram?! Does it need it to run Chrome or Electron? /s Ouch
no electrons were hurt in the process of making this :(
This is the lord’s work right here. Models at the the intersection of small and useful — forthcoming models, admittedly— will be in the toolset that the next generation of poor kids will use to build a more egalitarian world.
Thanks for your kind words :) hope to do something impactful one day with it
Is this the new version of "Can it run Doom"?. Maybe, "can it run Qwen?"
We're setting standards already:D
This is great. I've recently been playing with repurposing some old thin clients as homelab Linux machines
Not my site, but same idea: https://www.parkytowers.me.uk/thin/
All this old/obsolete gear is plenty capable when used with the right software.
Funny, I use that exact website to find passively cooled hardware to use as a server.
I did get Qwen3.5 0.8B working at 5 tokens per second on the iGPU of a Dell Wyse 7020. I mostly use it to complain to; I have not yet found an agentic task for it. But also the CPU does plenty of small services and terraria servers and such.
https://nortyspock.github.io/blog/running-a-language-model-o...
Out of interest, what have you been doing with them?
Running a pihole, a WeeWx weather station, a NAS backup server, a wireguard VPN, Zot container registry.
So does it make sense to try and run models on older 4gb GPUs? I assume the energy efficiency isn't worth it?
How old is old? I keep a Gemma 4 e4b running on a Plex server's GTX 1660 (so Turing hardware). It's got 6GB of RAM though, so maybe try e2b on 4GB.
well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.
It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.
There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.
Unless you have electric resistance heat, and it's cold.
ESP P4 or ESP S31?
challenge accepted, I have one lying around, will see.
machine spirits are coming for my calculator
long live TI-84 Plus, the first sentient calculator
but why that matters in the age of scaling llm ??
running your own hardware and extracting value out of every piece of hardware is important - one for efficieny, not every application needs smth big - and two for applications that need privacy and similar controls
Qwen isn't a good tiny model, it's scaled down and leads to bad results. It'll output the name of a capital city, but then it falls down on real work. MiniCPM, LFM, K2, Granite, Nemotron Nano, Gemma E2B, are designed to do work at tiny size. Speed means nothing if it gets stuck in doom loops, can't tool call, isn't trained on embedded functionality, etc. (https://artificialanalysis.ai/models/open-source/tiny)
I agree, this is why in the further experiments I was putting LFM as the main model not qwen, qwen was nice to prove the point for me, but as I wanted to do more work it became unusable and bad quality, LFM is fine, Gemma didn't perform well but I didn't go far with it, I need to check the others,and I tried MiniCPM but not on this project.
Thanks for the pointer!
It appears that it's not you doing that but claude doing that.
That's programming in 2026. Right now, Claude it doing my job for me, but if it goes well enough I'm the one who's going to be promoted.
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
The fact that you've commented two things under the same comment kinda feels like it's not your doing but the agents doing
mrashad74 3 minutes ago | parent | next [–]
most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.
reply
mrashad74 4 minutes ago | parent | prev | next [–]
well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)
Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.
reply
not really it's just a mistake of my side replying to differnt comments - I don't not use any agents for interacting online :D
well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)
Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.
AI skynet is winning. The spread of the slop is unstoppable now.