Hey guys, I had a few esp32 boards around for a while and always thought - how do we go beyond just one device, would it be possible to apply cloud-native thinking to distributed edge computing?
Result is Kube32. I got the cluster and a full-stack web app with real-time sync running on esp32 s3 dev board and cardputer - https://7f0c-180-74-71-21.ngrok-free.app/ (may go down) (should support at least a few concurrent users - break it)
You won't see one cool thing though, which is the hud pod that uses Cardputer as a keyboard that syncs keystrokes to all clients in real-time.
Honestly, it opens so many possibilities for distributed edge computing and multi-device systems. Let me know what use-cases do you think would benefit from kube32 the most.
Thank you for working on this. The most impressive part for me is getting Kubernetes running on such low spec hardware. I can't think of a use case I have for Kube32 at the moment, but I do have use cases for Kubernetes running on a small amount of RAM. Is there any chance of porting this back to x86/x86-64 PCs or ARM SBCs? For example, being able to run this on a cluster of Raspberry Pis would be interesting as it would free up more RAM for applications.
I feel like a gradualist approach may be better for those applications. Start with "vanilla" k8s, and iteratively target subsystems to shrink down. Either by porting to Rust or simply optimizing the Go code for low RAM environments.
Hmm, have you tried k3s? It promises low ram, but it could still be too much, I'm not sure about your requirements. In the meanwhile I'll see if it can be pushed further, but I guess we both know the answer :)
Super cool! One thought that would make this way more useful to me (and probably a thousand times more complicated, sorry) is if it had a way to split out “GPIO” subroutines from the main app, and schedule them separately. So GPIO would become an addressable unit like pods or load balancers in k8s.
The idea is I could specify something like “I have this app that needs to pull data from a GPIO labeled as a ds18b20 and outdoors” and it takes care of scheduling my app, starting a ds18b20 reader, and networks them together. I never need to know if they’re on the same esp32, separate ones, or in a crazy alternate world the compute bit goes on a VM on a laptop with no GPIOs and the sensors go on a real esp32.
Having GPIOs be an addressable resource would be so cool. Like if nothing else, I could start a new esp32 project without wiring up a breadboard. Most of my stuff is some combination of the same 10 chips anyways.
- trustless code execution on someone's hardware (kube32 supports gpio permissions, amongst other stuff)
- your standard Kubernetes cluster can now reach the physical world natively
- OTA updates at arbitrary scale
- fault-tolerance in the physical world, orchestration of workloads based on physical proximity or other sensor data
Exactly, and I discarded the idea of porting k8s, building k8s-compatible system instead. It still carries some overhead, yes, and I'm yet to measure it fully, but it's a good trade-off for the applications it unlocks. If we remove the overhead form equation, do you think the capabilities I mentioned are unlocked with k8s-like system, or do you have a way to achieve all of them at the same time using something else in mind?
Hey guys, I had a few esp32 boards around for a while and always thought - how do we go beyond just one device, would it be possible to apply cloud-native thinking to distributed edge computing?
Result is Kube32. I got the cluster and a full-stack web app with real-time sync running on esp32 s3 dev board and cardputer - https://7f0c-180-74-71-21.ngrok-free.app/ (may go down) (should support at least a few concurrent users - break it)
You won't see one cool thing though, which is the hud pod that uses Cardputer as a keyboard that syncs keystrokes to all clients in real-time.
Honestly, it opens so many possibilities for distributed edge computing and multi-device systems. Let me know what use-cases do you think would benefit from kube32 the most.
Thank you for working on this. The most impressive part for me is getting Kubernetes running on such low spec hardware. I can't think of a use case I have for Kube32 at the moment, but I do have use cases for Kubernetes running on a small amount of RAM. Is there any chance of porting this back to x86/x86-64 PCs or ARM SBCs? For example, being able to run this on a cluster of Raspberry Pis would be interesting as it would free up more RAM for applications.
I feel like a gradualist approach may be better for those applications. Start with "vanilla" k8s, and iteratively target subsystems to shrink down. Either by porting to Rust or simply optimizing the Go code for low RAM environments.
True, for RPI it makes more sense. For microcontrollers it's a different story :D
I think it'd be great to put some effort into this, as a separate project perhaps.
Hmm, have you tried k3s? It promises low ram, but it could still be too much, I'm not sure about your requirements. In the meanwhile I'll see if it can be pushed further, but I guess we both know the answer :)
k3s does well on low ram but I've found it still uses a chunk of CPU.
Super cool! One thought that would make this way more useful to me (and probably a thousand times more complicated, sorry) is if it had a way to split out “GPIO” subroutines from the main app, and schedule them separately. So GPIO would become an addressable unit like pods or load balancers in k8s.
The idea is I could specify something like “I have this app that needs to pull data from a GPIO labeled as a ds18b20 and outdoors” and it takes care of scheduling my app, starting a ds18b20 reader, and networks them together. I never need to know if they’re on the same esp32, separate ones, or in a crazy alternate world the compute bit goes on a VM on a laptop with no GPIOs and the sensors go on a real esp32.
Having GPIOs be an addressable resource would be so cool. Like if nothing else, I could start a new esp32 project without wiring up a breadboard. Most of my stuff is some combination of the same 10 chips anyways.
Yes! This is the philosophy. Omw to ship!
What kind of overhead does the Wasm approach add in terms of memory and cpu time? I also have way too many ESP32 boards lying around
I'll measure and let you know later, would appreciate if can do measurements on your hardware too as I only have a few options to test on
cloud native emedded devices are an oxymoron
What's the benifit?
there are few imo:
- trustless code execution on someone's hardware (kube32 supports gpio permissions, amongst other stuff) - your standard Kubernetes cluster can now reach the physical world natively - OTA updates at arbitrary scale - fault-tolerance in the physical world, orchestration of workloads based on physical proximity or other sensor data
those are just a few that come to mind
None of this needs the overhead of k8s
Exactly, and I discarded the idea of porting k8s, building k8s-compatible system instead. It still carries some overhead, yes, and I'm yet to measure it fully, but it's a good trade-off for the applications it unlocks. If we remove the overhead form equation, do you think the capabilities I mentioned are unlocked with k8s-like system, or do you have a way to achieve all of them at the same time using something else in mind?
Slop