was expecting a bit more substance but always interesting to read different approaches.
but I do wonder if BEAM is a bit of an overkill, even fly.io (known among developer threads to be very unreliable uptime) wouldn't be my first pick.
Cloudflare Durable Objects really is generous, not often can you get replicated database and realtime sync for such low barrier in cost and implementation.
I do wonder tho what Cloudflare's ultimate game is, as I'd imagine once they corner the market and practically everything is running off it, wouldn't be hard to say raise the $5/month to something higher.
> Cloudflare Durable Objects really is generous, not often can you get replicated database and realtime sync for such low barrier in cost and implementation.
BEAM gives it to you for free, and you don't rely on an Internet-scale monopoly to run it.
I love the BEAM and programmed on it for many years, but it really does not provide anything like durable objects.
1) It's very difficult to ensure globally serialized ownership with strong consistency in a distributed Erlang cluster when nodes are allowed to fail. Stuff like Horde will let you do some rough "run an instance of this process somewhere in the cluster", but it's eventually consistent (you may have multiple instances at times) and doesn't deal with netsplits well.
2) Mnesia is fine to replicate state within a network switch or very reliable LAN, but not over WLAN/Internet. It can enter split brain conditions and require external reconciliation. RabbitMQ suffered from Mnesia problems for many years and ended up replacing it with their own DB implementation using the Raft protocol.
See these are things I did not know. This is why I said I was 'not qualified', in the post, haha. I do use Horde for the matchmaking. I have 2 hosts in the cluster.
Yeah I could've been more technical. Maybe in another post. On the fly issue, I love an underdog story. I know people have complained a lot about their uptime. I suspect they are grinding very hard on that. I've not really had any issues. They also seem to be a popular elixir host so I thought why not. Deploying is heroku style, "fly deploy".
> but I do wonder if BEAM is a bit of an overkill...
Why would it matter if something's overkill? If it's not much more work, and it runs on the systems you're targeting, it's better to start with something that's more reliable than you might strictly need than to start with something that you believe to be "proper kill", only to discover that you underestimated the amount of "kill" required. Retrofitting things like proper supervision and data/crash isolation can be a huge, huge pain in the ass.
> Also, I would encourage others to target Mac in addition to iOS for a reason you might not expect: build times. The simulators and Xcode are really pretty slow. It’s all a lot faster if you’re targeting Mac.
Yes, this makes my day: (also) target macOS for continuous rapid loops on one project, or target iOS and end up flitting back and forth between many projects. ("mac (made for iPad)" does not work.)
Yeah, I do the same thing with Flutter when I'm in "rapid iteration mode" and build for desktop Linux in my case, though wireless debugging on Android is almost fast enough it's more that I don't want to dart back and forth between my laptop and phone.
I've been really enjoying retro games in the form of '80s arcade games lately. Last weekend I had Claude build me a web-based Rally-X inspired game and the experience was pretty good. I like that I can just drop it into a github.io and be done with it, but that isn't going to work for a game that needs a backend. https://linsomniac.github.io/rally-xy/
I'm now working on a "nsnipes" game, and that, because of the multi-player aspect, is going to need a backend server.
Thanks for the write up. Wondering why not use something like Flutter for good cross platform support with good graphical performance on all systems.
One could argue that with agentic programming, releasing on a new platform is simply a matter of prompting for a new target, but I fear that the non-determinism of LLMs will lead into a whac-a-mole situation where you’re constantly fighting framework specifics and the code that the agent spit out to get to pixel perfect solutions across the platforms. Flutter in this case would be a more token efficient solution and quite fast on most platforms (particular the ones where their Impeller engine runs).
A question for OP: why did you choose Crunchy Data for your database, instead of Fly's own managed postgres offering? Because the latency between Fly and Crunchy Data must be quite high, given that they are probably not in the same datacenter.
Really interested in hearing more about the architecture. Especially (1) why you chose Elixir, Phoenix over TS (2) how you dealt with real-time multiplayer.
Aside: The links to the web game in the post don't lead to being able to play.
Not OP, but TS (as in TypeScript, right?) is not even in the same universe as Elixir with Phoenix when it comes to building backend services.
It's a very productive and readable programming language with excellent documentation and conventions, and the most ergonomic way of handling concurrent operations (thanks BEAM) I've encountered.
The VM it runs on was originally designed for telephone switches, which, it turns out, cleanly translates to the internet/http era.
It makes it trivial to do soft-realtime because it's just actors (GenServers) passing messages.
I invite you (and others) to try it out and do a small weekend project. It'll make you reconsider reaching for TS on the backend :)
Thanks, interestingly Elixir handles concurrency in what seems like a natural way, actors passing messages to each other. It's cool that this way is used in production, though it seems to be used mainly in niche distributed scaling.
What people are finding now when building agenic AI orchestrators is that they are reinventing some of the ideas behind BEAM and OTP. The big one being queues (mailboxes), but also fault isolation.
Async runtimes can’t really do preemptive scheduling so those end up using other methods that, while getting low latency, may not improve reliability or reduce variance in latency.
Orchestrating tasks across a number of different services that are not in your control becomes a distributed system where not everything is reliable.
Hi! I'm sorry I took down the web client shortly after I write this in order to narrow my focus on the client codebase. I realize I ruined my point about being able to compare the client performance. I'm editing the post shortly.
I've been curious about elixir for years. Since maybe 2016. I tried in the days before AI, but I couldn't really get over the hump of syntax and BEAM and all of that.
I'd read it was the kind of thing that would be good for games, but I didn't really understand why until AI helped me build this.
A room is a GenServer. Which means its a process. Which means the unit of gameplay matches the unit of compute, kinda. That's really great. You can't really do that with node. Well, I guess with Durable Objects / PartyKit you have a closer match there.
There is a matchmaker process that runs. It assigns your socket to a room. But yeah, after that, your socket is exchanging messages with that room.
Because arrow physics are simple, the client predicts how it thinks the server will say things will be, and reconciles if / when wrong when it hears from the server about the room state.
As a TS dev also who wouldn't really think to consider Elixir, that's interesting.
To sum it up, it seems that Elixir naturally provides the distributed routing and computing model needed for realtime communication, while you'd need specific libraries and infrastructure with TS. Thanks!
not op either but I built my entire startup using elixir. We have a realtime sync component across a range of devices that sync data between them.
It (phoenix channels) worked out of the box. in the last 6 years of development and growth, not once have we ever had to even discuss scaling issues related to it.
It just works and when you're in a startup, the more you can focus on growing your product instead of scaling issues, the more you're winning.
Phoenix is delightfully fast and not having to deal with two entirely different application stacks where your application is split down the middle (or the javascript ecosystem) is a breath of fresh air.
In my own projects I've found the feedback loop to be much slower in native mobile apps compared to web. For web apps my workflow has involved lots of human verification just through using the app myself and testing out the latest features/fixes. Sometimes I get the agent itself to do this before I do using Codex's chrome extension. This same process on mobile is a lot slower. How have you approached this aspect? Have you figured out a way to get the agent to control an emulator?
Nope. I guess I should try that at some point. I’d be looking into interacting with the Mac build for the performance reasons mentioned in the post, that iOS sims are slowww
I'm sorry I took down the web client shortly after writing this post. Really ruins the point I made of being able to contrast web vs native performance, doesnt it? I realized I wanted to narrow my focus on client codebases. Fixing the post shortly.
Yet, there is hardly any Web game that it is at the level of Infinity Blade, the game used to show off OpenGL ES 2.0 support back in 2010.
Browsers 3D support is a black box for game developers, still no debugging tools other than having a native build that you can plug RenderDoc, not a priority for browser vendors for the past decade.
Then there is the whole issue how assets get loaded, space on browser for caching them, and security (F12 away to mess with the game engine).
was expecting a bit more substance but always interesting to read different approaches.
but I do wonder if BEAM is a bit of an overkill, even fly.io (known among developer threads to be very unreliable uptime) wouldn't be my first pick.
Cloudflare Durable Objects really is generous, not often can you get replicated database and realtime sync for such low barrier in cost and implementation.
I do wonder tho what Cloudflare's ultimate game is, as I'd imagine once they corner the market and practically everything is running off it, wouldn't be hard to say raise the $5/month to something higher.
> Cloudflare Durable Objects really is generous, not often can you get replicated database and realtime sync for such low barrier in cost and implementation.
BEAM gives it to you for free, and you don't rely on an Internet-scale monopoly to run it.
I love the BEAM and programmed on it for many years, but it really does not provide anything like durable objects.
1) It's very difficult to ensure globally serialized ownership with strong consistency in a distributed Erlang cluster when nodes are allowed to fail. Stuff like Horde will let you do some rough "run an instance of this process somewhere in the cluster", but it's eventually consistent (you may have multiple instances at times) and doesn't deal with netsplits well.
2) Mnesia is fine to replicate state within a network switch or very reliable LAN, but not over WLAN/Internet. It can enter split brain conditions and require external reconciliation. RabbitMQ suffered from Mnesia problems for many years and ended up replacing it with their own DB implementation using the Raft protocol.
Chris McCord just recently released a library trying to solve for this https://github.com/phoenixframework/durable_server
See these are things I did not know. This is why I said I was 'not qualified', in the post, haha. I do use Horde for the matchmaking. I have 2 hosts in the cluster.
BEAM is rarely overkill. It’s fairly simple, once you grasp the concepts. It’s just from the outside people may think it’s rocket science.
Yeah I could've been more technical. Maybe in another post. On the fly issue, I love an underdog story. I know people have complained a lot about their uptime. I suspect they are grinding very hard on that. I've not really had any issues. They also seem to be a popular elixir host so I thought why not. Deploying is heroku style, "fly deploy".
> but I do wonder if BEAM is a bit of an overkill...
Why would it matter if something's overkill? If it's not much more work, and it runs on the systems you're targeting, it's better to start with something that's more reliable than you might strictly need than to start with something that you believe to be "proper kill", only to discover that you underestimated the amount of "kill" required. Retrofitting things like proper supervision and data/crash isolation can be a huge, huge pain in the ass.
> Also, I would encourage others to target Mac in addition to iOS for a reason you might not expect: build times. The simulators and Xcode are really pretty slow. It’s all a lot faster if you’re targeting Mac.
Yes, this makes my day: (also) target macOS for continuous rapid loops on one project, or target iOS and end up flitting back and forth between many projects. ("mac (made for iPad)" does not work.)
Yeah, I do the same thing with Flutter when I'm in "rapid iteration mode" and build for desktop Linux in my case, though wireless debugging on Android is almost fast enough it's more that I don't want to dart back and forth between my laptop and phone.
I've been really enjoying retro games in the form of '80s arcade games lately. Last weekend I had Claude build me a web-based Rally-X inspired game and the experience was pretty good. I like that I can just drop it into a github.io and be done with it, but that isn't going to work for a game that needs a backend. https://linsomniac.github.io/rally-xy/
I'm now working on a "nsnipes" game, and that, because of the multi-player aspect, is going to need a backend server.
Thanks. That just killed 15 minutes of my working day. Much appreciated. I just needed that.
Love it. Went into my bookmarks for later tonight.
I should've said more about the tech. I will later. The server is in NJ. If you're in the EU the game might really lag. Sorry!
You mention it is possible to play via Web, but all links just lead to pages which point to Apple's app store. Game over for Android and the web?
Yeah. Shortly after I wrote this I shut down the web client. I wanted to focus on the other clients. Sorry. Kind of ruined one of my points
Thanks for the write up. Wondering why not use something like Flutter for good cross platform support with good graphical performance on all systems.
One could argue that with agentic programming, releasing on a new platform is simply a matter of prompting for a new target, but I fear that the non-determinism of LLMs will lead into a whac-a-mole situation where you’re constantly fighting framework specifics and the code that the agent spit out to get to pixel perfect solutions across the platforms. Flutter in this case would be a more token efficient solution and quite fast on most platforms (particular the ones where their Impeller engine runs).
Historically, iOS has been a second class citizen performance wise in Flutter. I certainly wouldn't rely on it as a gaming company.
My (possibly outdated) take is that Flutter is all about Dart, and Dart is a Google thing w/ virtually no other adoption.
A question for OP: why did you choose Crunchy Data for your database, instead of Fly's own managed postgres offering? Because the latency between Fly and Crunchy Data must be quite high, given that they are probably not in the same datacenter.
Honestly? It's cheaper. hahaha.
Really interested in hearing more about the architecture. Especially (1) why you chose Elixir, Phoenix over TS (2) how you dealt with real-time multiplayer.
Aside: The links to the web game in the post don't lead to being able to play.
Not OP, but TS (as in TypeScript, right?) is not even in the same universe as Elixir with Phoenix when it comes to building backend services.
It's a very productive and readable programming language with excellent documentation and conventions, and the most ergonomic way of handling concurrent operations (thanks BEAM) I've encountered.
The VM it runs on was originally designed for telephone switches, which, it turns out, cleanly translates to the internet/http era.
It makes it trivial to do soft-realtime because it's just actors (GenServers) passing messages.
I invite you (and others) to try it out and do a small weekend project. It'll make you reconsider reaching for TS on the backend :)
Thanks, interestingly Elixir handles concurrency in what seems like a natural way, actors passing messages to each other. It's cool that this way is used in production, though it seems to be used mainly in niche distributed scaling.
What people are finding now when building agenic AI orchestrators is that they are reinventing some of the ideas behind BEAM and OTP. The big one being queues (mailboxes), but also fault isolation.
Async runtimes can’t really do preemptive scheduling so those end up using other methods that, while getting low latency, may not improve reliability or reduce variance in latency.
Orchestrating tasks across a number of different services that are not in your control becomes a distributed system where not everything is reliable.
Not at all. It’s used to build standard web applications too.
Hi! I'm sorry I took down the web client shortly after I write this in order to narrow my focus on the client codebase. I realize I ruined my point about being able to compare the client performance. I'm editing the post shortly.
I've been curious about elixir for years. Since maybe 2016. I tried in the days before AI, but I couldn't really get over the hump of syntax and BEAM and all of that.
I'd read it was the kind of thing that would be good for games, but I didn't really understand why until AI helped me build this.
A room is a GenServer. Which means its a process. Which means the unit of gameplay matches the unit of compute, kinda. That's really great. You can't really do that with node. Well, I guess with Durable Objects / PartyKit you have a closer match there.
There is a matchmaker process that runs. It assigns your socket to a room. But yeah, after that, your socket is exchanging messages with that room.
Because arrow physics are simple, the client predicts how it thinks the server will say things will be, and reconciles if / when wrong when it hears from the server about the room state.
As a TS dev also who wouldn't really think to consider Elixir, that's interesting.
To sum it up, it seems that Elixir naturally provides the distributed routing and computing model needed for realtime communication, while you'd need specific libraries and infrastructure with TS. Thanks!
not op either but I built my entire startup using elixir. We have a realtime sync component across a range of devices that sync data between them.
It (phoenix channels) worked out of the box. in the last 6 years of development and growth, not once have we ever had to even discuss scaling issues related to it.
It just works and when you're in a startup, the more you can focus on growing your product instead of scaling issues, the more you're winning.
I’d guess that 2 is a huge part of the answer to 1 here - Elixir makes real-time multiplayer easy.
Phoenix is delightfully fast and not having to deal with two entirely different application stacks where your application is split down the middle (or the javascript ecosystem) is a breath of fresh air.
Isn't split stack architectural, i.e. client heavy vs thin client. Not a hard TS feature?
With Phoenix you can write actually 0 code that runs on the client, just the backend and html templating.
In my own projects I've found the feedback loop to be much slower in native mobile apps compared to web. For web apps my workflow has involved lots of human verification just through using the app myself and testing out the latest features/fixes. Sometimes I get the agent itself to do this before I do using Codex's chrome extension. This same process on mobile is a lot slower. How have you approached this aspect? Have you figured out a way to get the agent to control an emulator?
Nope. I guess I should try that at some point. I’d be looking into interacting with the Mac build for the performance reasons mentioned in the post, that iOS sims are slowww
Maestro but honestly you should always be testing on real devices
I'm sorry I took down the web client shortly after writing this post. Really ruins the point I made of being able to contrast web vs native performance, doesnt it? I realized I wanted to narrow my focus on client codebases. Fixing the post shortly.
My post about my recent project, Migo Games
I’ve never thought about how the room model matches the LiveView socket model before. Which socket library are you using?
a room is a genserver. For the socket itself, https://github.com/davidstump/SwiftPhoenixClient
funny enough - distribution on the web would've been easier than the App Store.
likely - it would grow organically too since all one has to do is share a link.
for social games yeah - Elixir | BEAM is an excellent stack.
Yet, there is hardly any Web game that it is at the level of Infinity Blade, the game used to show off OpenGL ES 2.0 support back in 2010.
Browsers 3D support is a black box for game developers, still no debugging tools other than having a native build that you can plug RenderDoc, not a priority for browser vendors for the past decade.
Then there is the whole issue how assets get loaded, space on browser for caching them, and security (F12 away to mess with the game engine).
I see your point. My counterpoint is that I think I’ve returned to zero web games that aren’t wordle. I think in the long run it’s the wrong choice