I’ve always found it crazy that my LLM has access to such terrible tools compared to mine.
It’s left with grepping for function signatures, sending diffs for patching, and running `cat` to read all the code at once.
I however, run an IDE and can run a simple refactoring tool to add a parameter to a function, I can “follow symbol” to see where something is defined, I can click and get all usages of a function shown at a glance, etc etc.
Is anyone working on making it so LLM’s get better tools for actually writing/refactoring code? Or is there some “bitter lesson”-like thing that says effort is always better spent just increasing the context size and slurping up all the code at once?
> I however, run an IDE and can run a simple refactoring tool to add a parameter to a function, I can “follow symbol” to see where something is defined, I can click and get all usages of a function shown at a glance, etc etc
I am so surprised that all of the AI tooling mostly revolves around VSC or its forks and that JetBrains seem to not really have done anything revolutionary in the space.
With how good their refactoring and code inspection tools are, you’d really think they’d pass of that context information to AI models and that they’d be leaps and bounds ahead.
Recently, all these agents can talk LSP (language server protocol) so it should get better soon. That said, yeah they don't seem to default to use `ripgrep` when that is clearly better than `grep`
Are you? I'm not surprised at all, considering that the biggest investment juggernaut in AI is also the author of VSC. I wonder what the connection is? ;)
> Claude Code officially added native support for the Language Server Protocol (LSP) in version 2.0.74, released in December 2025.
I think from training it's still biased towards simple tooling.
But also, there is real power to simple tools, a small set of general purpose tools beats a bunch of narrow specific use case tools. It's easier for humans to use high level tools, but for LLM's they can instantly compose the low level tools for their use case and learn to generalize, it's like writing insane perl one liners is second nature for them compared to us.
If you watch the tool calls you'll see they write a ton of one off small python programs to test, validate explore, etc...
If you think about it any time you use a tool there is probably a 20 line python program that is more fit to your use case, it's just that it would take you too long to write it, but for an LLM that's 0.5 seconds
Correct. If you try to create a coding agent using the raw Codex or Claude code API and you build your own “write tool”, and don’t give the model their “native patch tool”, 70%+ of the time it’s write/ patch fails because it tries to do the operation using the write/ patch tool it was trained on.
LLMs aren't like you or me. They can comprehend large quantities of code quickly and piece things together easily from scattered fragments. so go to reference etc become much less important. Of course though things change as the number of usages of a symbol becomes large but in most cases the LLM can just make perfect sense of things via grep.
To provide it access to refactoring as a tool also risks confusing it via too many tools.
It's the same reason that waffling for a few minutes via speech to text with tangents and corrections and chaos is just about as good as a carefully written prompt for coding agents.
Faster for worse results, though. Determining the source of a symbol is not as trivial as finding the same piece of text somewhere else, it should also reliably be able to differentiate among them. What better source for that then the compiler itself?
Yeah, especially for languages that make heavy use of type inference. There’s nothing you can really grep for most of the time… to really know “who’s using this code” you need to know what the compiler knows.
An LLM can likely approach compiler-level knowledge just from being smart and understanding what it’s reading, but it costs a lot of context to do this. Giving the LLM access to what the compiler knows as an API seems like it’s a huge area for improvement.
It depends on the language and codebase. For something very dynamic like Python it may be the case that grepping finds real references to a symbol that won’t be found by a language server. Also language servers may not work with cross-language interfaces or codegen situations as well as grep.
OTOH for a giant monorepo, grep probably won’t work very well.
You can give agents the ability to check VSCode Diagnostics, LSP servers and the like.
But they constantly ignore them and use their base CLI tools instead, it drives me batty. No matter what I put in AGENTS.md or similar, they always just ignore the more advanced tooling IME.
Not coding agents but we do a lot of work trying to find the best tools, and the result is always that the simplest possible general tool that can get the job done always beats a suite of complicated tools and rules on how to use them.
LSP also kind of sucks. But the problem is all the big companies want big valuations, so they only chase generic solutions. That's why everything is a VS Code clone, etc..
This isn’t completely the answer to what you want but skills do open a lot of doors here. Anything you can do on a command line can turn into a skill, after all.
If you are willing to go language-specific, the tooling can be incredibly rich if you go through the effort. I’ve written some rust compiler drivers for domain-specific use cases, and you can hook into phases of the compiler where you have amazingly detailed context about every symbol in the code. All manner of type metadata, locations where values are dropped, everything is annotated with spans of source locations too. It seems like a worthy effort to index all of it and make it available behind a standard query interface the LLM can use. You can even write code this way, I think rustfmt hooks into the same pipeline to produce formatted code.
I’ve always wished there were richer tools available to do what my IDE already does, but without needing to use the UI. Make it a standard API or even just CLI, and free it from the dependency on my IDE. It’d be very worth looking into I think.
Did you eval using screenshots or some sort of rendered visualization instead of the CLI? I wonder if Claude has better visual intelligence when viewing images (lots of these in its training set) rather than ascii schematics (probably very few of these in the corpus).
Computer use and screenshots are context intensive. Text is not. The more context you give to an LLM, the dumber it gets. Some people think at 40% context utilization, the LLM starts to get into the dumb zone. That is where the limitations are as of today. This is why CLI based tools like Claude Code are so good. And any attempt at computer use has fallen by the wayside.
There are some potential solutions to this problem that come to mind. Use subagents to isolate the interesting bits about a screenshot and only feed that to the main agent with a summary. This will all still have a significantly higher token usage compared to a text based interface, but something like this could potentially keep the LLM out of the dumb zone a little longer.
More to the point though, you should be using Agents in Claude Code to limit context pollution. Agents run with their own context, and then only return salient details. Eg, I have an Agent to run "make" and return the return status and just the first error message if there is one. This means the hundreds/thousands of lines of compilation don't pollute the main Claude Code context, letting me get more builds in before I run out of context there.
Claude helped me immensely getting an image converter to work. Giving it screenshots of wrong output (lots of layers had an unpredictable offsets that was not supposed to be there) and output as I expected it helped Claude understand the problems and it fixed the bugs immediately.
I had tried the browser screenshotting feature for agents in Cursor and found it wasn't very reliable - screenshots eat a lot of context, and the agent didn't have a good sense for when to use them. I didn't try it in this project. I bet it would work in some specific cases.
> As a mirror to real-world agent design: the limiting factor for general-purpose agents is the legibility of their environments, and the strength of their interfaces. For this reason, we prefer to think of agents as automating diligence, rather than intelligence, for operational challenges.
> The only other notable setback was an accidental use of the word "revert" which Codex took literally, and ran git revert on a file where 1-2 hours of progress had been accumulating.
If I tell Claude to "revert that last change, it isn't right, try this instead" and Claude hasn't committed recently it will happily `git checkout ...` and blow away all recent changes instead of reverting the "last change".
(Which, it's not wrong or anything -- I did say "revert that change" -- it's just annoying. And telling `CLAUDE.md` to commit more often doesn't work consistently, because Claude is a dummy sometimes).
Amazing that these tools don't maintain a replayable log of everything they've done.
Although git revert is not a destructive operation, so it's surprising that it caused any loss of data. Maybe they meant git reset --hard or something like that. Wild if Codec would run that.
I have had codex recover things for me from its history after claude had done a git reset hard, codex is one of the more reliable models/harneses when it comes to performing undo and redo operations in my experience.
Claude (can’t remember if was 4.1 Opus, 4.5 Sonnet, or 4.5 Opus) once just started playing with git worktrees and royally f-d up the local repo and lost several hours of work. Since then, I watch it like a hawk.
`git reset --hard` doesn't remove unreferenced commits or rewrite the reflog so I don't think that would do it. Something like `git reset && git gc` would have to be done.
Yea, it does so this would likely have been to be a `--yolo` (I don't care, let me `rm -rf /`). I've found even with the "workspace write" mode and no additional writable directories I can't do git operations without approval so it seems to exclude `.git` by default.
You're correct for an actual git revert, but it seems pretty clear that the original authors have mangled the story and it was actually either a "git checkout" or "git reset". The "file where 1-2 hours of progress had been accumulating" phrasing only makes sense if those were uncommitted changes.
And the reason jj helps in that case is that for jj there is no such thing as an uncommitted change.
> Having no such thing as an uncommitted change seems like it would be a nightmare, but perhaps I'm just too git-oriented.
Why? What's the problem you see? The only problem I see is when you let these extra commits pollute the history reachable from any branch you care about.
Let's look at the following:
Internally, 'git stash' consists of two operations: one that makes an 'anonymous' commit of your files, and another that resets those files to whatever they were in HEAD. (That commit is anonymous in the sense that no branch points at it.)
The git libraries expose the two operations separately. And you can build something yourself that works similarly.
You can use these capabilities to build an undo/redo log in git, but without polluting any of the history you care about.
To be honest, I have no clue how Jujutsu does it. They might be using a totally different design.
The problem is git's index let's you write a bunch of unconnected code, then commit it separately. To different branches, even! This works great for stacking diffs but is terribly confusing if you don't know what you're doing.
Yes, this exact scenario has happened to me a couple times with both Claude and Codex, and it's usually git checkout, more rarely git reset. They immediately realize they fucked up and spend a few minutes trying to undo by throwing random git commands at it until eventually giving up.
Yeap - this is why when running it in a dev container, I just use ZFS and set up a 1 minute auto-snapshot - which is set up as root - so it generally cannot blow it away. And cc/codex/gemini know how to deal with zfs snapshots to revert from them.
Of course if you give an agentic loop root access in yolo mode - then I am not sure how to help...
Start with env args like AGENT_ID for indicating which Merkle hash of which model(s) generated which code with which agent(s) and add those attributes to signed (-S) commit messages. For traceability; to find other faulty code generated by the same model and determine whether an agent or a human introduced the fault.
Then, `git notes` is better for signature metadata because it doesn't change the commit hash to add signatures for the commit.
And then, you'd need to run a local Rekor log to use Sigstore attestations on every commit.
Sigstore.dev is SLSA.dev compliant.
Sigstore grants short-lived release attestation signing keys for CI builds on a build farm to sign artifacts with.
So, when jujutsu autocommits agent-generated code, what causes there to be an {{AGENT_ID}} in the commit message or git notes? And what stops a user from forging such attestations?
This is funny. I tried it once and didn't see what the benefit was. Then, when I tried to reset it back to normal git, I realized that the devs had not (at the time) made any clean way to revert it back, just a one-way conversion to jj. I haven't tried it since.
What were you trying to “revert back”? You should have been able to just stop using jj, there’s nothing to revert back to. It’s also possible that I’m misunderstanding what you mean.
First time I am seeing realistic timelines from a vibe-coded project. Usually everyone who vibe codes just says they did in few hours, no matter the project.
Hmm. My experience with it is that a few hours of that will get you a sprint if you're lucky and the prompt hits the happy path. I had… I think two of those, over 5 weeks? I can believe plenty of random people stumble across happy-path examples.
Exciting when it works, but I think a much more exciting result for people with less experience who may not know that the "works for me" demo is the dreaded "first 90%", and even fairly small projects aren't done until the fifth-to-tenth 90%.
(That, and that vibe coding in the sense of "no code review" are prone to balls of mud, so you need to be above average at project management to avoid that after a few sprint-equivalents of output).
It’s possible to vibe code certain generic things in a few hours if you’re basically combining common, thoroughly documented, mature building blocks. It’s not going to be production ready or polished but you can get surprisingly far with some things.
For real work, that phase is like starting from a template or a boilerplate repo. The real work begins after the basics are wired together.
I would’ve walked for days to a CompUSA and spent my life savings if there was anything remotely equivalent to this when I was learning C on my Macintosh 4400 in 1997
Did you actually learn C? Be thankful nothing like this existed in 1997.
A machine generating code you don't understand is not the way to learn a programming language. It's a way to create software without programming.
These tools can be used as learning assistants, but the vast majority of people don't use them as such. This will lead to a collective degradation of knowledge and skills, and the proliferation of shoddily built software with more issues than anyone relying on these tools will know how to fix. At least people who can actually program will be in demand to fix this mess for years to come.
> There was a time when you had to know ‘as’, ‘ld’ and maybe even ‘ar’ to get an executable.
No, there wasn't: you could just run the shell script, or (a bit later) the makefile. But there were benefits to knowing as, ld and ar, and there still are today.
> But there were benefits to knowing as, ld and ar, and there still are today.
This is trivially true. The constraint for anything you do in your life is time it takes to know something.
So the far more interesting question is: At what level do you want to solve problems – and is it likely that you need knowledge of as, ld and ar over anything else, that you could learn instead?
Knowledge of as, ld, ar, cc, etc is only needed when setting up (or modifying) your build toolchain, and in practice you can just copy-paste the build script from some other, similar project. Knowledge of these tools has never been needed.
Knowledge of cc has never been needed? What an optimist! You must never have had headers installed in a place where the compiler (or Makefile author) didn’t expect them. Same problems with the libraries. Worse when the routine you needed to link was in a different library (maybe an arch-specific optimized lib).
The library problems you described are nothing that can't be solved using symlinks. A bad solution? Sure, but it works, and doesn't require me to understand cc. (Though when I needed to solve this problem, it only took me about 15 minutes and a man page to learn how to do it. `gcc -v --help` is, however, unhelpful.)
"A similar project" as in: this isn't the first piece of software ever written, and many previous examples can be found on the computer you're currently using. Skim through them until you find one with a source file structure you like, then ruthlessly cannibalise its build script.
If you don't see a difference between a compiler and a probabilistic token generator, I don't know what to tell you.
And, yes, I'm aware that most compilers are not entirely deterministic either, but LLMs are inherently nondeterministic. And I'm also aware that you can tweak LLMs to be more deterministic, but in practice they're never deployed like that.
Besides, creating software via natural language is an entirely different exercise than using a structured language purposely built for that.
We're talking about two entirely different ways of creating software, and any comparison between them is completely absurd.
People negating down your comment are just "engineers" doomed to fail sooner or later.
Meanwhile, 9front users have read at least the plan9 intro and know about nm, 1-9c, 1-9l and the like. Wibe coders will be put on their place sooner or later. It´s just a matter of time.
I don't understand how OP thinks that being oblivious how anything work underneath is a good thing. There is a threshold of abstraction to which you must know how it works to effectively fix it when it breaks.
You can be a super productive Python coder without any clue how assembly works. Vibe coding is just one more level of abstraction.
Just like how we still need assembly and C programmers for the most critical use cases, we'll still need Python and Golang programmers for things that need to be more efficient than what was vibe coded.
But do you really need your $whatever to be super efficient, or is it good enough if it just works?
Humans writing code are also non deterministic. When you vibe code you're basically a product owner / manager. Vibe coding isn't a higher level programming language, it's an abstraction over a software engineer / engineering team.
That's not what determinism means though. A human coding something, irrespective of whether the code is right or wrong, is deterministic. We have a well defined cause and effect pathway. If I write bad code, I will have a bug - deterministic. If I write good code, my code compiles - still deterministic. If the coder is sick, he can't write code - deterministic again. You can determine the cause from the effect.
Every behavior in the physical World has a cause and effect chain.
On the other hand, you cannot determine why a LLM hallucinated. There is no way to retrace the path taken from input parameters to generated output. At least as of now. Maybe it will change in the future where we have tools that can retrace the path taken.
You misunderstand. A coder will write different code for the same problem each time unless they have the solution 100% memorised. And even then a huge number of factors can influence them not being able to remember 100% of the memorised code, or opt for different variations.
People are inherently nondeterministic.
The code they (and AI) writes, once written, executes deterministically.
> A coder will write... or opt for different variations.
Agreed.
> People are inherently nondeterministic.
We are getting into the realm of philosophy here. I, for one, believe in the idea of living organisms having no free will (or limited will to be more precise. but can also go so far as to say "dependent will"). So one can philosophically explain that people are deterministic, via concepts of Karma and rebirth. Of course none of this can be proven. So your argument can be true too.
> The code they (and AI) writes, once written, executes deterministically.
Yes. Execution is deterministic. I am however talking only about determinism in terms of being able to know the entire path: input to output. Not just the outputs characteristic (which is always going to be deterministic). It is the path from input to output that is not deterministic due to presence of a black box - the model.
Perhaps there is no need to actually understand assembly, but if you don't understand certain basic concepts actually deploying any software you wrote to production would be a lottery with some rather poor prizes. Regardless of how "productive" you were.
Somebody needs to understand, to the standard of "well enough".
The investors who paid for the CEO who hired your project manager to hire you to figure that out, didn't.
I think in this analogy, vibe coders are project managers, who may indeed still benefit from understanding computers, but when they don't the odds aren't anywhere near as poor as a lottery. Ignorance still blows up in people's faces. I'd say the analogy here with humans would be a stereotypical PHB who can't tell what support the dev needs to do their job and then puts them on a PIP the moment any unclear requirement blows up in anyone's face.
It would’ve been nice to have a system that I could just ask questions to teach me how it works instead of having to pour through the few books that existed on C that was actually accessible to a teenager learning on their own
Going to arcane websites, forum full of neckbeards to expect you to already understand everything isn’t exactly a great way to learn
The early Internet was unbelievably hostile to people trying to learn genuinely
I had the books (from the library) but never managed to get a compiler for many years! Was quite confusing trying to understand all the unix references when my only experience with a computer was the Atari ST.
Since there are no humans involved, it's more like growing a tree. Sure it's good to know how trees grow, but not knowing about cells didn't stop thousands of years of agriculture.
The Gas Town piece reminded me of this as well. The author there leaned into role playing, social and culture analogies, and it made a lot more sense than an architecture diagram in which one node is “black box intelligence” with a single line leading out of it…
Except that the tree is so malformed and the core structure so unsound that it can't grow much past its germination and dies of malnourishment because since you have zero understanding of biology, forestry and related fields there is no knowledge to save it or help it grow healthy.
Also out of nowhere an invasive species of spiders that was inside the seed starts replicating geometrically and within seconds wraps the whole forest with webs and asks for a ransom in order to produce the secret enzyme that can dissolve it. Trying to torch it will set the whole forest on fire, brute force is futile. Unfortunately, you assumed the process would only plagiarize the good bits, but seems like it also sometimes plagiarizes the bad bits too, oops.
Its not like tree at all because tree is one and done.
Code is a project that has to be updated, fixed, etc.
So when something breaks - you have to ask the contractor again. It may not find an issue, or mess things up when it tries to fix it making project useless, etc.
Its more like a car. Every time something goes wrong you will pay for it - sometimes it will get back in even worse shape (no refunds though), sometimes it will cost you x100 because there is nothing you can do, you need it and you can't manage it on your own.
Trees are not static, unchanging, pop into existence and forget about, things.
Trees that don't get regular "updates" of adequate sunlight, water, and nutrients die. In fact, too much light or water could kill it. Or soil that is not the right courseness or acidity level could hamper or prevent growth. Now add "bugs". Literal bugs, diseases, and even competing plants that could eat, poison, or choke the tree.
You might be thinking of trees that are indigenous to an area. Even these compete for the resources and plagues of their area, but are more apt than the trees accustom to different environments, and even they go through the cycle of life.
I think his analogy was perfect, because this is the first time coding could resemble nature. We are just used to the carefully curated human made code, as there has not been such a thing as naturally occuring, no human interaction, code before
I wouldn't say it is a tree as such as at least trees are deterministic where input parameters (seed, environment, sunlight) define the output.
LLM outputs are akin to a mutant tree that can decide to randomly sprout a giant mushroom instead of a branch. And you won't have any idea why despite your input parameters being deterministic.
You haven't done a lot of gardening if you don't know plants get 'randomly' (there's a biological explanation, but with the massive amounts of variables it feels random) attacked by parasites all the time. Go look at pot growing subreddits, they spend an enormous chunk of their time fighting mites.
Determinism is not strictly anti-randomness (though I can see why one can confuse it to be polar opposites). Rather we do not even have true randomness (at least not proven) and should actually be called pseudorandom. Determinism just means that if you have the same input parameters (considering all parameters have been accounted for), you will get the same result. In other words, you can start with a particular random seed (pseudorandom seed to be precise) and always end up with the same end result and that would be considered deterministic.
> You haven't done a lot of gardening if you don't know plants
I grow "herbs".
> there's a biological explanation
Exactly. There is always an explanation for every phenomena that occurs in this observable, physical World. There is a defined cause and effect. Even if it "feels random". That's not how it is with LLMs. Because in between your deterministic input parameters and the output that is generated, there is a black box: the model itself. You have no access to the billions of parameters within the models which means you are not sure you can always reproduce the output. That black box is what causes non-determinism.
EDIT: just wanted to add - "attacked by parasites all the time", is why I said if you have control over the environment. Controlling environment encompasses dealing with parasites as well. Think of well-controlled environment like a lab.
Do you think LLMs sidestep cause and effect somehow ? There's an explanation there too, we just don't know it, But that's the case for many natural phenomena.
In what world are trees deterministic? There are a set of parameters that you can control that give you a higher probability of success, but uncontrollable variables can wipe you out.
Explained here [1]. We live in a pseudorandom World. So everything is deterministic if you have the same set of input parameters. That includes trees as well.
I am not talking about controllable/uncontrollable variables. That has no bearing on whether a process is deterministic in theory or not. If you can theoretically control all variables (even if you practically cannot), you have a deterministic process as you can reproduce the entire path: from input to output. LLMs are currently a black box. You have no access to the billions of parameters within the model, making it non-deterministic. The day we have tools where we can control all the billions of parameters within the model, then we can retrace the exact path taken, thereby making it deterministic.
I love the interview at the end of the video. The kubectl-inspired CLI, and the feedback for improvements from Claude, as well as the alerts/segmentation feedback.
You could take those, make the tools better, and repeat the experience, and I'd love to see how much better the run would go.
I keep thinking about that when it comes to things like this - the Pokemon thing as well. The quality of the tooling around the AI is only going to become more and more impactful as time goes on. The more you can deterministically figure out on behalf of the AI to provide it with accurate ways of seeing and doing things, the better.
Ditto for humans, of course, that's the great thing about optimizing for AI. It's really just "if a human was using this, what would they need"? Think about it: The whole thing with the paths not being properly connected, a human would have to sit down and really think about it, draw/sketch the layout to visualize and understand what coordinates to do things in. And if you couldn't do that, you too would probably struggle for a while. But if the tool provided you with enough context to understand that a path wasn't connected properly and why, you'd be fine.
I see this sentiment of using AI to improve itself a lot but it never seems to work well in practice. At best you end up with a very verbose context that covers all the random edge cases encountered during tasks.
For this to work the way people expect you’d need to somehow feed this info back into fine tuning rather than just appending to context. Otherwise the model never actually “learns”, you’re just applying heavy handed fudge factors to existing weights through context.
I've been playing around with an AI generated knowledge base to grok our code base, I think you need good metrics on how the knowledge base is used. A few things is:
1. Being systematic. Having a system for adding, improving and maintaining the knoweldge base
2. Having feedback for that system
3. Implementing the feedback into a better system
I'm pretty happy I have an audit framework and documentation standards. I've refactored the whole knowledge base a few times. In the places where it's overly specific or too narrow in it's scope of use for the retained knowledge, you just have to prune it.
Any garden has weeds when you lay down fertile soil.
Sometimes they aren't weeds though, and that's where having a person in the driver's seat is a boon.
Interesting article but it doesn’t actually discuss how well it performs at playing the game. There is in fact a 1.5 hour YouTube video but it woulda been nice for a bit of an outcome postmortem. It’s like “here’s the methods and set up section of a research paper but for the conclusion you need to watch this movie and make your own judgements!”
It does discuss that? Basically it has good grasp of finances and often knows what "should" be done, but it struggles with actually building anything beyond placing toilets and hotdog stalls. To be fair, its map interface is not exactly optimal, and a multimodal model might fare quite a bit better at understanding the 2D map (verticality would likely still be a problem).
Follow up Q: what are you supposed to do when the context becomes too large? Start a new conversation/context window and let Claude start from scratch?
Context filling up is sort of the Achilles heel of CLI agents. The main remedy is to have it output some type of handoff document and then run /compact which leaves you with a summary of the latest task. It sort of works but by definition it loses information, and you often find yourself having to re-explain or re-generate details to continue the work.
I made a tool[1] that lets you just start a new session and injects the original session file path, so you can extract any arbitrary details of prior work from there using sub-agents.
Either have Claude /compact or have it output things to a file it can read in on the next session. That file would be a summary of progress for work on a spec or something similar. Also good to prime it again with the Readme or any other higher level context
It’s a good idea to have Claude write down the execution plan (including todos). Or you can use something like Linear / GH Issues to track the big items. Then small/tactical todos are what you track in session todos.
This approach means you can just kill the session and restart if you hit limits.
(If you hit context limits you probably also want to look into sub-agents to help prevent context bloat. For example any time you are running and debugging unit tests, it’s usually best to start with a subagent to handle the easy errors. )
Start in plan mode, generating a markdown file with the plan, keep it up to date as it is executed, and after each iteration commit, clear the context and tell it to read the plan and execute the next step.
It feels like one could produce a digest of the context that works very similarly but fits in the available context window - not just by getting the LLM to use succinct language, but also mathematically; like reducing a sparse matrix.
There might be an input that would produce that sort of effect, perhaps it looks like nonsense (like reading zipped data) but when the LLM attempts to do interactive in it the outcome is close to consuming the context?
I ask it to write a markdown file describing how it should go about performing the task. Then have it read the file next time. Works well for things like creating tests for controller methods where there is a procedure it should follow that was probably developed over a session with several prompts and feedback on its output.
Yes you can literally just ask Claude Code to create a status line showing context usage. I had it make this colored progress bar of context usage, changing thru green, yellow, orange, red as context fills up. Instructions to install:
I corroborate that spatial reasoning is a challenge still. In this case, it's the complexity of the game world, but anyone who has used Codex/Claude with complex UIs in CSS or a native UI library will recognize the shortcomings fairly quickly.
Given dwarf fortress has an ASCII interface it may actually be a lot easier to set up claude to work with it. Also, a lot of the challenges of dwarf fortress is just knowing all the different mechanics and how they work which is something claude should be good at.
And it’s (Claude) almost certainly accumulated a fair amount of knowledge about the game itself, given the number of tutorials, guides, and other resources that have been written about DF over the last two decades.
People have been botting on Runescape since the early 2000s. Obviously not quite at the Claude level :). The botting forums were a group of very active and welcoming communities. This is actually what led me to Java programming and computer science more broadly--I wrote custom scripts for my characters.
I still have some parts of the old Rei-net forum archived on an external somewhere.
I've done this! Given the right interface I was surprised at how well it did. Prompted it "You're controlling a character in Old School RuneScape, come up with a goal for yourself, and don't stop working on it until you've achieved it". It decided to fish for and cook 100 lobsters, and it did it pretty much flawlessly!
Biggest downside was it's inability to see (literally), getting lists of interact-able game objects, NPCs, etc was fine when it decided to do something that didn't require any real-time input. Sailing, or anything that required it to react to what's on screen was pretty much impossible without more tooling to manage the reacting part for it (e.g. tool to navigate automatically to some location).
They detect bots but let a ton of them run free because any character having membership = revenue and an extremely significant chunk of active characters are bots. They nuked them all in 2011 I think and the game was nearly empty.
SirPugger's youtube channel has loads of videos monitoring various bot farms.
This is a cool idea. I wanted to do something like this by adding a Lua API to OpenRCT2 that allows you to manipulate and inspect the game world. Then, you could either provide an LLM agent the ability to write and run scripts in the game, or program a more classic AI using the Lua API. This AI would probably perform much better than an LLM - but an interesting experiment nonetheless to see how a language model can fare in a task it was not trained to do.
It's been several times that I see ASCII being used initially for these kinds of problems. I think it's because its counter-intuitive, in the sense that for us humans ASCII is text but we tend to forget spacial awareness.
I find this very interesting of us humans interacting with AIs.
Dota 2 is a real time strategy game with an arguably more complex micro game (but a far simpler macro game than AoE2, but that's far easier for an AI to master), and OpenAI Five completely destroyed the reigning champions. In 2019. Perfect coordination between units, superhuman mechanical skill, perfect consistency.
I see no reason why AoE2 would be any different.
Worth noting that openAI Five was mostly deep reinforcement learning and massive distributed training, it didn't use image to text and an LLM for reasoning about what it sees to make its "decisions". But that wouldn't be a good way to do an AI like that anyway.
Oh, and humans still play Dota. It's still a highly competitive community. So that wasn't destroyed at all, most teams now use AI to study tactics and strategy.
The opening paragraph I thought was the agent prompt haha
> The park rating is climbing. Your flagship coaster is printing money. Guests are happy, for now. But you know what's coming: the inevitable cascade of breakdowns, the trash piling up by the exits, the queue times spiraling out of control.
Surely it must have digested plenty of walkthroughs for any game?
A linear puzzle game like that I would just expect the ai to fly through first time, considering it has probably read 30 years of guides and walkthroughs.
Moravec's paradox likely comes in to play, what's easy is hard and vice versa.
The puzzles would probably be easy. Myst's puzzles are basically IQ tests, and LLMs ace traditional IQ tests: https://trackingai.org/home
On the other hand, navigating the environment, I think the models may fail spectacularly. From what we've seen from Claude Plays Pokemon, it would get in weird loops and try to interact with non-interactive elements of the environment.
This sounds as expected to me as a heavy user of Opus. Claude absolutely has a "personality" that is a lot less formal and more willing to "play along" with more creative tasks than Codex. If you want an agent that's prepared to just jump in, it's a plus. If you want an agent that will be careful, considered and plan things out meticulously, it's not always so great - I feel that when you want Claude to do reptitive, tedious tasks, you need to do more work to prevent it from getting "bored" and try to take shortcuts or find something else to do, for example.
Yup - most models ignore specific initial instructions once you pass ~50% of usable context window, and revert to their defaults eg generating overtly descriptive yet useless docs / summaries
Yes I believe so. Also things like forcing a "key insight" summary after the excels vs struggles section.
I would take any descriptions like "comprehensive", "sophisticated" etc with a massive grain of salt. But the nuts and bolts of how it was done should be accurate.
I’ve been doing game development and it starts to hallucinate more rapidly when it doesn’t understand things like the direction it placing things or which way the camera is oriented
Gemini models are a little bit better about spatial reasoning, but we’re still not there yet because these models were not designed to do spatial reasoning they were designed to process text
In my development, I also use the ascii matrix technique.
Spatial awareness was also a huge limitation to Claude playing pokemon.
It really seems to me that the first AI company getting to implement "spatial awareness" vector tokens and integrating them neatly with the other conventional text, image and sound tokens will be reaping huge rewards.
Some are already partnering with robot companies, it's only a matter of time before one of those gets there.
I disagree. With opus I'll screenshot an app and draw all over it like a child with me paint and paste it into the chat - it seems to reasonably understand what I'm asking with my chicken scratch and dimensions.
As far as 3d I don't have experience however it could be quite awful at that
Yeah at least for 2D, Opus 4.5 seems decent. It can struggle with finer details, so sometimes I’ll grab a highlighter tool in Photoshop and mark the points of interest.
I wonder if they could integrate a secondary "world model" trained/fine-tuned on Rollercoaster Tycoon to just do the layout reasoning, and have the main agent offload tasks to it.
While this seems cool at first, it does not demonstrate superiority over a true custom built AI for rollercoaster tycoon.
It is a curiosity, good for headlines, but the takeaway is if you really need an actual good AI, you are still better off not using an LLM powered solution.
Interesting this is on the ramp.com domain? I'm surprised in this tech market they can pay devs to hack on Rollercoaster Tycoon. Maybe there's some crossover I'm missing but seems like a sweet gig honestly.
This is brilliant SEO work, I doubt that they loose money with it. With 40h and some additional for the landingpage it might be an expensive link bait, but definitely worth it. Kudos!
If not for SEO, it’s building quite a good reputation for this company, they got a lot of open positions.
I’m a big fan of transport tycoon, used to play it for hours as a kid and with Open Transport Tycoon it also might have been a good choice, but maybe not B2C?
This was an interesting application of AI, but I don't really think this is what LLMs excel at. Correct me if I'm wrong.
It was interesting that the poster vibe-coded (I'm assuming) the CTL from scratch; Claude was probably pretty good at doing that, and that task could likely have been completed in an afternoon.
Pairing the CTL with the CLI makes sense, as that's the only way to gain feedback from the game. Claude can't easily do spatial recognition (yet).
A project like this would entirely depend on the game being open source. I've seen some very impressive applications of AI online with closed-source games and entire algorithms dedicated to visual reasoning.
Was able to have AI learn to play Mario Kart nearly perfectly. I find his work to be very impressive.
I guess because RCT2 is more data-driven than visually challenging, this solution works well, but having an LLM try to play a racing game sounds like it would be disastrous.
Not sure if you clocked this, but the Mario Kart AI is not an LLM. It's a randomized neural net that was trained with reinforcement learning. Apologies if I misread.
Crusader Kings is a franchise I really could see LLMs shine. One of the current main criticisms on the game is that there's a lack of events, and that they often don't really feel relevant to your character.
An LLM could potentially make events far more aimed at your character, and could actually respond to things happening in the world far more than what the game currently does. It could really create some cool emerging gameplay.
In general you are right, I expect something like this to appear in the future and it would be cool.
But isn't the criticism rather that there are too many (as you say repetitive, not relevant) events - its not like there are cool stories emerging from the underlying game mechanics anymore ("grand strategy") but players have to click through these boring predetermined events again and again.
You get too many events, but there aren't actually that many different events written, so you repeat the same ones over and over again. Eventually it just turns into the player clicking on the 'optimal' choice without actually reading the event.
You could mod the game with more varied events, which were of course AI generated to begin with. Bit of an inception scenario where AI plays an AI modded game.
The other option is to have an AI play another AI which is working as an antagonist, trying to make the player fail. More global plagues! More scheming underlings! More questionable choices for relaxation! Bit of an arms race there.
Honestly I prefer Crusader Kings II if for no other reason that the UI is just so brilliantly insanely obtuse while also being very good looking.
I actually think it would be pretty fun to code something to play video games for me, it has a lot of overlap with robotics. Separately, I learned about assembly from cheat engine when I was a kid.
That’s not the point of this. This was an exercise to measure the strengths and weaknesses of current LLMs in operating a company and managing operations, and the video game was just the simulation engine.
You do you. I find this exceedingly cool and I think it's a fun new thing to do.
It's kind of like how people started watching Let's Plays and that turned into Twitch.
One of the coolest things recently is VTubers in mocap suits using AI performers to do single person improv performances with. It's wild and cool as hell. A single performer creating a vast fantasy world full of characters.
LLMs and agents playing Pokemon and StarCraft? Also a ton of fun.
This is the dumbest, most AI-laden nonsense I've read today.
I'm glad I have cancer and don't have long left with idiots like you praising regurgitative garbage. There is nothing left for humanity with your non-existent imagination and attention span. Just watch TikTok and drink slug, "human"
AI is one of the best tool categories we've invented. I don't know why people are so pearl-clutchy, fisting-at-clouds about it.
Some of the worst human behavior I've experienced outside of grade school is the anti-AI crowd sending me death threats and endless streams of insults. It's surreal how twisted and vile the words that some anti-AI people throw are.
This is the fifth technological wave, after the chip, PC, internet, and smartphone.
All of human programming cannot do what AI is already showing signs of being capable of automating. Our image and video models can render things even 80 years of optical physics and algorithms cannot do.
I am legitimately excited in a way I never have been before. We're lucky to be able to witness this.
Related:
I’ve always found it crazy that my LLM has access to such terrible tools compared to mine.
It’s left with grepping for function signatures, sending diffs for patching, and running `cat` to read all the code at once.
I however, run an IDE and can run a simple refactoring tool to add a parameter to a function, I can “follow symbol” to see where something is defined, I can click and get all usages of a function shown at a glance, etc etc.
Is anyone working on making it so LLM’s get better tools for actually writing/refactoring code? Or is there some “bitter lesson”-like thing that says effort is always better spent just increasing the context size and slurping up all the code at once?
> I however, run an IDE and can run a simple refactoring tool to add a parameter to a function, I can “follow symbol” to see where something is defined, I can click and get all usages of a function shown at a glance, etc etc
I am so surprised that all of the AI tooling mostly revolves around VSC or its forks and that JetBrains seem to not really have done anything revolutionary in the space.
With how good their refactoring and code inspection tools are, you’d really think they’d pass of that context information to AI models and that they’d be leaps and bounds ahead.
Recently, all these agents can talk LSP (language server protocol) so it should get better soon. That said, yeah they don't seem to default to use `ripgrep` when that is clearly better than `grep`
Codex likes to use ripgrep.
Are you? I'm not surprised at all, considering that the biggest investment juggernaut in AI is also the author of VSC. I wonder what the connection is? ;)
Well, Google also has their own AIs and lots of money to throw around.
Agreed - this seems like a no brainer, surely this is something that is being worked on.
> Claude Code officially added native support for the Language Server Protocol (LSP) in version 2.0.74, released in December 2025.
I think from training it's still biased towards simple tooling.
But also, there is real power to simple tools, a small set of general purpose tools beats a bunch of narrow specific use case tools. It's easier for humans to use high level tools, but for LLM's they can instantly compose the low level tools for their use case and learn to generalize, it's like writing insane perl one liners is second nature for them compared to us.
If you watch the tool calls you'll see they write a ton of one off small python programs to test, validate explore, etc...
If you think about it any time you use a tool there is probably a 20 line python program that is more fit to your use case, it's just that it would take you too long to write it, but for an LLM that's 0.5 seconds
Correct. If you try to create a coding agent using the raw Codex or Claude code API and you build your own “write tool”, and don’t give the model their “native patch tool”, 70%+ of the time it’s write/ patch fails because it tries to do the operation using the write/ patch tool it was trained on.
LLMs aren't like you or me. They can comprehend large quantities of code quickly and piece things together easily from scattered fragments. so go to reference etc become much less important. Of course though things change as the number of usages of a symbol becomes large but in most cases the LLM can just make perfect sense of things via grep.
To provide it access to refactoring as a tool also risks confusing it via too many tools.
It's the same reason that waffling for a few minutes via speech to text with tangents and corrections and chaos is just about as good as a carefully written prompt for coding agents.
Zed Editor gives the LLM tools that use the LSP as you'd expect as a normal IDE user, like "go to symbol definition" so it greps a lot less.
If you can read fast enough, grepping is probably faster than waiting for a compiler to tell you anything.
Faster for worse results, though. Determining the source of a symbol is not as trivial as finding the same piece of text somewhere else, it should also reliably be able to differentiate among them. What better source for that then the compiler itself?
Yeah, especially for languages that make heavy use of type inference. There’s nothing you can really grep for most of the time… to really know “who’s using this code” you need to know what the compiler knows.
An LLM can likely approach compiler-level knowledge just from being smart and understanding what it’s reading, but it costs a lot of context to do this. Giving the LLM access to what the compiler knows as an API seems like it’s a huge area for improvement.
It depends on the language and codebase. For something very dynamic like Python it may be the case that grepping finds real references to a symbol that won’t be found by a language server. Also language servers may not work with cross-language interfaces or codegen situations as well as grep.
OTOH for a giant monorepo, grep probably won’t work very well.
You can give agents the ability to check VSCode Diagnostics, LSP servers and the like.
But they constantly ignore them and use their base CLI tools instead, it drives me batty. No matter what I put in AGENTS.md or similar, they always just ignore the more advanced tooling IME.
I’ve been saying this for a while. CPU demand is about to go through the roof.
I think about it, to get these tools to be most effective you have to be able to page things in and out of their context windows.
What was once a couple of queries is now gonna be dozens or hundreds or even more from the LLM
For code that means querying the AST and query it in a way that allows you to limit the results of the output
I wonder which SAST vendor Anthropic will buy.
Kit looks like a good step in this direction:
https://github.com/cased/kit
Not coding agents but we do a lot of work trying to find the best tools, and the result is always that the simplest possible general tool that can get the job done always beats a suite of complicated tools and rules on how to use them.
Well, jump to definition isn't exactly complicated?
And you can use whatever interface the language servers already use to expose that functionality to eg vscode?
LSP also kind of sucks. But the problem is all the big companies want big valuations, so they only chase generic solutions. That's why everything is a VS Code clone, etc..
https://paulgraham.com/ds.html
I've never used an LSP plugin half as good as a JetBrains IDE.
Always wondered what happened to the era of IDEs actually knowing the language you're using.
This isn’t completely the answer to what you want but skills do open a lot of doors here. Anything you can do on a command line can turn into a skill, after all.
Anthropic, for one.
> Added LSP (Language Server Protocol) tool for code intelligence features like go-to-definition, find references, and hover documentation
https://github.com/anthropics/claude-code/blob/main/CHANGELO...
their c# LSP theoretically worked for a week or so (I never saw it in action though), but now it always errors on launch :(
An LSP MCP?
Yeah, or something even smarter than that.
If you are willing to go language-specific, the tooling can be incredibly rich if you go through the effort. I’ve written some rust compiler drivers for domain-specific use cases, and you can hook into phases of the compiler where you have amazingly detailed context about every symbol in the code. All manner of type metadata, locations where values are dropped, everything is annotated with spans of source locations too. It seems like a worthy effort to index all of it and make it available behind a standard query interface the LLM can use. You can even write code this way, I think rustfmt hooks into the same pipeline to produce formatted code.
I’ve always wished there were richer tools available to do what my IDE already does, but without needing to use the UI. Make it a standard API or even just CLI, and free it from the dependency on my IDE. It’d be very worth looking into I think.
If the compiler just dumped all that data out as structured text, you could use current LLMs to swallow it in a single gulp.
Author here - some bonus links!
Session transcript using Simon Willison's claude-code-transcripts
https://htmlpreview.github.io/?https://gist.githubuserconten...
Reddit post
https://www.reddit.com/r/ClaudeAI/comments/1q9fen5/claude_co...
OpenRCT2!!
https://github.com/jaysobel/OpenRCT2
Project repo
https://github.com/jaysobel/OpenRCT2
Did you eval using screenshots or some sort of rendered visualization instead of the CLI? I wonder if Claude has better visual intelligence when viewing images (lots of these in its training set) rather than ascii schematics (probably very few of these in the corpus).
Computer use and screenshots are context intensive. Text is not. The more context you give to an LLM, the dumber it gets. Some people think at 40% context utilization, the LLM starts to get into the dumb zone. That is where the limitations are as of today. This is why CLI based tools like Claude Code are so good. And any attempt at computer use has fallen by the wayside.
There are some potential solutions to this problem that come to mind. Use subagents to isolate the interesting bits about a screenshot and only feed that to the main agent with a summary. This will all still have a significantly higher token usage compared to a text based interface, but something like this could potentially keep the LLM out of the dumb zone a little longer.
> And any attempt at computer use has fallen by the wayside.
You're totally right! I mean, aside from Anthropic launching "Cowork: Claude Code for the rest of your work" 5 days ago. :)
https://claude.com/blog/cowork-research-preview
https://news.ycombinator.com/item?id=46593022
More to the point though, you should be using Agents in Claude Code to limit context pollution. Agents run with their own context, and then only return salient details. Eg, I have an Agent to run "make" and return the return status and just the first error message if there is one. This means the hundreds/thousands of lines of compilation don't pollute the main Claude Code context, letting me get more builds in before I run out of context there.
Claude helped me immensely getting an image converter to work. Giving it screenshots of wrong output (lots of layers had an unpredictable offsets that was not supposed to be there) and output as I expected it helped Claude understand the problems and it fixed the bugs immediately.
I had tried the browser screenshotting feature for agents in Cursor and found it wasn't very reliable - screenshots eat a lot of context, and the agent didn't have a good sense for when to use them. I didn't try it in this project. I bet it would work in some specific cases.
> Claude is at a pretty steep visuo-spatial disadvantage,
How hard would it be to use with OpenAI's offerings instead? Particularly, imo, OpenAI's better at "looking" at pictures than Claude.
> As a mirror to real-world agent design: the limiting factor for general-purpose agents is the legibility of their environments, and the strength of their interfaces. For this reason, we prefer to think of agents as automating diligence, rather than intelligence, for operational challenges.
> The only other notable setback was an accidental use of the word "revert" which Codex took literally, and ran git revert on a file where 1-2 hours of progress had been accumulating.
If I tell Claude to "revert that last change, it isn't right, try this instead" and Claude hasn't committed recently it will happily `git checkout ...` and blow away all recent changes instead of reverting the "last change".
(Which, it's not wrong or anything -- I did say "revert that change" -- it's just annoying. And telling `CLAUDE.md` to commit more often doesn't work consistently, because Claude is a dummy sometimes).
Amazing that these tools don't maintain a replayable log of everything they've done.
Although git revert is not a destructive operation, so it's surprising that it caused any loss of data. Maybe they meant git reset --hard or something like that. Wild if Codec would run that.
I was looking at the insanity known as Gas Town [0] the other day, and it does use Git to store historical work state in something it calls "beads":
https://github.com/steveyegge/gastown?tab=readme-ov-file
If anyone is curious, Beads is an agent memory project from the same developer: https://github.com/steveyegge/beads
Bees?
BEADS
I have had codex recover things for me from its history after claude had done a git reset hard, codex is one of the more reliable models/harneses when it comes to performing undo and redo operations in my experience.
Claude Code has had this feature for a few months now.
Claude (can’t remember if was 4.1 Opus, 4.5 Sonnet, or 4.5 Opus) once just started playing with git worktrees and royally f-d up the local repo and lost several hours of work. Since then, I watch it like a hawk.
`git reset --hard` doesn't remove unreferenced commits or rewrite the reflog so I don't think that would do it. Something like `git reset && git gc` would have to be done.
And git gc doesn't collect any garbage less than two weeks old by default, either.
Claude Code has /rewind. Not sure if it is foolproof, but this has been tried.
I found this tool to be the solution I was looking for to address this specific problem:
https://contextify.sh
I wonder how they accidentaly used a word like that.
“Please revert that last change you did”, referring to like a smaller change that had just been done
Codex reverted kindly.
Does Codex not let you set command permissions?
Yea, it does so this would likely have been to be a `--yolo` (I don't care, let me `rm -rf /`). I've found even with the "workspace write" mode and no additional writable directories I can't do git operations without approval so it seems to exclude `.git` by default.
Yet another reason to use Jujutsu. And put a `jj status` wrapper in your PS1. ;-)
> Yet another reason to use Jujutsu
And what would that reason be? You can git revert a git revert.
You're correct for an actual git revert, but it seems pretty clear that the original authors have mangled the story and it was actually either a "git checkout" or "git reset". The "file where 1-2 hours of progress had been accumulating" phrasing only makes sense if those were uncommitted changes.
And the reason jj helps in that case is that for jj there is no such thing as an uncommitted change.
Having no such thing as an uncommitted change seems like it would be a nightmare, but perhaps I'm just too git-oriented.
> Having no such thing as an uncommitted change seems like it would be a nightmare, but perhaps I'm just too git-oriented.
Why? What's the problem you see? The only problem I see is when you let these extra commits pollute the history reachable from any branch you care about.
Let's look at the following:
Internally, 'git stash' consists of two operations: one that makes an 'anonymous' commit of your files, and another that resets those files to whatever they were in HEAD. (That commit is anonymous in the sense that no branch points at it.)
The git libraries expose the two operations separately. And you can build something yourself that works similarly.
You can use these capabilities to build an undo/redo log in git, but without polluting any of the history you care about.
To be honest, I have no clue how Jujutsu does it. They might be using a totally different design.
> perhaps I'm just too git-oriented.
The problem is git's index let's you write a bunch of unconnected code, then commit it separately. To different branches, even! This works great for stacking diffs but is terribly confusing if you don't know what you're doing.
Well, git doesn't really commit 'to' a branch.
You just build commits, and then later on you muck around with the mutable pointers that are branches.
Things like the index become a workflow pattern, rather than a feature, if that makes any sense.
Also JJ undo is there and easy to tell the model to use, I have it in my Claude.md
surely Claude is much better at using git because of the massive training data difference.
If it didn't undo git, it would do it with JJ either.
It does fine with jj. Sometimes better, because jj is much easier to use non-interactively.
Probably it actually ran git checkout or reset. As you say git revert only operates on committed snapshots so it will all be in the reflog
Yes, this exact scenario has happened to me a couple times with both Claude and Codex, and it's usually git checkout, more rarely git reset. They immediately realize they fucked up and spend a few minutes trying to undo by throwing random git commands at it until eventually giving up.
Yeap - this is why when running it in a dev container, I just use ZFS and set up a 1 minute auto-snapshot - which is set up as root - so it generally cannot blow it away. And cc/codex/gemini know how to deal with zfs snapshots to revert from them.
Of course if you give an agentic loop root access in yolo mode - then I am not sure how to help...
Start with env args like AGENT_ID for indicating which Merkle hash of which model(s) generated which code with which agent(s) and add those attributes to signed (-S) commit messages. For traceability; to find other faulty code generated by the same model and determine whether an agent or a human introduced the fault.
Then, `git notes` is better for signature metadata because it doesn't change the commit hash to add signatures for the commit.
And then, you'd need to run a local Rekor log to use Sigstore attestations on every commit.
Sigstore.dev is SLSA.dev compliant.
Sigstore grants short-lived release attestation signing keys for CI builds on a build farm to sign artifacts with.
So, when jujutsu autocommits agent-generated code, what causes there to be an {{AGENT_ID}} in the commit message or git notes? And what stops a user from forging such attestations?
- "Diffwatch – Watch AI agents touch the FS and see diffs live" (2025) https://news.ycombinator.com/item?id=45786382 :
> you can manually stage against @-: [with jujutsu]
It's not going to happen...
Stop spamming
The feature of "there is no such thing as an uncommitted working directory" is very relevant to the situation.
It's not. There are so many ways to just solve this non issue that no one will just switch to just another random tool.
Especially not away from git.
> It's not
Given that other posts solved the problem by scripting this feature on top of git, I guess you're telling them their solution isn't relevant too.
This is funny. I tried it once and didn't see what the benefit was. Then, when I tried to reset it back to normal git, I realized that the devs had not (at the time) made any clean way to revert it back, just a one-way conversion to jj. I haven't tried it since.
What were you trying to “revert back”? You should have been able to just stop using jj, there’s nothing to revert back to. It’s also possible that I’m misunderstanding what you mean.
> We don't know any C++ at all, and we vibe-coded the entire project over a few weeks. The core pieces of the build are…
what a world!
Everyone should read that section. It was really interesting reading about their experiences/challenges getting it all working.
First time I am seeing realistic timelines from a vibe-coded project. Usually everyone who vibe codes just says they did in few hours, no matter the project.
Hmm. My experience with it is that a few hours of that will get you a sprint if you're lucky and the prompt hits the happy path. I had… I think two of those, over 5 weeks? I can believe plenty of random people stumble across happy-path examples.
Exciting when it works, but I think a much more exciting result for people with less experience who may not know that the "works for me" demo is the dreaded "first 90%", and even fairly small projects aren't done until the fifth-to-tenth 90%.
(That, and that vibe coding in the sense of "no code review" are prone to balls of mud, so you need to be above average at project management to avoid that after a few sprint-equivalents of output).
It’s possible to vibe code certain generic things in a few hours if you’re basically combining common, thoroughly documented, mature building blocks. It’s not going to be production ready or polished but you can get surprisingly far with some things.
For real work, that phase is like starting from a template or a boilerplate repo. The real work begins after the basics are wired together.
I would’ve walked for days to a CompUSA and spent my life savings if there was anything remotely equivalent to this when I was learning C on my Macintosh 4400 in 1997
People don’t appreciate what they have
Did you actually learn C? Be thankful nothing like this existed in 1997.
A machine generating code you don't understand is not the way to learn a programming language. It's a way to create software without programming.
These tools can be used as learning assistants, but the vast majority of people don't use them as such. This will lead to a collective degradation of knowledge and skills, and the proliferation of shoddily built software with more issues than anyone relying on these tools will know how to fix. At least people who can actually program will be in demand to fix this mess for years to come.
That’s what a C compiler does when generating a binary.
There was a time when you had to know ‘as’, ‘ld’ and maybe even ‘ar’ to get an executable.
In the early days of g++, there was no guarantee the object code worked as intended. But it was fun working that out and filing the bug reports.
This new tool is just a different sort of transpiler and optimiser.
Treat it as such.
> There was a time when you had to know ‘as’, ‘ld’ and maybe even ‘ar’ to get an executable.
No, there wasn't: you could just run the shell script, or (a bit later) the makefile. But there were benefits to knowing as, ld and ar, and there still are today.
> But there were benefits to knowing as, ld and ar, and there still are today.
This is trivially true. The constraint for anything you do in your life is time it takes to know something.
So the far more interesting question is: At what level do you want to solve problems – and is it likely that you need knowledge of as, ld and ar over anything else, that you could learn instead?
Knowledge of as, ld, ar, cc, etc is only needed when setting up (or modifying) your build toolchain, and in practice you can just copy-paste the build script from some other, similar project. Knowledge of these tools has never been needed.
Knowledge of cc has never been needed? What an optimist! You must never have had headers installed in a place where the compiler (or Makefile author) didn’t expect them. Same problems with the libraries. Worse when the routine you needed to link was in a different library (maybe an arch-specific optimized lib).
That post is only true in the most vacuous sense.
“A similar project” discovered where, on BITNET?
The library problems you described are nothing that can't be solved using symlinks. A bad solution? Sure, but it works, and doesn't require me to understand cc. (Though when I needed to solve this problem, it only took me about 15 minutes and a man page to learn how to do it. `gcc -v --help` is, however, unhelpful.)
"A similar project" as in: this isn't the first piece of software ever written, and many previous examples can be found on the computer you're currently using. Skim through them until you find one with a source file structure you like, then ruthlessly cannibalise its build script.
If you don't see a difference between a compiler and a probabilistic token generator, I don't know what to tell you.
And, yes, I'm aware that most compilers are not entirely deterministic either, but LLMs are inherently nondeterministic. And I'm also aware that you can tweak LLMs to be more deterministic, but in practice they're never deployed like that.
Besides, creating software via natural language is an entirely different exercise than using a structured language purposely built for that.
We're talking about two entirely different ways of creating software, and any comparison between them is completely absurd.
People negating down your comment are just "engineers" doomed to fail sooner or later.
Meanwhile, 9front users have read at least the plan9 intro and know about nm, 1-9c, 1-9l and the like. Wibe coders will be put on their place sooner or later. It´s just a matter of time.
Competent C programmers know about nm, as, ld and a bunch of other binary sections in order to understand issues and proper debugging.
Everyone else are deluding themselves. Even the 9front intro requieres you to at least know the basics of nm and friends.
I don't understand how OP thinks that being oblivious how anything work underneath is a good thing. There is a threshold of abstraction to which you must know how it works to effectively fix it when it breaks.
You can be a super productive Python coder without any clue how assembly works. Vibe coding is just one more level of abstraction.
Just like how we still need assembly and C programmers for the most critical use cases, we'll still need Python and Golang programmers for things that need to be more efficient than what was vibe coded.
But do you really need your $whatever to be super efficient, or is it good enough if it just works?
One is deterministic the other is not. I leave it to you to determine which is which in this scenario.
Humans writing code are also non deterministic. When you vibe code you're basically a product owner / manager. Vibe coding isn't a higher level programming language, it's an abstraction over a software engineer / engineering team.
> Humans writing code are also non deterministic
That's not what determinism means though. A human coding something, irrespective of whether the code is right or wrong, is deterministic. We have a well defined cause and effect pathway. If I write bad code, I will have a bug - deterministic. If I write good code, my code compiles - still deterministic. If the coder is sick, he can't write code - deterministic again. You can determine the cause from the effect.
Every behavior in the physical World has a cause and effect chain.
On the other hand, you cannot determine why a LLM hallucinated. There is no way to retrace the path taken from input parameters to generated output. At least as of now. Maybe it will change in the future where we have tools that can retrace the path taken.
You misunderstand. A coder will write different code for the same problem each time unless they have the solution 100% memorised. And even then a huge number of factors can influence them not being able to remember 100% of the memorised code, or opt for different variations.
People are inherently nondeterministic.
The code they (and AI) writes, once written, executes deterministically.
> A coder will write... or opt for different variations.
Agreed.
> People are inherently nondeterministic.
We are getting into the realm of philosophy here. I, for one, believe in the idea of living organisms having no free will (or limited will to be more precise. but can also go so far as to say "dependent will"). So one can philosophically explain that people are deterministic, via concepts of Karma and rebirth. Of course none of this can be proven. So your argument can be true too.
> The code they (and AI) writes, once written, executes deterministically.
Yes. Execution is deterministic. I am however talking only about determinism in terms of being able to know the entire path: input to output. Not just the outputs characteristic (which is always going to be deterministic). It is the path from input to output that is not deterministic due to presence of a black box - the model.
> The code they (and AI) writes, once written, executes deterministically.
very rarely :)
This is true. What are the implications of that?
Perhaps there is no need to actually understand assembly, but if you don't understand certain basic concepts actually deploying any software you wrote to production would be a lottery with some rather poor prizes. Regardless of how "productive" you were.
Somebody needs to understand, to the standard of "well enough".
The investors who paid for the CEO who hired your project manager to hire you to figure that out, didn't.
I think in this analogy, vibe coders are project managers, who may indeed still benefit from understanding computers, but when they don't the odds aren't anywhere near as poor as a lottery. Ignorance still blows up in people's faces. I'd say the analogy here with humans would be a stereotypical PHB who can't tell what support the dev needs to do their job and then puts them on a PIP the moment any unclear requirement blows up in anyone's face.
I’m vaguely aware that transistors are like electronic switches and if I serve my memory I could build and and/or/not gate
I have no idea how an i386 works, let alone a modern cpu. Sure there are registers and different levels of cache before you get to memory.
My lack of knowledge of all this doesn’t prevent me from creating useful programs using higher abstraction layers like c.
It would’ve been nice to have a system that I could just ask questions to teach me how it works instead of having to pour through the few books that existed on C that was actually accessible to a teenager learning on their own
Going to arcane websites, forum full of neckbeards to expect you to already understand everything isn’t exactly a great way to learn
The early Internet was unbelievably hostile to people trying to learn genuinely
I had the books (from the library) but never managed to get a compiler for many years! Was quite confusing trying to understand all the unix references when my only experience with a computer was the Atari ST.
*pore through
(not a judgment, just mentioning in case the distinction is interesting to anyone)
It's just another layer.
Assembly programmers from years gone by would likley be equally dismissive of the self-aggrandizing code block stitchers of today.
(on topic, RCT was coded entirely in assembly, quite the achievement)
It’s worse. They’re proud they don’t know.
"They" are? I didn't see that in the article. It sounds like you are projecting your prejudices on to a non-defined out group.
Its like ordering a project from upwork- someone did it for you, you have no idea what is going on, kinda works though.
Since there are no humans involved, it's more like growing a tree. Sure it's good to know how trees grow, but not knowing about cells didn't stop thousands of years of agriculture.
The Gas Town piece reminded me of this as well. The author there leaned into role playing, social and culture analogies, and it made a lot more sense than an architecture diagram in which one node is “black box intelligence” with a single line leading out of it…
Very interesting analogy
Except that the tree is so malformed and the core structure so unsound that it can't grow much past its germination and dies of malnourishment because since you have zero understanding of biology, forestry and related fields there is no knowledge to save it or help it grow healthy.
Also out of nowhere an invasive species of spiders that was inside the seed starts replicating geometrically and within seconds wraps the whole forest with webs and asks for a ransom in order to produce the secret enzyme that can dissolve it. Trying to torch it will set the whole forest on fire, brute force is futile. Unfortunately, you assumed the process would only plagiarize the good bits, but seems like it also sometimes plagiarizes the bad bits too, oops.
Its not like tree at all because tree is one and done.
Code is a project that has to be updated, fixed, etc.
So when something breaks - you have to ask the contractor again. It may not find an issue, or mess things up when it tries to fix it making project useless, etc.
Its more like a car. Every time something goes wrong you will pay for it - sometimes it will get back in even worse shape (no refunds though), sometimes it will cost you x100 because there is nothing you can do, you need it and you can't manage it on your own.
Trees are not static, unchanging, pop into existence and forget about, things. Trees that don't get regular "updates" of adequate sunlight, water, and nutrients die. In fact, too much light or water could kill it. Or soil that is not the right courseness or acidity level could hamper or prevent growth. Now add "bugs". Literal bugs, diseases, and even competing plants that could eat, poison, or choke the tree. You might be thinking of trees that are indigenous to an area. Even these compete for the resources and plagues of their area, but are more apt than the trees accustom to different environments, and even they go through the cycle of life. I think his analogy was perfect, because this is the first time coding could resemble nature. We are just used to the carefully curated human made code, as there has not been such a thing as naturally occuring, no human interaction, code before
I wouldn't say it is a tree as such as at least trees are deterministic where input parameters (seed, environment, sunlight) define the output.
LLM outputs are akin to a mutant tree that can decide to randomly sprout a giant mushroom instead of a branch. And you won't have any idea why despite your input parameters being deterministic.
You haven't done a lot of gardening if you don't know plants get 'randomly' (there's a biological explanation, but with the massive amounts of variables it feels random) attacked by parasites all the time. Go look at pot growing subreddits, they spend an enormous chunk of their time fighting mites.
Determinism is not strictly anti-randomness (though I can see why one can confuse it to be polar opposites). Rather we do not even have true randomness (at least not proven) and should actually be called pseudorandom. Determinism just means that if you have the same input parameters (considering all parameters have been accounted for), you will get the same result. In other words, you can start with a particular random seed (pseudorandom seed to be precise) and always end up with the same end result and that would be considered deterministic.
> You haven't done a lot of gardening if you don't know plants
I grow "herbs".
> there's a biological explanation
Exactly. There is always an explanation for every phenomena that occurs in this observable, physical World. There is a defined cause and effect. Even if it "feels random". That's not how it is with LLMs. Because in between your deterministic input parameters and the output that is generated, there is a black box: the model itself. You have no access to the billions of parameters within the models which means you are not sure you can always reproduce the output. That black box is what causes non-determinism.
EDIT: just wanted to add - "attacked by parasites all the time", is why I said if you have control over the environment. Controlling environment encompasses dealing with parasites as well. Think of well-controlled environment like a lab.
Do you think LLMs sidestep cause and effect somehow ? There's an explanation there too, we just don't know it, But that's the case for many natural phenomena.
I am not saying LLM sidesteps cause-effect. I am saying it is a black box. So yes "we just don't know it" is basically describing a black box.
In what world are trees deterministic? There are a set of parameters that you can control that give you a higher probability of success, but uncontrollable variables can wipe you out.
Explained here [1]. We live in a pseudorandom World. So everything is deterministic if you have the same set of input parameters. That includes trees as well.
I am not talking about controllable/uncontrollable variables. That has no bearing on whether a process is deterministic in theory or not. If you can theoretically control all variables (even if you practically cannot), you have a deterministic process as you can reproduce the entire path: from input to output. LLMs are currently a black box. You have no access to the billions of parameters within the model, making it non-deterministic. The day we have tools where we can control all the billions of parameters within the model, then we can retrace the exact path taken, thereby making it deterministic.
[1]: https://news.ycombinator.com/item?id=46663052
Great analogy. “I don’t know any C++ but I hired some people on Upwork and they delivered this software demo.”
Con fuckign gratys, u can buy compute
I love the interview at the end of the video. The kubectl-inspired CLI, and the feedback for improvements from Claude, as well as the alerts/segmentation feedback.
You could take those, make the tools better, and repeat the experience, and I'd love to see how much better the run would go.
I keep thinking about that when it comes to things like this - the Pokemon thing as well. The quality of the tooling around the AI is only going to become more and more impactful as time goes on. The more you can deterministically figure out on behalf of the AI to provide it with accurate ways of seeing and doing things, the better.
Ditto for humans, of course, that's the great thing about optimizing for AI. It's really just "if a human was using this, what would they need"? Think about it: The whole thing with the paths not being properly connected, a human would have to sit down and really think about it, draw/sketch the layout to visualize and understand what coordinates to do things in. And if you couldn't do that, you too would probably struggle for a while. But if the tool provided you with enough context to understand that a path wasn't connected properly and why, you'd be fine.
I see this sentiment of using AI to improve itself a lot but it never seems to work well in practice. At best you end up with a very verbose context that covers all the random edge cases encountered during tasks.
For this to work the way people expect you’d need to somehow feed this info back into fine tuning rather than just appending to context. Otherwise the model never actually “learns”, you’re just applying heavy handed fudge factors to existing weights through context.
I've been playing around with an AI generated knowledge base to grok our code base, I think you need good metrics on how the knowledge base is used. A few things is:
1. Being systematic. Having a system for adding, improving and maintaining the knoweldge base 2. Having feedback for that system 3. Implementing the feedback into a better system
I'm pretty happy I have an audit framework and documentation standards. I've refactored the whole knowledge base a few times. In the places where it's overly specific or too narrow in it's scope of use for the retained knowledge, you just have to prune it.
Any garden has weeds when you lay down fertile soil.
Sometimes they aren't weeds though, and that's where having a person in the driver's seat is a boon.
Interesting article but it doesn’t actually discuss how well it performs at playing the game. There is in fact a 1.5 hour YouTube video but it woulda been nice for a bit of an outcome postmortem. It’s like “here’s the methods and set up section of a research paper but for the conclusion you need to watch this movie and make your own judgements!”
It does discuss that? Basically it has good grasp of finances and often knows what "should" be done, but it struggles with actually building anything beyond placing toilets and hotdog stalls. To be fair, its map interface is not exactly optimal, and a multimodal model might fare quite a bit better at understanding the 2D map (verticality would likely still be a problem).
I was told the important part of AI is the generation part, not the verification or quality.
> kept the context above the ~60% remaining level where coding models perform at their absolute best
Maybe this is obvious to Claude users but how do you know your remaining context level? There is UI for this?
You can also show context in the statusline within claude code: https://code.claude.com/docs/en/statusline#context-window-us...
Follow up Q: what are you supposed to do when the context becomes too large? Start a new conversation/context window and let Claude start from scratch?
Context filling up is sort of the Achilles heel of CLI agents. The main remedy is to have it output some type of handoff document and then run /compact which leaves you with a summary of the latest task. It sort of works but by definition it loses information, and you often find yourself having to re-explain or re-generate details to continue the work.
I made a tool[1] that lets you just start a new session and injects the original session file path, so you can extract any arbitrary details of prior work from there using sub-agents.
[1] aichat tool https://github.com/pchalasani/claude-code-tools?tab=readme-o...
Either have Claude /compact or have it output things to a file it can read in on the next session. That file would be a summary of progress for work on a spec or something similar. Also good to prime it again with the Readme or any other higher level context
It’s a good idea to have Claude write down the execution plan (including todos). Or you can use something like Linear / GH Issues to track the big items. Then small/tactical todos are what you track in session todos.
This approach means you can just kill the session and restart if you hit limits.
(If you hit context limits you probably also want to look into sub-agents to help prevent context bloat. For example any time you are running and debugging unit tests, it’s usually best to start with a subagent to handle the easy errors. )
Start in plan mode, generating a markdown file with the plan, keep it up to date as it is executed, and after each iteration commit, clear the context and tell it to read the plan and execute the next step.
It feels like one could produce a digest of the context that works very similarly but fits in the available context window - not just by getting the LLM to use succinct language, but also mathematically; like reducing a sparse matrix.
There might be an input that would produce that sort of effect, perhaps it looks like nonsense (like reading zipped data) but when the LLM attempts to do interactive in it the outcome is close to consuming the context?
``` §CONV_DIGEST§ T1:usr_query@llm-ctx-compression→math-analog(sparse-matrix|zip)?token-seq→nonsense-input→semantic-equiv-output? T2:rsp@asymmetry_problem:compress≠decompress|llm=predict¬decode→no-bijective-map|soft-prompts∈embedding-space¬token-space+require-training|gisting(ICAE)=aux-model-compress→memory-tokens|token-compress-fails:nonlinear-distributed-mapping+syntax-semantic-entanglement|works≈lossy-semantic-distill@task-specific+finetune=collapse-instruction→weights §T3:usr→design-full-python-impl§ T4:arch_blueprint→ DIR:src/context_compressor/{core/(base|result|pipeline)|compressors/(extractive|abstractive|semantic|entity_graph|soft_prompt|gisting|hybrid)|embeddings/(providers|clustering)|evaluation/(metrics|task_performance|benchmark)|models/(base|openai|anthropic|local)|utils/(tokenization|text_processing|config)} CLASSES:CompressionMethod=Enum(EXTRACTIVE|ABSTRACTIVE|SEMANTIC_CLUSTERING|ENTITY_GRAPH|SOFT_PROMPT|GISTING|HYBRID)|CompressionResult@(original_text+compressed_text+original_tokens+compressed_tokens+method+compression_ratio+metadata+soft_vectors?)|TokenCounter=Protocol(count|truncate_to_limit)|EmbeddingProvider=Protocol(embed|embed_single)|LLMBackend=Protocol(generate|get_token_limit)|ContextCompressor=ABC(token_counter+target_ratio=0.25+min_tokens=50+max_tokens?→compress:abstract)|TrainableCompressor(ContextCompressor)+(train+save+load) COMPRESSORS:extractive→(TextRank|MMR|LeadSentence)|abstractive→(LLMSummary|ChainOfDensity|HierarchicalSummary)|semantic→(ClusterCentroid|SemanticChunk|DiversityMaximizer)|entity→(EntityRelation|FactList)|soft→(SoftPrompt|PromptTuning)|gist→(GistToken|Autoencoder)|hybrid→(Cascade|Ensemble|Adaptive) EVAL:EvaluationResult@(compression_ratio+token_reduction+embedding_similarity+entailment_score+entity_recall+fact_recall+keyword_overlap+qa_accuracy?+reconstruction_bleu?)→composite_score(weights)|CompressionEvaluator(embedding_provider+llm?+nli?)→evaluate|compare_methods PIPELINE:CompressionPipeline(steps:list[Compressor])→sequential-apply|AdaptiveRouter(compressors:dict+classifier?)→content-based-routing DEPS:numpy|torch|transformers|sentence-transformers|tiktoken|networkx|sklearn|spacy|openai|anthropic|pandas|pydantic+optional(accelerate|peft|datasets|sacrebleu|rouge-score) ```
I ask it to write a markdown file describing how it should go about performing the task. Then have it read the file next time. Works well for things like creating tests for controller methods where there is a procedure it should follow that was probably developed over a session with several prompts and feedback on its output.
Yes you can literally just ask Claude Code to create a status line showing context usage. I had it make this colored progress bar of context usage, changing thru green, yellow, orange, red as context fills up. Instructions to install:
https://github.com/pchalasani/claude-code-tools?tab=readme-o...
Claude code has a /context command.
/context
the beauty of this game was that it was developed in Assembly Code and on top of that by majorly one person.
I've been trying to locate the dev of this game since a long time, so I can thank them for an amazing experience.
If anyone knows their social or anything, please do share, including OP.
Also, nice work on CC in this. May actually be interested in Claude Code now.
I think something like Civilization would be better because:
1) The map is a grid
2) Turn based
I corroborate that spatial reasoning is a challenge still. In this case, it's the complexity of the game world, but anyone who has used Codex/Claude with complex UIs in CSS or a native UI library will recognize the shortcomings fairly quickly.
This is what I want but for PoE/PoE2 builds. I always get a headache just looking at the passive tree https://poe.ninja/poe2/passive-skill-tree
Claude Code in dwarf fortress would be wild
https://www.youtube.com/watch?v=FLmPN03ZQbM
Given dwarf fortress has an ASCII interface it may actually be a lot easier to set up claude to work with it. Also, a lot of the challenges of dwarf fortress is just knowing all the different mechanics and how they work which is something claude should be good at.
And it’s (Claude) almost certainly accumulated a fair amount of knowledge about the game itself, given the number of tutorials, guides, and other resources that have been written about DF over the last two decades.
> In this article we'll tell you why we decided to put Claude Code into RollerCoaster Tycoon, and what lessons it taught us about B2B SaaS.
What is this? A LinkedIn post?
I want to get off MR ALTMANS WILD RIDE.
Can't wait for someone to let Claude control a runescape character from scratch
People have been botting on Runescape since the early 2000s. Obviously not quite at the Claude level :). The botting forums were a group of very active and welcoming communities. This is actually what led me to Java programming and computer science more broadly--I wrote custom scripts for my characters.
I still have some parts of the old Rei-net forum archived on an external somewhere.
I've done this! Given the right interface I was surprised at how well it did. Prompted it "You're controlling a character in Old School RuneScape, come up with a goal for yourself, and don't stop working on it until you've achieved it". It decided to fish for and cook 100 lobsters, and it did it pretty much flawlessly!
Biggest downside was it's inability to see (literally), getting lists of interact-able game objects, NPCs, etc was fine when it decided to do something that didn't require any real-time input. Sailing, or anything that required it to react to what's on screen was pretty much impossible without more tooling to manage the reacting part for it (e.g. tool to navigate automatically to some location).
https://www.reddit.com/r/2007scape/comments/1qeh3nc/i_added_...
https://ubos.tech/mcp/runescape-mcp-server-rs-osrs/
Wouldn't that break Jagex's TOS though? Is there a way of getting caught?
I imagine Jagex must be up there with having the most sophisticated bot detection out of anyone. Its been a thing for decades
They detect bots but let a ton of them run free because any character having membership = revenue and an extremely significant chunk of active characters are bots. They nuked them all in 2011 I think and the game was nearly empty.
SirPugger's youtube channel has loads of videos monitoring various bot farms.
This is a cool idea. I wanted to do something like this by adding a Lua API to OpenRCT2 that allows you to manipulate and inspect the game world. Then, you could either provide an LLM agent the ability to write and run scripts in the game, or program a more classic AI using the Lua API. This AI would probably perform much better than an LLM - but an interesting experiment nonetheless to see how a language model can fare in a task it was not trained to do.
As far as a scripting API, it looks like the devs beat me to it with a JS/TS plugin system: https://github.com/OpenRCT2/OpenRCT2/blob/develop/distributi...
It's been several times that I see ASCII being used initially for these kinds of problems. I think it's because its counter-intuitive, in the sense that for us humans ASCII is text but we tend to forget spacial awareness.
I find this very interesting of us humans interacting with AIs.
Does this website do anything besides host the article with an animated background?
Question: There is still a competitive AoE2 community. Will that be destroyed by AI?
Dota 2 is a real time strategy game with an arguably more complex micro game (but a far simpler macro game than AoE2, but that's far easier for an AI to master), and OpenAI Five completely destroyed the reigning champions. In 2019. Perfect coordination between units, superhuman mechanical skill, perfect consistency.
I see no reason why AoE2 would be any different.
Worth noting that openAI Five was mostly deep reinforcement learning and massive distributed training, it didn't use image to text and an LLM for reasoning about what it sees to make its "decisions". But that wouldn't be a good way to do an AI like that anyway.
Oh, and humans still play Dota. It's still a highly competitive community. So that wasn't destroyed at all, most teams now use AI to study tactics and strategy.
The opening paragraph I thought was the agent prompt haha
> The park rating is climbing. Your flagship coaster is printing money. Guests are happy, for now. But you know what's coming: the inevitable cascade of breakdowns, the trash piling up by the exits, the queue times spiraling out of control.
Most interesting phrase: "Keeping all four agents busy took a lot of mental bandwidth."
Wonder how it would do with Myst.
Surely it must have digested plenty of walkthroughs for any game?
A linear puzzle game like that I would just expect the ai to fly through first time, considering it has probably read 30 years of guides and walkthroughs.
The real test would be to try it on a new game of the same style and complexity
Moravec's paradox likely comes in to play, what's easy is hard and vice versa.
The puzzles would probably be easy. Myst's puzzles are basically IQ tests, and LLMs ace traditional IQ tests: https://trackingai.org/home
On the other hand, navigating the environment, I think the models may fail spectacularly. From what we've seen from Claude Plays Pokemon, it would get in weird loops and try to interact with non-interactive elements of the environment.
> "Where Claude excels:"
Am I reading a Claude generated summary here?
I thought it sounded more like an ad for Claude written by Anthropic:
> "This was surprising, but fits with Claude's playful personality and flexible disposition."
This sounds as expected to me as a heavy user of Opus. Claude absolutely has a "personality" that is a lot less formal and more willing to "play along" with more creative tasks than Codex. If you want an agent that's prepared to just jump in, it's a plus. If you want an agent that will be careful, considered and plan things out meticulously, it's not always so great - I feel that when you want Claude to do reptitive, tedious tasks, you need to do more work to prevent it from getting "bored" and try to take shortcuts or find something else to do, for example.
> when you want Claude to do reptitive, tedious tasks, you need to do more work to prevent it from getting "bored"
Is this sentance seriously about a computer? Have we gone so far that computers wont just do what we tell them to anymore?
Yup - most models ignore specific initial instructions once you pass ~50% of usable context window, and revert to their defaults eg generating overtly descriptive yet useless docs / summaries
Yes I believe so. Also things like forcing a "key insight" summary after the excels vs struggles section.
I would take any descriptions like "comprehensive", "sophisticated" etc with a massive grain of salt. But the nuts and bolts of how it was done should be accurate.
Would a way to take screenshots help? It seems to work for browser testing.
I’ve been doing game development and it starts to hallucinate more rapidly when it doesn’t understand things like the direction it placing things or which way the camera is oriented
Gemini models are a little bit better about spatial reasoning, but we’re still not there yet because these models were not designed to do spatial reasoning they were designed to process text
In my development, I also use the ascii matrix technique.
Spatial awareness was also a huge limitation to Claude playing pokemon.
It really seems to me that the first AI company getting to implement "spatial awareness" vector tokens and integrating them neatly with the other conventional text, image and sound tokens will be reaping huge rewards. Some are already partnering with robot companies, it's only a matter of time before one of those gets there.
This is also my experience with attempting to use Claude and GLM-4.7 with OpenSCAD. Horrible spatial reasoning abilities.
I disagree. With opus I'll screenshot an app and draw all over it like a child with me paint and paste it into the chat - it seems to reasonably understand what I'm asking with my chicken scratch and dimensions.
As far as 3d I don't have experience however it could be quite awful at that
Yeah at least for 2D, Opus 4.5 seems decent. It can struggle with finer details, so sometimes I’ll grab a highlighter tool in Photoshop and mark the points of interest.
They would need a spatial reason or layout specific tool, to translate to English and back
I wonder if they could integrate a secondary "world model" trained/fine-tuned on Rollercoaster Tycoon to just do the layout reasoning, and have the main agent offload tasks to it.
this is cute but i imagined prompting the ai for a loop-di-loop roller coaster. If this could build complex ride it would be a game changer.
yeah I was expecting it to... do something in the game? like build a ride
not just make up bullshit about events
Edit: HN's auto-resubmit in action, ignore.
What
So, this link is actually 5 days old, if you hover the "2 hours ago" you'll see the date 5 days ago.
HN second-chance pool shenanigans.
Can you point to any documentation which explains how this works?
Genuinely interested.
Dang gave some explanation here: https://news.ycombinator.com/item?id=26998308
> We don't know any C++ at all, and we vibe-coded the entire project over a few weeks.
And these are the same people that put countless engineers through gauntlets of bizarre interview questions and exotic puzzles to hire engineers.
But when it comes to C++ just vibe it obviously.
Oh, I almost didn't realise this is done by a company. I was like this must have costed a lot, didn't realize its just an advertisement for ramp
so the janitors will finally stay on their assigned footpaths?
While this seems cool at first, it does not demonstrate superiority over a true custom built AI for rollercoaster tycoon.
It is a curiosity, good for headlines, but the takeaway is if you really need an actual good AI, you are still better off not using an LLM powered solution.
Interesting this is on the ramp.com domain? I'm surprised in this tech market they can pay devs to hack on Rollercoaster Tycoon. Maybe there's some crossover I'm missing but seems like a sweet gig honestly.
yeah really - ramp.com is a credit card/expense platform that surely loses money right now...
pretty heavy/slow javascript but pretty functional nonetheless...
This is brilliant SEO work, I doubt that they loose money with it. With 40h and some additional for the landingpage it might be an expensive link bait, but definitely worth it. Kudos!
If not for SEO, it’s building quite a good reputation for this company, they got a lot of open positions.
I’m a big fan of transport tycoon, used to play it for hours as a kid and with Open Transport Tycoon it also might have been a good choice, but maybe not B2C?
*OpenRCT2
This was an interesting application of AI, but I don't really think this is what LLMs excel at. Correct me if I'm wrong.
It was interesting that the poster vibe-coded (I'm assuming) the CTL from scratch; Claude was probably pretty good at doing that, and that task could likely have been completed in an afternoon.
Pairing the CTL with the CLI makes sense, as that's the only way to gain feedback from the game. Claude can't easily do spatial recognition (yet).
A project like this would entirely depend on the game being open source. I've seen some very impressive applications of AI online with closed-source games and entire algorithms dedicated to visual reasoning.
I'm still trying to figure out how this guy: https://www.youtube.com/watch?v=Doec5gxhT_U
Was able to have AI learn to play Mario Kart nearly perfectly. I find his work to be very impressive.
I guess because RCT2 is more data-driven than visually challenging, this solution works well, but having an LLM try to play a racing game sounds like it would be disastrous.
Not sure if you clocked this, but the Mario Kart AI is not an LLM. It's a randomized neural net that was trained with reinforcement learning. Apologies if I misread.
Yeah, that was the point of my post. LLMs traditionally aren't used in gaming like this.
next up: Crusader Kings III
Crusader Kings is a franchise I really could see LLMs shine. One of the current main criticisms on the game is that there's a lack of events, and that they often don't really feel relevant to your character.
An LLM could potentially make events far more aimed at your character, and could actually respond to things happening in the world far more than what the game currently does. It could really create some cool emerging gameplay.
In general you are right, I expect something like this to appear in the future and it would be cool.
But isn't the criticism rather that there are too many (as you say repetitive, not relevant) events - its not like there are cool stories emerging from the underlying game mechanics anymore ("grand strategy") but players have to click through these boring predetermined events again and again.
You get too many events, but there aren't actually that many different events written, so you repeat the same ones over and over again. Eventually it just turns into the player clicking on the 'optimal' choice without actually reading the event.
You could mod the game with more varied events, which were of course AI generated to begin with. Bit of an inception scenario where AI plays an AI modded game.
The other option is to have an AI play another AI which is working as an antagonist, trying to make the player fail. More global plagues! More scheming underlings! More questionable choices for relaxation! Bit of an arms race there.
Honestly I prefer Crusader Kings II if for no other reason that the UI is just so brilliantly insanely obtuse while also being very good looking.
> You’re right, I did accidentally slaughter all the residents of Béziers. I won’t do that again. But I think that you’ll find God knows his own.
Paradox future hire right here
"i vibe coded a thing to play video games for me"
i enjoy playing video games my own self. separately, i enjoy writing code for video games. i don't need ai for either of these things.
Yeah, but can you use your enjoyment of video games as marketing material to justify a $32B valuation?
If you look at submissions from this website, its all just self glazing and "We did X with claude code"
Haha exactly. This screams “we have too many people working here and don’t know what to do with them”.
actually it was all to drive traffic to my 'rollercoaster coasters' Etsy store
https://bansostudio.etsy.com
^ this guy funds
Not so sure. He said justify.
I actually think it would be pretty fun to code something to play video games for me, it has a lot of overlap with robotics. Separately, I learned about assembly from cheat engine when I was a kid.
That’s not the point of this. This was an exercise to measure the strengths and weaknesses of current LLMs in operating a company and managing operations, and the video game was just the simulation engine.
That's fine. Tool-assisted speedruns long predate LLMs and they're boring as hell: https://youtu.be/W-MrhVPEqRo
It's still a neat perspective on how to optimize for super-specific constraints.
That TAS is spliced. The stairs beyond the door aren't loaded, you need the key to load it.
This is a real console 0-star TAS: https://youtu.be/iUt840BUOYA
> Tool-assisted speedruns long predate LLMs and they're boring as hell
You and I have _very_ different definitions for the word boring. A lot of effort goes into TAS runs.
AI for the sake of AI. Feels like a lot of the internet right now
You do you. I find this exceedingly cool and I think it's a fun new thing to do.
It's kind of like how people started watching Let's Plays and that turned into Twitch.
One of the coolest things recently is VTubers in mocap suits using AI performers to do single person improv performances with. It's wild and cool as hell. A single performer creating a vast fantasy world full of characters.
LLMs and agents playing Pokemon and StarCraft? Also a ton of fun.
This is the dumbest, most AI-laden nonsense I've read today.
I'm glad I have cancer and don't have long left with idiots like you praising regurgitative garbage. There is nothing left for humanity with your non-existent imagination and attention span. Just watch TikTok and drink slug, "human"
AI is one of the best tool categories we've invented. I don't know why people are so pearl-clutchy, fisting-at-clouds about it.
Some of the worst human behavior I've experienced outside of grade school is the anti-AI crowd sending me death threats and endless streams of insults. It's surreal how twisted and vile the words that some anti-AI people throw are.
This is the fifth technological wave, after the chip, PC, internet, and smartphone.
All of human programming cannot do what AI is already showing signs of being capable of automating. Our image and video models can render things even 80 years of optical physics and algorithms cannot do.
I am legitimately excited in a way I never have been before. We're lucky to be able to witness this.
Sorry for your cancer.