> It didn't work, and why it didn't work turned out to be more interesting than the policy would have been.
Spoken like a true Claude.
Snarking aside, I am glad that our AI agents make it cheap enough to do these experiments and publish these write-ups that people finally bother to publish null findings. Very useful!
I'm guessing he means overall the kind of circling and concluding that Claude does, like "the actual truth is more interesting than it seems" or "the real smoking gun is not the x, it's the y that was under our noses all this time" type of writing.
LRU seems like the ideal strategy for most things LLM-related. Everything in this realm is about recency bias. I think it is a feature in this context, not a problem.
When I give an agent a piece of corrected information regarding a long running task, the last thing I want it to do is try and statistically compensate for the fact that it is new information. I want this new information to dominate the old information.
This is almost unreadable. The "papers" are never referenced anywhere, so the claims being refuted cannot be evaluated. The whole fact of the TTL doesn't seem relevant at all. There are many, many well-researched admission and eviction policies that this readme doesn't mention. I just don't get why we are reading this.
There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.
The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.
it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"
One side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries.
Just FYI.
This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations.
Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English.
I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.
Interesting! I admit the AI-written text is rough to read, it could have used a pass or two from an actual human. E.g. "Publishing it unresolved rather than tuning until it matches." -- thanks for not lying, I guess?
Fun:
> In my first run, Belady — an offline oracle — lost to LRU. That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common.
> The cause: inserting a long chain into a near-full cache lets a policy evict the very prefix it is currently building. LRU is accidentally immune because just-inserted blocks have the newest timestamp.
> It didn't work, and why it didn't work turned out to be more interesting than the policy would have been.
Spoken like a true Claude.
Snarking aside, I am glad that our AI agents make it cheap enough to do these experiments and publish these write-ups that people finally bother to publish null findings. Very useful!
That line doesn’t strike me as overtly AI written, overall yes though
It's using a couple textbook [1] LLM tropes, 'negative parallelism' with a 'here's the kicker' tone.
[1] https://gist.github.com/ossa-ma/f3baa9d25154c33095e22272c631...
This phrasing is a very specific "Hello, this is Claude"
But it is specifically one of Claude's ticks
It is. And the reason for that is* surprising: the base knowledge corpus is clickbait internet articles.
*not at all
I'm guessing he means overall the kind of circling and concluding that Claude does, like "the actual truth is more interesting than it seems" or "the real smoking gun is not the x, it's the y that was under our noses all this time" type of writing.
"That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common."
That's Claude. But I agree and am also glad the author, uh, clauded this up. :)
LRU seems like the ideal strategy for most things LLM-related. Everything in this realm is about recency bias. I think it is a feature in this context, not a problem.
When I give an agent a piece of corrected information regarding a long running task, the last thing I want it to do is try and statistically compensate for the fact that it is new information. I want this new information to dominate the old information.
This is almost unreadable. The "papers" are never referenced anywhere, so the claims being refuted cannot be evaluated. The whole fact of the TTL doesn't seem relevant at all. There are many, many well-researched admission and eviction policies that this readme doesn't mention. I just don't get why we are reading this.
Here is the repo link: https://github.com/gauravapiscean/agentic-kv-cache
Author here. Context for why I did this:
There's a growing literature arguing LRU is the wrong eviction policy for agentic LLM serving, because agent sessions idle and LRU can't distinguish a paused session from a dead one. I found the argument convincing and built a simulator to exploit it. Three separate mechanisms, all lost to plain radix-leaf LRU.
The reason turned out to be more useful than the policy. When I measured — policy-independently — where recompute actually comes from on 393 real Claude Code sessions, requests arriving after a gap longer than the 5-minute provider TTL account for 17.5% of it. Requests arriving within 10 seconds account for 33.1%. The dominant waste is tight tool loops whose 88k-token working sets exceed cache capacity, not sessions idling past a TTL. That's a capacity problem, and liveness prediction can't touch it.
just curious, why the LLM writing even here?
it's just a bit disheartening to read Claude output for such a small comment like this.. it'd be great to read your own writings even if it's not as "perfect"
One side effect of extremely-accessible, high-quality English translation and, more importantly, English-grammar-and-idiosyncrasy-obeying AI, is that there will be more and more text that looks like this which comes from foreign, non-native-English countries.
Just FYI.
This will be especially true from non-English cultures where "avoiding shame" is high on the list of motivations.
Consider the upside, though: A much broader range of written perspectives written in high-quality, if slightly annoying, English.
I'm already seeing the benefit of this on X thanks to its autotranslation btw: I follow a few Chinese-language accounts now that I would have never been able to digest otherwise.
Its only high quality of this their thoughts. Autotranslation is different from choosing to have AI shape the expression permanently.
> Author here.
The best kind of correct.
Why/when would people expect agents to be idling?
I'd have thought they'd be busy (using cached prompt-prefixes) until they were finished.
Interesting! I admit the AI-written text is rough to read, it could have used a pass or two from an actual human. E.g. "Publishing it unresolved rather than tuning until it matches." -- thanks for not lying, I guess?
Fun:
> In my first run, Belady — an offline oracle — lost to LRU. That's not a result, that's a broken harness, and it's worth publishing because I expect it to be common.
> The cause: inserting a long chain into a near-full cache lets a policy evict the very prefix it is currently building. LRU is accidentally immune because just-inserted blocks have the newest timestamp.