stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
gonna try this out this week, one ask: when a wall does beat it, return "blocked, here's why" instead of empty, avoiding silent fetch failures.
Token reduction is useful, but knowing whether the agent actually accessed the real page is even more important.
stripping to markdown with Jina Reader or Trafilatura before passing to the agent cuts that 68k down to ~3-5k for most Wikipedia pages, and handles the JS-rendered case too.
+1. i think the the additional value prop is the bypassing blockers. ideally op should just use jina on top of whatever they are building.
one of the early jina.ai products was/is reader api, and they trained ReaderLM for this purpose. definitely a good idea to check out existing implementations
> on js-rendered and some anti-bot pages it returns nothing
yeah, and my (human piloted) browser gets blocked by so many websites now. I routinely get "Sorry." when trying to log into hn.
sigh.
Hacker News is the one page that hasn't assumed I'm a bot, yet.
At this rate, we'll need to use bots to browse the web for us, because they're the only way to get through the anti-bot filters.
And how much water?