I use Brave browser & Brave Search and I find them both excellent. But, it clearly sounds like their profits have been coming from scraping the web and selling that to other AI companies, rather than advertising on their search engine (I almost never see ads on there despite have adblock off on that domain) or their browser products (VPN, their stupid Brave Rewards crypto which everyone should disable immediately, etc).
But, bare minimum they should have a useragent that people can opt out of through robots.txt. That's just basic etiquette. Beyond that, I'll wait for their response.
If they respected the robots.txt, the quality of the search engine would be much lower unfortunately. Many of the most visited websites refuse to be indexed these days. Brave search gives good result because Brave doesn't care.
I don't know how true that is, to be honest. I see far more people using Cloudflare turnstile and other much more restrictive tools to try and prevent users of Brave browser, VPNs, and other anti-fingerprinting/adblocking tech from accessing their website and that no doubt includes headless browsers used for scrapping.
I understand Brave wanting to have the best results for their search, but that doesn't seem to be their primary motivation if the lawsuit is to be believed. It seems to be they're doing the whole "chad third party scraper meme"[1] and then making big bucks selling those results to the other AI companies like Perplexity that have already gotten in trouble doing that same thing in the past.
I wonder in the future if they're be a whole underground/darkweb or p2p/distributed scraping economy to get scraping data from people/countries that have no restriction on where they get their data and have no qualms about getting sued about it, like Anna's Archive. Or some kind of local client people can run on their home computer that works as a peer to peer service, slowly scraping websites from their residential IPs and getting paid a few pennies to submit the results to a huge service that collects it then resells it to the big AI companies.
I use Brave browser & Brave Search and I find them both excellent. But, it clearly sounds like their profits have been coming from scraping the web and selling that to other AI companies, rather than advertising on their search engine (I almost never see ads on there despite have adblock off on that domain) or their browser products (VPN, their stupid Brave Rewards crypto which everyone should disable immediately, etc).
But, bare minimum they should have a useragent that people can opt out of through robots.txt. That's just basic etiquette. Beyond that, I'll wait for their response.
If they respected the robots.txt, the quality of the search engine would be much lower unfortunately. Many of the most visited websites refuse to be indexed these days. Brave search gives good result because Brave doesn't care.
I don't know how true that is, to be honest. I see far more people using Cloudflare turnstile and other much more restrictive tools to try and prevent users of Brave browser, VPNs, and other anti-fingerprinting/adblocking tech from accessing their website and that no doubt includes headless browsers used for scrapping.
I understand Brave wanting to have the best results for their search, but that doesn't seem to be their primary motivation if the lawsuit is to be believed. It seems to be they're doing the whole "chad third party scraper meme"[1] and then making big bucks selling those results to the other AI companies like Perplexity that have already gotten in trouble doing that same thing in the past.
I wonder in the future if they're be a whole underground/darkweb or p2p/distributed scraping economy to get scraping data from people/countries that have no restriction on where they get their data and have no qualms about getting sued about it, like Anna's Archive. Or some kind of local client people can run on their home computer that works as a peer to peer service, slowly scraping websites from their residential IPs and getting paid a few pennies to submit the results to a huge service that collects it then resells it to the big AI companies.
[1] https://knowyourmeme.com/photos/2615826-twitters-view-rate-l...
[dead]
Free SEO? It could happen to you.