ScrapingBee Blog

I've thought https://www.scrapingbee.com/ looked great, especially their auto rotation of IP addresses.

I'm working on a personal project that involves A LOT of scraping, and through several iterations I've gotten some stuff that works quite well. Here's a quick summary of what I've explored (both paid and free): * Apify (https://apify.com/) is a great, comprehensive system if you need to get fairly low-level. Everything is hosted there, they've got their own proxy service (or you can roll your own), and their open source framework (https://github.com/apify/crawlee) is excellent. * I've also experimented with running both their SDK (crawlee) and Playwright directly on Google Cloud Run, and that also works well and is an order-of-magnitude less expensive than running directly on their platform. * Bright Data nee Luminati is excellent for cheap data center proxies ($0.65/GB pay as you go), but prices get several orders of magnitude more expensive if you need anything more thorough than data center proxies. * For some direct API crawls that I do, all of the scraping stuff is unnecessary and I just ping the APIs directly. * If the site you're scraping is using any sort of anti-bot protection, I've found that ScrapingBee (https://www.scrapingbee.com/) is by far the easiest solution. I spent many many hours fighting anti-bot protection doing it myself with some combination of Bright Data, Apify and Playwright, and in the end I kinda stopped battling and just decided to let ScrapingBee deal with it for me. I may be lucky in that the sites I'm scraping don't really use JS…

> Cloudflare can always spot crawlers I mention in the post a scraping service that Cloudflare isn't spotting: https://www.scrapingbee.com/blog/how-to-bypass-cloudflare-antibot-prot… Plenty of open-source ones as well that could bypass, eg maybe this one that came up in search https://github.com/VeNoMouS/cloudscraper Combine with residential proxies and you're just not going to find them. > Comprehensive crawl of LinkedIn, FB, instagram, IMDB, Amazon, would be worth a lot. Just from a quick Google search: - LinkedIn: https://brightdata.com/products/datasets/linkedin - Amazon: https://www.junglescout.com/features/product-database/

If you’re worried about the security risks, edge cases, maintenance pain and scaling challenges of self hosting there are various solid hosted alternatives: - https://browserless.io - low level browser control - https://scrapingbee.com - scraping specialists - https://urlbox.com - screenshot specialists* They’re all profitable and have been around for years so you can depend on the businesses and the tech. * Disclosure: I work on this one and was a customer before I joined the team.

Theoretically yes! Unfortunately there's no API with more restrictive permissions :P Good point around releasing the source code for self-hosting. The only painful parts are that I two commercial services here and there for stealth proxying requests to fetch articles like (https://www.scrapingbee.com/) and Puppeteer on Cloudflare Workers (https://developers.cloudflare.com/browser-rendering/platform/puppeteer… to generate PDFs, so it's like $50/month to self-host something that actually works But maybe if I open source this others can contribute too and find ways to make this cheaper to run

I've used ScrapingBee which has similar pricing and has worked well, can't say which one is better: https://www.scrapingbee.com/

https://scrapfly.io/pricing https://scrapestack.com/product https://www.scrapingbee.com/#pricing There's quite a few products in this segment around that price range.

There's services specialized in crawling + VPN. Often used to crawl shopping websites or Google result pages. One example is https://www.scrapingbee.com and you select the country https://www.scrapingbee.com/documentation/country_codes/ They can do screenshots, too. If the concern is uptime there's a couple of service that also do regular screenshots and alerts when the website is slow or unresponsive. https://www.debugbear.com/ for example. For a super quick check (every couple of months I need this) I usually use https://www.webpagetest.org/ Under 'advanced configuration' there's a dropdown of locations around the world. I gives a screenshot but not access to the full HTML.

Really depends on how big your scraping operation is going to be. These days there's a lot of "managed" providers that give you headless browsers / proxy rotators through an easy API so it's relatively easy to plug them into your code. Examples of these would be https://www.browserless.io or https://www.scrapingbee.com for headless browsers to render JS. From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.

Don't know if this gets to the heart of your question, but I was surprised to not have seen https://www.scrapingbee.com/ mentioned here yet.

I'm the founder of https://www.scrapingbee.com/, an API for web scraping. At the beginning we had a banner on the dashboard with a CTA to book a call with me. To incentivize users to book the call, we offered free API credits: https://twitter.com/PierreDeWulf/status/1329765250707623939/photo/1 I had 100+ calls with users in the first months, it helped us a lot to understand our market, what they needed etc. It worked so well that we had to remove the free credit incentive because taking those calls took all my time.

https://www.scrapingbee.com/ does this as a service + a headless browser.

Obviously tooting my own horn here, but these are some products: - https://checklyhq.com - synthetic monitoring - https://microlink.io - automation - https://www.browserless.io - testing - https://www.scrapingbee.com/ - scraping All are business leveraging these types of frameworks

I found https://www.scrapingbee.com/ the other day. Checkout their blog as well, filled with useful stuff.

Agreed, this is why on the Selenium paragraph we link to this article that go much more in depth https://www.scrapingbee.com/blog/selenium-python/

This is almost cool, unfortunately the product itself (a network of bots to allow websites to be scraped when they obviously don't want to be) seems a little shifty. For example, put these three exhibits together: Exhibit 1: The ScrapingBee terms and conditions state "We assume that you use the Website Platform and Services legally and ethically and that you have obtained permission, if necessary, to use it on the targeted websites and/or other data sources." This is even backed up with an indemnity clause in which the user has to cover ScrapingBee for any third-party legal claim arising out of their use of the product. Reference: https://www.scrapingbee.com/terms-and-conditions/ Exhibit 2: ScrapingBee explicitly advertises a feature allowing you to get Google search results via an API call. These results are presumably generated by scraping Google's search pages: Reference: https://www.scrapingbee.com/features/google/ Exhibit 3: Google's own documentation explicitly states that automated querying is prohibited, so if you use this advertised ScrapingBee service, you are naturally violating Google's terms, and could be liable to cover ScrapingBee's legal costs if Google decide to come after them. Reference: https://developers.google.com/search/docs/advanced/guidelines/automate… $1MM in ARR is all well and good, but there's a limit to how large this business can grow without being pursued by the websites whose scraping they are enabling, and in the case of…