Browserless Blog

For more generic web indexing you need to use a browser. You do not index pages served by a server anymore, you index pages rendered by javascript apps in the browser. So as a part of the "fetch" stage I usually let parsing of title and other page metadata to a javascript script running inside the browser (using https://www.browserless.io/) and then as part of the "parse" phase I use cheerio to extract links and such. It is very tempting to do everything in the browser, but architecturally it does not belong there. So you need to find the balance that works best for you.

I'm not sure if I'd call it simple, but the product itself is fairly straightforward (at least to me :). I run a SaaS product called browserless (https://browserless.io). Goal here was to make it really simple to run headless browser work in a production environment, without worrying about all the provisioning and maintenance. However, like anything simple on the exterior, there's still layers of complexity behind it. How do you charge users? Where do you host? How do you acquire users? How do you interact with customers? Don't mean to be cynical, but even $500/mo is going to have some cost and time complexity. Best of luck!

Thanks for sharing your experience! I'm quite opinionated on this topic so buckle up :D We avoided the fork & patch route because it's both labor intensive for a limited return on investment, and a game of catching up. Updating the forked framework is challenging on its own right, let alone porting existing customer payloads to newer versions, locking you de-facto to older versions. I did maintain a custom fork at a previous workplace that was similar in scope to Browserless[0] and I can tell you it was a real pain. Developing your own framework (besides satisfying the obvious NIH itch) allows you to precisely control your exposure (reduce the attack surface) from a security perspective, and protects your customers from upstream decisions such as deprecations or major changes that might not be aligned with your customer requirements. I also have enough experience in this space to know exactly what we need to implement and the capabilities we want to enable. No bloat (yet) > you will burn the user's fingerprint depending on scale It's relative to your activity. See my other comment about scale and use cases, for personal device usage this is not an issue in practice, and users can automate several websites[1] using their personal agents without worrying about this. For more involved scenarios we have appropriate strategies that avoid this issue. > we fix leaks and bugs of automation frameworks Sounds interesting! I'd love to read a write up, or PRs if you have…

Hey justhw, I launched something similar [1] a few months back. I’d love to talk with you about your stack. Here are the three biggest problems I ran into. 1. Request timeouts Even "fast" blogs sometimes have hiccups or rate-limit scrapers/crawlers. The more I scale the more often I see requests timing out. I've tried to mitigate this with retry buttons and trying the request again, but it doesn't always work. 2. Scaling Puppeteer I saw from another of your comments that this is built on top of Puppeteer. From my experience, trying to scale to even 3 concurrent renders starts to fall apart when you have to spin up a new Chrome instance for each request. I've looked into Browserless [2] but they charge per second of usage, and I'm not at the point where I want to have that big of an expense. 3. Getting a relevant image I love that your tool adds an image to the thumbnail as well! I struggled with searching Unsplash using the text in the title of the article but they never came out that well. I love it when people make these massively useful services stitching together a few simple tools. Congrats on the launch! [1] https://mugshotbot.com [2] https://www.browserless.io

If you’re worried about the security risks, edge cases, maintenance pain and scaling challenges of self hosting there are various solid hosted alternatives: - https://browserless.io - low level browser control - https://scrapingbee.com - scraping specialists - https://urlbox.com - screenshot specialists* They’re all profitable and have been around for years so you can depend on the businesses and the tech. * Disclosure: I work on this one and was a customer before I joined the team.

Very cool - and impressive speed! How does it compare with headless browser services like https://browserless.io or https://browserbase.com ?

https://browserless.io might be what you’re looking for. Open source although they do have a SaaS option.

In my opinion it seems like the right choice plus you get some free advertising :) Web isolation definitely makes sense, but for something like automation why not just use something like https://www.browserless.io/ I know browserboxpro provides a head but does that matter? Because alternatively you could use something like selenium with vnc

Really depends on how big your scraping operation is going to be. These days there's a lot of "managed" providers that give you headless browsers / proxy rotators through an easy API so it's relatively easy to plug them into your code. Examples of these would be https://www.browserless.io or https://www.scrapingbee.com for headless browsers to render JS. From my work experience of working on a large scraping stack with thousands of integrations, I can say that we are very happy with our own custom framework, written in Go (https://github.com/PuerkitoBio/goquery for HTML parsing) and using headless Chrome for JS rendering.

Looking over the API docs now https://docs.hyperbeam.com/start-chromium-session First thing that comes to my mind is... this is kind of like Puppeteer. Even if that is what they are using in the background... would they ever expose it so existing Puppeteer apps could plug and play (like https://www.browserless.io/)?

If you're interested in running the puppeteer in containers, take a look at chrome-aws-lambda[1] and browserless docker container[2] Not affiliated with browserless, but they do have a free/paid cloud service. https://www.browserless.io [1] https://github.com/alixaxel/chrome-aws-lambda [2] https://github.com/browserless/chrome

Is this somehow associated with https://www.browserless.io ? If not, this is confusing.

Interesting! How does this compare to https://www.browserless.io/?

Check out browserless for something that's the same direction: https://www.browserless.io/

Obviously tooting my own horn here, but these are some products: - https://checklyhq.com - synthetic monitoring - https://microlink.io - automation - https://www.browserless.io - testing - https://www.scrapingbee.com/ - scraping All are business leveraging these types of frameworks

Heads up that the article's first link to your site is broken The link is showing up as https://sourcesort.com/interview/browserless.io instead of https://browserless.io

I think this might be what you're looking for? https://www.browserless.io/

maybe with this? https://Browserless.io

Weird question: Since you mention 'bare-metal' and this is "enables you to run Postgres in the browser" UH, could you do: bare_metal --> HeadlessGPU-Chrome --> PGLite with whatever orchesrtration and you have a an automatable 'browser' that can scrape/store/logic/log/be-config'd from/lookup directly to your browserDB and maybe run a fastAPI ontop of such that you have a full stack, baremetal, headless, browserDB/fastAPI/endpoint processing doohicky? Or does this sound dumb? Because this is exactly what I actually need and am attempting to create - but its beyond my capabilities in any functional/elegant implementation... https://www.browserless.io/blog/browserless-gpu-instances https://github.com/dhamaniasad/HeadlessBrowsers