Scrapinghub Blog

Scrapinghub writes some useful blog posts at https://blog.scrapinghub.com/. It obviously has to do with using their frameworks and services, so it may not be very useful in your case.

Not sure if you're interested, but we (Scrapinghub) do offer a Kimono to Portia migration https://blog.scrapinghub.com/2016/02/25/migrate-your-kimono-projects-t… Otherwise, I'd recommend you check out Portia (open source). We're in the middle of releasing the beta 2.0 version.

There is ScrapyRT: http://blog.scrapinghub.com/2015/01/22/introducing-scrapyrt-an-api-for… In the project I work on we do have the usual periodic crawls and use ScrapyRT to let the frontend trigger realtime scrapes of specific items, all of this using the same spider code. Edit: Worth nothing that we trigger the realtime scrapes via AMQP.

All sorts of careers, for example: - developers who want to develop some data-based product (a travel agency website, who finds the best deals from airline companies); - lawyers can use it to structure the data from Judgments and Laws (so that they are able to query the data for things like: which judges have interpreted this law in their judgments) (more on this: http://blog.scrapinghub.com/2016/01/13/vizlegal-rise-of-machine-readab… - (data-)journalists who work on investigative data-based articles (they use it to gather the data to build visualizations, infographics, and also to support their arguments). - real state agencies can use it to grab the prices of their competitors, or to get a map of what people are selling, what are the areas where there is more demand. - large companies that want to track their online reputation can scrape forums, blogs, etc, for further analysis. - online retailers that want to keep their prices balanced with their competitors can scrape the competitors websites collecting prices from them. More on Quora: https://www.quora.com/What-are-examples-of-how-real-businesses-use-web…

There are AI solutions that can work on the HTML and manage to avoid the ads. Checkout https://blog.scrapinghub.com/extracting-clean-article-html-with-autoex…

ScrapingHub was something I was interested in - as an example of opensource project which the founders used as a base to build a company. Does it help to use their model for you - https://blog.scrapinghub.com/2015/03/16/history-of-scrapinghub

Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash. [0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-in-scrapy-… [1] https://splash.readthedocs.io/en/stable/index.html [2] https://github.com/scrapy-plugins/scrapy-splash

How a colleague of mine managed to get internet access in rural Uruguay https://blog.scrapinghub.com/2016/04/27/a-not-so-short-story-on-gettin…

Disclaimer, I wrote the article. > I'm not fond of the implication at the end that scraping is justifiable because old websites are dinosaurs without APIs, and those websites are jerks for not doing so, and therefore scraping is the moral thing to do. It was not my intention to give that implication. The main implication behind CityBikes is that public services should already provide this information since, well, it is a public service. On the same line, a private company providing a public service should already do so. See motives [1]. > I've scraped my share of BuzzFeed data and Foursquare data to make data visualizations (with the latter explicitly saying "don't scrape" in their Terms). But if either one told me to stop and take down my results, I would not contest, since data is what drives the Internet ecosystem. Same as CityBikes is doing. If we receive a cease and desist, we remove their service from our API. As for Foursquare, I do not see Foursquare as a public service. Your taxdollars at work, and all that. I tried to keep the article balanced but maybe it wasn't clear. There are many transportation companies willing and happy to be scraped, or looking forward to provide their information for people to reuse [2]. [1]: https://blog.scrapinghub.com/2016/03/30/web-scraping-to-create-open-da… [2]: http://nabsa.net/current-members/