Leonard Richardson

Good resource - I've been using BeautifulSoup[1] for the scraper I set up for my needs, and it's probably worth checking out as well! [1]: http://www.crummy.com/software/BeautifulSoup/

BeautifulSoup is in fact still actively maintained. “The current release is Beautiful Soup 4.1.3 (August 20, 2012).” http://www.crummy.com/software/BeautifulSoup/ I hear it recommended the most among Pythonistas, and it's plenty clean and fast for my use. But if you're skeptical, I'd still look for a more up to date benchmark (or run your own) rather than rely on results from >4 years ago.

Yup, but all of the stuff in the Quick Start section of http://www.crummy.com/software/BeautifulSoup/bs4/doc/ has more or less close equivalents in lxml.html. That being said, I think it's great that he's still maintaining BS, and porting it to Python 3 in particular - it keeps existing code working and will allow more people to make the switch. And a responsible and committed maintainer is good advertising for a package in itself, of course.

I echo the nod for Metaprogramming Ruby Quite honestly, I haven't gotten into Ruby Best Practices yet, but credit must be given to Gregory Brown for making it freely available: http://rubybestpractices.com/ As a middling programmer trying to learn Ruby, O'Reilly Ruby Cookbook has been by far the best reference. Great example code, great explanation, and best of all, the recipes are actually useful: http://www.crummy.com/writing/RubyCookbook/

There are a few variations on CSV, but a great many variations on HTML. You could try to hack together some sort of quick and dirty regular expression based extractor, but if I were you I'd use http://www.crummy.com/software/BeautifulSoup/. It does a reasonably good job of parsing HTML, even invalid HTML such as is often found in the wild. Then it should be pretty straightforward to scrape out the data you want. The scraper will be reasonably resilient and because the parsing code is separate from the code that deals with the structural details of the data source you're dealing with, it will be reasonably easy to maintain that when the provider redesigns their web pages.

This reminds me of UltraGleeper, a recommendation system with "indie rock" sensibilities: http://www.crummy.com/software/UltraGleeper/IntroPaper.html Past a certain threshold, additional links to an item decrease its score rather than increase it. This is useful because you've probably already heard of (or will hear of) the most popular stuff. The recommendation system instead tries to find things that you aren't already finding on your own.

Have you tried following the instructions on the BeautifulSoup site[1]? The HTML parser it depended on was removed from the Python standard library, and its replacement doesn't work as well at processing bad markup. You should install html5lib[2] and use it for parsing: import html5lib from html5lib.treebuilders import getTreeBuilder parser = html5lib.HTMLParser(tree=getTreeBuilder("beautifulsoup")) soup = parser.parse(open("mydocument.html")) [1] http://www.crummy.com/software/BeautifulSoup/3.1-problems.html [2] http://code.google.com/p/html5lib/

I know this is ruby but might be worth a look. Will help you no end and no need for a web browser. http://mechanize.rubyforge.org/mechanize/ might be worth a look http://www.crummy.com/software/BeautifulSoup/

Unlike Ruby Python doesn't have great facilities for writing making implicit APIs like this: with Table(): header(...) row(...) You can do something like this but I'd recommend against it as it involves a lot of magic and has trade-offs which make it an uncommon choice. Instead you'd do something like: with Table() as table: table.header(...) table.row(...) This follows through to your template API as well. So you'd want to construct an object that represents your template, and have methods on that to add a table. For example: page = sierra.Page(...) with page.div(...), page.add_table() as table: table.header(...) table.row() print(page) # __str__() should return the raw HTML response This simplifies things, and you can use an in-memory StringIO object rather than a file. You can then return this string to flask as a text/html response and avoid Jinja2 entirely. Also don't use CamelCase, please! It looks very unpythonic. However you're really trying to create an API for producing a tree of nodes (some of which might logically map to individual tags, some of which might not) which are then serialized to HTML. I'd recommend looking at how you can use BeautifulSoup for a lot of this. BS4 contains a lot of classes and functions for creating trees of HTML-like nodes, so you could use this rather than a StringIO object and directly writing "<div>" and having to support all of the attributes as method…

I feel similarly about Python. I never really had to interact with it beyond using it as a calculator on the command line until a project I was a part of earlier in the year had a utility Python script introduced and I needed to hack on it to make desired changes. Then, more recently, I started using BeautifulSoup[0] for personal scraping. Of course it feels more trivial now (and now I wish I documented my frustrations so I could be more specific and perhaps help out other Python learners in the future), but it does take a while to adjust to Pythonic code where everyone is using cultural conventions like _foo and __foo and you wonder what the hell __foo__ is[1]. More on-topic, searching is a major weakness of Python's docs. What's that? Python just got multi-line strings allowing embedded expressions? They're called f-strings? Let me go read about them[2]. I can't link it, but even using Startpage the immediate results are a post from RealPython and PEP498 and while both are great, they aren't terse enough to be frequently referenced. If you search for "formatted string literal"[3] in the docs it takes 10 results before you reach something explicitly talking about them. And then you need to follow the links a couple times to reach https://docs.python.org/3/reference/lexical_analysis.html#f-strings (notice it is anchored as "f-strings" which was the first term we searched for). While the Python docs, when I've found them, have been adequate for me so far -- although I still…

That really depends on your project and tech stack. If you're into Python and are going to deal with relatively static HTML, then the Python modules Scrapy [1], BeautifulSoup [2] and the whole Python data crunching ecosystem are at your disposal. There's lots of great posts about getting such a stack off the ground and using it in the wild [3]. It can get you pretty darn far, the architecture is solid and there are lots of services and plugins which probably do everything you need. Here's where I hit the limit with that setup: dynamic websites. If you're looking at something like discourse-powered communities or similar, and don't feel a bit too lazy to dig into all the ways requests are expected to look, it's no fun anymore. Luckily, there's lots of js-goodness which can handle dynamic website, inject your javascript for convenience and more [4]. The recently published Headless Chrome [5] and puppeteer [6] (a Node API for it), are really promising for many kinds of tasks - scraping among them. You can get a first impression in this article [7]. The ecosystem does not seem to be as mature yet, but I think this will be foundation of the next go-to scraping tech stack. If you want to try it yourself, I've written a brief intro [8] and published a simple dockerized development environment [9], so you can give it a go without cluttering your machine or find out what dependencies you need and how the libraries are called. [1] https://scrapy.org/ [2]…

I learned Ruby and Ruby on Rails much faster than I had any right to, when my new job in NYC required it. I practiced not by writing Ruby on the job, but writing Ruby to scrape Craigslist apartment listings and feeding them into a spreadsheet. I started teaching myself computer programming in 2011/2012: I bought a bunch of Head First books published by O'Reilly, watched the entry-level CS video lectures that Stanford provided for free, participated in the first few Udacity courses, and I tried a couple Coursera classes. But I think I learned the most (in the shortest amount of time) when I went through your online book, 'The Bastards Book of Ruby'[0] I loved the real-world examples you used: fetching Tweets through the Twitter API, web scraping with Nokogiri, and manipulating images with ImageMagick/RMagick are a few that stick out to me. I'm sure you're doing great things at Stanford but I'm also confident that you could come out with a new book (covering the topics you spoke of above) that would help motivate people who are on the fence on whether or not they should start/continue learning computer programming. I'd be happy to collaborate on something but I would argue that Python would be a better language to use than Ruby. Here are some of the topics I'd like to share with people (which I've found to be useful in my career/side-projects): reading data from CSV/Excel files (xlrd[1]), fetching data from APIs (requests[2]), web scraping (BeautifulSoup[3], Selenium[4]),…

> pretty sure that's what BS uses under the hood? it's an option[1], and my strong advice is to not use lxml for html since html5lib[2] has the explicitly stated goal of being WHATWG compliant: https://github.com/html5lib/html5lib-python#html5lib 1: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-pa… 2: https://pypi.org/project/html5lib/

A detailed analysis of what would happen if Linus Torvalds got hit by a bus was conducted[1] as far back as in 2000. Unsurprisingly the result showed this would most likely have had a devastating impact on the development of Linux. [1]: https://www.crummy.com/writing/segfault.org/Bus.html

> fun fact: most books published in the US before 1964 are in the public domain <https://www.crummy.com/2019/07/22/0> Submitted to HN 2019 August 02. 62 comments. <https://news.ycombinator.com/item?id=20591071>

Why the web archive link? (why the post at all OP?) The original is archived over here on the author's website along with the rest of his segfault writings: https://www.crummy.com/writing/segfault.org/Bus.html

Under US law, between 1925 and 1964, it was necessary to renew copyright every 28 years.[1] There are many works which have fallen out of copyright for nonrenewal, and there are projects which are now going through copyright renewal records to determine which of these are in fact now public domain in the US. Roughly 3/4 of potentially copyrighted works (published since 1924 and initially registered) have proved to be public domain. Once copyright has lapsed, it cannot be reinstated. https://www.crummy.com/2019/07/22/0 https://www.nypl.org/blog/2019/05/31/us-copyright-history-1923-1964 ________________________________ Notes: 1. Technically, the obligation existed until the 1976 act, at which time copyright status was automatic, but was retroactively waived for works never having lapsed, back to 1964, in the 1992 act. It's complicated. (NYPL link above.)

https://www.crummy.com/software/BeautifulSoup/ https://github.com/encode/httpx

Mostly this: https://requests.readthedocs.io/en/master/ https://www.crummy.com/software/BeautifulSoup/bs4/doc/

https://www.crummy.com/software/BeautifulSoup/bs4/doc/ I mean... soup.find(id="link3") # <a class="sister" href="http://example.com/tillie" id="link3">Tillie</a> Is this hard?

I use a modified Scrape ‘N Feed to scrape the sites I need without RSS. With this I can go another level deeper and scrape the entire article and return that back to the feed instead of just getting the snippet. https://www.crummy.com/software/ScrapeNFeed/

> Is there a better way to surf the web, retrieve the source code of the pages and extract data from them ? Yes, of course! To get the source code of a web site you don't need a browser and all its complexity. It makes me so sad how far we have come in terms of unnecessary complexity for simple tasks. If you want to extract data from web pages without requiring hundreds of megabytes for something like Electron, there are lots of scraping libraries out there. There are for example at least two good Python implementations: Scrapy[1] and BeautifulSoup[2]. [1]: https://scrapy.org/ [2]: https://www.crummy.com/software/BeautifulSoup/

You can use the BeautifulSoup API with the `lxml` parser: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#installing-a-pa… I've heard that `lxml` can choke on certain badly-formed markup, but it's very fast. Personally has never failed on me.

For simple, scripted, one-off tasks, I've had good experiences with Beautiful Soup for Python[1]. If it's necessary that the javascript actually gets executed though, you're more likely to need something like headless chrome (recently on HN [2]) or PhantomJS [3], but I can't speak to how easy they are to use, though. [1]: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ [2]: https://news.ycombinator.com/item?id=14101233 https://news.ycombinator.com/item?id=14239194 [3]: http://phantomjs.org/

I wonder if Unicode, Dammit would be helpful. https://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit

Like the very usable Beautiful Soup: http://www.crummy.com/software/BeautifulSoup/

for script in soup_obj(["script", "style"]): script.extract() # Only need these two elements from the BS4 object That comment is a bit misleading. soup.extract() will remove those tags from the tree, 'script' and 'style' are the two tags you __don't__ need. http://www.crummy.com/software/BeautifulSoup/bs4/doc/#extract

Very common problem in web scraping, for example forum site might contain a mix of MacRoman, Windows codepage and various european codepages in a single page (yes, even in 2014!). Seems like a more advanced version of UnicodeDammit module ( http://www.crummy.com/software/BeautifulSoup/bs4/doc/#unicode-dammit ). Note: this module, like UnicodeDammit, is very US/English-centric, and is practically useless for worldwide web scraping. For non-english pages, it is necessary to statistically estimate the codepage and language of each page segment, and then try to normalize each segment to unicode.

Well, web scraping can be fun, too! But depends on what you know about the page. If you don't know anything, it's tough. But if you have an idea about how it's structured... (see below) http://www.crummy.com/software/BeautifulSoup/bs3/download/2.x/document…

Python, not Ruby.. but Beautiful Soup is a lenient parser and a really awesome library: http://www.crummy.com/software/BeautifulSoup/