Internet Archive Blog

Internet Archive's trustworthiness took a hit when they waded into fact checking - https://blog.archive.org/2020/10/30/fact-checks-and-context-for-waybac… and wiping content they disapproved of - https://news.ycombinator.com/item?id=32743325

We have a big and ongoing focus on amateur radio right now thanks to a grant from ARDC and really hard work by our awesome Program Manager Kay Savetz. Worth checking out: https://blog.archive.org/2022/11/16/digital-library-of-amateur-radio-c… https://archive.org/details/dlarc

The most recent notice IA have blogged was in 2017, and there's no indication that the service has reversed course on robots.txt since. <https://blog.archive.org/?s=robots.txt>

Before building an app that depends on the Wayback Machine (or other Archive infrastructure) it's good to keep in mind this post from their blog: <https://blog.archive.org/2023/05/29/let-us-serve-you-but-dont-bring-us… One of my favorite tricks when coming across a blog with a longtail of past posts is to verify that it's hosted on WordPress and then to ingest the archives into my feedreader. Once you have the WordPress feed URL, you can slurp it all in by appending `?paged=n` (or `&paged=n`) for the nth page of the feed. (This is a little tedious in Thunderbird; up till now I've generated a list of URLs and dragged and dropped each one into the subscribe-to-feed dialog. The whole process is amenable to scripting by bookmarklet, though—gesture at a blog with the appropriate metadata, and then get a file that's one big RSS/Atom container with every blog post.)

I agree that Moby-Dick [1] is “far funnier, weirder, and gripping than you likely imagine.” But that is in contrast not only to contemporary novels that might—or might not—have been influenced by television. The vast majority of English-language novels published in the 19th century were also driven by plot and character and lacked long digressions about the author’s obsessions. Thousands of such novels, most of them no longer read, are now available at the Internet Archive [2]. Some of those might be worth reading, too. [1] There’s a hyphen on the title page of the first edition but not in the body text or page heads: https://archive.org/details/mobydickorwhale01melv/page/n7/mode/2up https://archive.org/details/mobydickorwhale01melv/page/196/mode/2up?q=… [2] https://blog.archive.org/2021/07/14/forgotten-novels-of-the-19th-centu…

It's not a torrent, but I think the best way to retrieve the entire collection is using the internetarchive "ia" command line tool. Here's a couple pointers: https://blog.archive.org/2020/10/21/want-some-terabytes-from-the-inter… https://archive.org/developers/internetarchive/cli.html

http://blog.archive.org/tag/distributed-web/ My personal opinion is that they’re going about this the wrong way, with IPFS, anything blockchain, etc. They can support this entire model with a DHT (supporting PKI for identity) and torrent advertising, the latest version of BitTorrent (which has great support for cross swarm file serving by hash), and RSS or other feeds for advertising the corpus or chunks of it. I maintain a metadata copy in R2 of all IA items and their metadata, and would love to work on the ArchiveTeam equivalent of a Warrior appliance: appliance or containers starts, consumes an RSS feed to request the least seeded items, download them up to disk space allowed, and then serve downloaded item files to the swarm.

They also work with Internet Archive to send books that IA deems desirable for scanning. https://blog.archive.org/2019/11/06/for-the-love-of-literacy-better-wo…

The best I can quickly find is this: http://blog.archive.org/2020/04/27/forging-a-cooperative-path-forward-… https://docs.google.com/document/u/1/d/e/2PACX-1vTRPssQgr90yah0sp--Vyj… Which pretty clearly indicates the move was unilateral by IA, but they got by-in after the fact.

I mean even honoring the opt-out of robots.txt would be fantastic. As another commenter pointed out they willfully ignore it: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engine… It's fairly unethical.

Interestingly, only one comment [1] in one thread from all the ones 'dang pulled up [2] mentions Carl (his son) at all. Despite the article calling out that Richard Feynman's involvement with the Connection Machine was to follow along w/ Carl's involvement: > I got to know Richard through his son. I was a graduate student at the MIT Artificial Intelligence Lab and Carl was one of the undergraduates helping me with my thesis project. Also of note is that Thinking Machines is where the Long Now and Internet Archive folks had worked together (see the "group photo" from [3]). Definitely a unique group of folks came out of Thinking Machines! [1] https://news.ycombinator.com/item?id=8683903 [2] https://news.ycombinator.com/item?id=28982737 [3] https://blog.archive.org/2021/07/21/reflections-as-the-internet-archiv…

They actually posted about this in 2017: https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engine… . At the time it sounded like they might change their robots.txt policy. I guess they never followed up on it. (I checked and ezboard is still excluded.)

> When you find a page [...] and you want to make sure it's available to you later, what do you do? Instead of doing a bad and lossy job of archiving the page myself, I notify† our friendly neighbourhood archivists at the Internet Archive of the page; and they then do the best, most lossless job of preserving the page that they're able, given their cumulative experience. † http://blog.archive.org/2017/01/25/see-something-save-something/ As a side-benefit, they also then take care of keeping the archive they've made around and available online in perpetuity, with no additional marginal effort on my part. The same can't be said for something in my own "private collection."

Worth mentioning, the Internet Archive is now archiving Flash animations[0,1] using Ruffle.rs[2] [0]http://blog.archive.org/2020/11/19/flash-animations-live-forever-at-th… [1]https://news.ycombinator.com/item?id=25154128 [2]https://ruffle.rs/

For anybody interested in the IA's claimed ethical and legal justifications, they summarise and link to them at the following page: http://blog.archive.org/2020/03/30/internet-archive-responds-why-we-re…

>At best, that's a legally dubious position. Hence the lawsuit. The internet Archive is arguing this as part of the CDL's "fair use" portion. Presumably they view the program as allowing libraries to continue offering their "vital function to society" while being closed. http://blog.archive.org/2020/03/30/internet-archive-responds-why-we-re…

There's a great doc that came out last year on Marion Stokes, who recorded decades of television. Her footage is being added to The Internet Archive https://blog.archive.org/tag/marion-stokes/

TIL they still follow robots.txt -- https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engine… mentioned that they were planning to stop doing that (and I remember reading a news article based on it claiming that they already stopped following robots.txt, hence my confusion). Truly a shame. I get following robots.txt at collection time, I don't get following a robots.txt that was added later.

Original source on the Internet Archive blog: https://blog.archive.org/2019/11/06/for-the-love-of-literacy-better-wo…

They look at them, but they don't follow them strictly [1]. They make judgement calls on what they should do rather than treating robots.txt files as a legal contract. [1] - https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engine…

https://blog.archive.org/2019/04/10/official-eu-agencies-falsely-repor… The blog post on archive.org about it is worth a read. French IRU should be punished for this crap.

There is actually already an ongoing collaboration between Wikipedia and Internet Archive to snapshot almost every single web page to Wayback Machine as soon as they are referenced on any Wikipedia site: https://blog.archive.org/2018/10/01/more-than-9-million-broken-links-o… The pages are being archived at a rate of 20 million URLs/week. PS. This is not free. Internet Archive is one of those great online institutions that needs a steady stream of donations to buy the hardware that keeps the project going.

The Internet Archive, besides their fantastic work in web archiving, also does digitalization of books: https://archive.org/scanning Seem more here: https://blog.archive.org/2017/10/10/books-from-1923-to-1941-now-libera…

AWS should donate a few of these to the Internet Archive. They just specifically asked for money to help establish a second copy in Canada.[1] It would be great publicity for AWS among a demographic not far from their target... AND it would probably benefit AWS itself in some way eventually. 1: https://blog.archive.org/2016/11/29/help-us-keep-the-archive-free-acce… EDIT: "a few"?? One trip?

Excellent news. Should note that today is the 20th anniversary of the Internet Archive: https://blog.archive.org/2016/10/26/making-the-web-more-reliable-20-ye…

Here's the instructions if you want to do a batch download to your favorite offline e-whatever. https://blog.archive.org/2012/04/26/downloading-in-bulk-using-wget/

IA is very supportive of automated access, they even have a post about how to use wget to batch download lots of items: https://blog.archive.org/2012/04/26/downloading-in-bulk-using-wget/ Granted, that is not the Wayback Machine but I am sure they love people using them.

Yup. For example, the Internet Archive in SF launched what they call a bitcoin ATM back in March. [1] [2] [1] http://blog.archive.org/2013/03/05/bitcoin-to-cash-converter-box/ [2] http://blog.archive.org/2013/04/03/how-the-internet-archive-is-having-…

http://blog.archive.org/2011/09/03/hard-drive-archaeology-and-hackersp… is a great example of this.

This has been playing out for many years. And it's all because Brewster Kahle decided that an overly broad interpretation of the Internet Archive's mission trumped the rights of authors and publishers, and the laws of the United States. When IA was asked to stop CDL - many times - he continued. The National Writers Union tried to open a dialogue as early as 2010 but was ignored: The Internet Archive says it would rather talk with writers individually than talk to the NWU or other writers’ organizations. But requests by NWU members to talk to or meet with the Internet Archive have been ignored or rebuffed. https://nwu.org/nwu-denounces-cdl/ When the requests to abandon CDL turned into demands, Kahle dug in his heels. When the inevitable lawsuits followed, and IA lost, he insisted that he was still in the right and plowed ahead with appeals. He also opened a new front in the court of public opinion. In his blog posts and interviews with U.S. media, Kahle portrays the court cases and legal judgements as a crusade against the Internet Archive and all librarians (see https://blog.archive.org/2023/12/15/brewster-kahle-appeal-statement/). It's not. It's the logical outcome of one man's seemingly fanatical conviction against the law and the people who work very hard to bring new books into being. In addition, there has been real collateral damage to the many noble aspects of the Internet Archive. Legal fees and judgements have diverted resources away from the Wayback Machine,…