Intoli Blog

Awesome, I really love meta-analyses too! Our product involves web scraping so it's a natural fit to write articles about scraping and analyzing data in general. A few of our articles so far [1,2,3] would probably be classified as meta-analyses and we have some more in the pipeline! Feel free to sign up for the newsletter in the footer of http://intoli.com if you would like to read those articles after they're published :-). [1] http://sangaline.com/post/reverse-engineering-the-hacker-news-ranking-… [2] http://sangaline.com/post/the-stories-that-hacker-news-removes-from-th… [3] http://sangaline.com/post/wayback-machine-scraper/

I was exploring running FFmpeg in the cloud and I stumbled upon this: https://intoli.com/blog/transcoding-on-aws-lambda/ What would be great is FFmpeg in the cloud with GUI for easier use.

In their previous tutorial [1] their encoding settings show that they're targeting roughly 128 kb/s VBR, which is what YouTube AACs typically are, so that doesn't explain it. AAC (and Opus for that matter, which YouTube also often has) are superior formats, and lossy encoding should only happen once if possible- so this honestly makes me cringe a bit. In this case the (probably already lossy) source audio is lossily converted by YouTube and then again by ffmpeg/LAME with rather low bitrates at each step. A lot of quality is lost along the way. Support for Opus is unfortunately still somewhat sparse, but AAC support is essentially universal on modern devicese. So I can't think of any reason to favor this approach over: youtube-dl -f bestaudio[ext=m4a] "$URL" Even YouTube's 128 kb/s copies won't satisfy audiophiles. [1] https://intoli.com/blog/transcoding-on-aws-lambda/

My co-founder wrote a bit about Asciinema as part of a comprehensive comparison of terminal recorders [1]. There's a lot to love about it (wide availability, very nice JavaScript playback -- check out the Game of Life playback at [1]), but it's a bit of a chore to self host, if that's important for you. [1]: https://intoli.com/blog/terminal-recorders/

Let me be the devil advocate: why not use ttyrec + seq2gif instead? Or check https://intoli.com/blog/terminal-recorders/ if you don't like my recommendation ttyrec is a standard format that most people and tools understand. And if you want something fancy that asciinema output, you can use seq2gif which support all the bells n whistles asciinema does, while allowing you to also upload the ttyrec file. However, I would suggest not using unicode or Ansi color in your recordings. You want to demo a feature, not show your cool terminal or the amazing gif you made. The gif is just a byproduct. Keeping to plain text as much as possible makes your recording available to people using text terminals and Braille screen readers. My most recent example: https://github.com/csdvrx/pg_csdvrx/raw/master/pg_csdvrx.ttyrecord and https://github.com/csdvrx/pg_csdvrx/raw/master/pg_csdvrx.gif (in case you wonder why I'm doing a cat /dev/random into a live postgres database block file, this is to show how I can recover the database content from the raw directory. Kids, don't try this at home. Don't try it at your parents work on their production database either. In fact, just don't do that, it is only for when your database ends up in /lost+found)

Ah, thanks, now I get it: it doesn't bother them that fingerprinting lets them be tracked; it bothers them that fingerprinting (and VPN IPs) mean their scrapers get hit with captchas. FWIW, it takes very little effort to completely conceal the fact that firefox is running headless under marionette/webdriver/geckodriver. Chrom(ium) takes more effort, but these guys have solved it (and built a business around it): https://intoli.com/blog/making-chrome-headless-undetectable/ https://github.com/intoli/remote-browser Of course neither of these address fingerprinting -- all your scrape requests will have the same or similar fingerprint, which will lead to captchas pretty quickly. This might help (and might even be part of the reason for buying piles of cellphones): https://news.ycombinator.com/item?id=25379342

This article outlines how to run FFmpeg on AWS Lambda: https://intoli.com/blog/transcoding-on-aws-lambda/ With favorable cost comparisons to Elastic Transcoder. Not self-hosted, but looks like significant cost improvement. Also, one could set up a 'transcode farm' with virtual/real machines using FFmpeg running parallel with some light scripting for automation. Looking up 'render farm' might yield some ideas for distributed image computation. Just some ideas...

A bot absolutely can, you just need to use a more sophisticated bot. This article [1] is from August 2017, so the arms race has escalated and techniques improve, but the gist is the same: You just do a better job of simulating the "human" characteristics they monitor. Gen 4 bots (bots that run on an infected user's machine) can circumvent these measures as well. [1] https://intoli.com/blog/making-chrome-headless-undetectable/

This was built in '17: https://intoli.com/blog/hacker-news-title-tool/ HN discussion: https://news.ycombinator.com/item?id=14400603

I'd guess if they're crawling purely for the purpose of detecting cloaking (which would be a much smaller-scale job than the standard Googlebot indexing), they'd just use Chrome Headless[0][1] [0] https://intoli.com/blog/not-possible-to-block-chrome-headless/ [1] http://antoinevastel.github.io/bot%20detection/2018/01/17/detect-chrom…

And it’s possible to pretend not to be Chrome headless too. https://intoli.com/blog/making-chrome-headless-undetectable/

To every action there is always opposed an equal reaction... https://intoli.com/blog/making-chrome-headless-undetectable/

That file includes at least two non-standard syntax extensions[0]. Robots is just a de facto standard and respect of some directives varies[1]. So much for it being 'not difficult' while the task is not even clear because there isn't even a clear standard. Archive.org also dislikes how robots.txt is being used mainly for search engines and goes against their mission in particular[2]. Are they now hackers for not throwing away information just because someone was overzealous with robots.txt or retired a certain website and uses robots.txt as SEO to let another one take its place in Google search results? If some big corp wants to cry and bring legal matters into software they should first be accountable themselves for not securing themselves and the data of their clients (see the LinkedIn hack people mentioned elsewhere here and in general the high profile hacks like Equifax, Sony, etc.). Or should software shape up to be like many other areas today are - multi-million corporations are free to play fast and loose and endanger people while small guys get fried over meaningless bullshit and vaguely defined "crimes". [0] - https://en.wikipedia.org/wiki/Robots_exclusion_standard#Nonstandard_ex… [1] - https://intoli.com/blog/analyzing-one-million-robots-txt-files/ [2] - https://blog.archive.org/2017/04/17/robots-txt-meant-for-search-engine…