Perishable Press

You're missing the point - there is no 1px gif, or hidden link. The only place that you can find the link is in the robots.txt file, and it's set to Disallow: http://perishablepress.com/robots.txt So if you're a "bad bot", you'll go to /blackhole/ looking for juicy stuff, and get banned. If you're Google you won't follow it (or he has a whitelist) and if you're an end user you won't even see it. Edit: Ah, my bad. He does put a link to it in his footer.

So I'm dealing with this now too, although at about 10x the traffic mentioned in the post. The first place I started was using the 5G Blacklist/Firewall [http://perishablepress.com/5g-blacklist-2013/] which is really just a great set of .htaccess rules for blocking known bad bots. Legit bots will respect your robots.txt, so if one (looking at you Yandex) is getting too aggressive, slow them down with the 'Crawl-delay: 60' (time in s) directive. Of course rogue bots don't respect this, so they get added to the blacklist rules based on UserAgent. What I've discovered though is that bots are not my biggest worry; it's the scrapers that are stealing my client's content and re-posting it. We've successfully filed 5 DMCA complaints at this point which have been effective at stopping the known offenders, but the crawlers continue and new copycat sites keep popping up. I've found that running a 'grep Ruby access_log' returns a good chunk of the offending crawlers (not just Ruby, also search for Python and Java). Running 'host (ip address)' almost always traces back to AWS. These log entries also very rarely list a referrer. Obviously not all of the grep results are malicious. A little research can reveal the IP is linked to a service or co you want crawling your site. For those that are unknown, they usually get an IP ban until I can determine otherwise (which the client is totally OK with, they don't want their content re-published off-site). I've thought about setting up a…

after reading the very first thing i thought was: would be comical if the OP's blog were about AI i was wrong but i could find on the 1° page this [0]... which seem a pretty hypocrite act to consider a technology that violates copyright at its core. "you eat what you plant" [0] https://perishablepress.com/diving-into-ai/

I just block them by User Agent string[1]. The rest that fake the UA get clobbered by rate limiting[2] on the web server. Not perfect, but our site is not getting hammered any more. [1] https://perishablepress.com/ultimate-ai-block-list/ [2] https://github.com/jzdziarski/mod_evasive

Edit -- Perishable Press feed: https://perishablepress.com/feed/atom

I found a recent blog post on what seems to be the same phenomena: https://perishablepress.com/block-random-string-comment-spam/ They didn’t find a motive though for the spam. To me what’s interesting is that the strings are variable length and the emails appear to be real emails rather than randomly generated ones.

This. Trusted combo: Fail2Ban + 7G firewall https://perishablepress.com/7g-firewall-nginx/#download

What about just keeping existing flags in Chrome [0] which used to be there? Is it too much to ask? I don't want to use Google search + Chrome so I can get scrolled into middle of a page without any context with random highlighted text, because Google search believes this is what I want to see. Thankfully switching to Brave solved my issue and this stupid feature is disabled. [0] https://perishablepress.com/disable-chrome-scrolltotextfragment/

Is anyone on HN doing WordPress administration? I recently 'inherited' a webshop built on WP/WooCommerce, and all the conflicting security advice in the WP space is making my head spin. There are a dozen competing 'security' plugins, with some saying 'you don't need any of them, WP is secure enough by default', and others saying 'you actually need ModSecurity [1] / Jeff Starr's nG firewall [2] on your server'. The agency that (shoddily...) built the webshop installed Wordfence Free [3], so I've just kept that for now, though I feel it's kind of slow (but that might just be caused by the bottom-of-barrel performance of the shared webhost it's currently running on). [1] https://github.com/SpiderLabs/ModSecurity [2] https://perishablepress.com/7g-firewall/ [3] https://www.wordfence.com/

This update was a really bad decision, because it broke my wordpress websites which used carbon fields (https://carbonfields.net). This plugin disables Gutenberg https://perishablepress.com/disable-gutenberg/

Everyone's coming up with hollywood style explanations (it's encryption! it's AI!) but as a former pentester, one of the more likely explanations is that it's probably just looking for vulnerabilities in an automated way. The first thing you do when you get an account somewhere as a pentester is to try to stuff as many strings as possible into weird places just to see what happens. I don't think automated techniques are very effective (or weren't in 2016) but it seems more likely to be vuln hunting than choosing your Wordpress blog for encrypted comms vs established places like Twitter. EDIT: faizshah https://news.ycombinator.com/item?id=34866169 points out that https://perishablepress.com/block-random-string-comment-spam/ observes the same phenomenon, and the author notes that all of the IP addresses come from Russia. This seems to lend credence to the idea that it's looking for vulns, since, well, lots of traffic from Russia is looking for vulns.

There are issues with using `:` in an URL, if you plan on using the URL in a way that's compatible with the extant software out there. I remember: - I remember the Rails community trying to use `;` which broke Mongrel 1. Mongrel's parser was generated from the RFC. There was a huge flame war about that back in the day. The Rails core team at the time thought that Mongrel should make an exception to a reserved character. (And after all was said and done, it got changed back to `/` for that particular use-case). - When working on IPv6 support about 3 years ago, one of the things I added to an open source Ruby project was IPv6 literals into the URL. This was a case of using `:`. Even though this was defined in the RFC specifying the literal, I found out at that time the Ruby standard library was written in a way that assumes you would never have `:` in the URL other than to delimit the port. I ended up having to do some workarounds for that. That's with Ruby. I wouldn't be surprised if many other extant libraries parsing URLs that might break -- at least not without escaping those characters. See: https://perishablepress.com/stop-using-unsafe-characters-in-urls/ You don't NEED ":". You NEED some sort of delimiter that can clearly distinguish between database and dataset; you happen to pick ':' to satisfy that. There might be a different delimiter that works better. The other option is to not pretend that is a URL and call that something else. Post-script: I think this…