Incolumitas
incolumitas.com · Personal blog · on HN since 2016
Nikolai Tschacher writes about web scraping, bot detection, VPN and proxy fingerprinting, and security research such as package typosquatting.
What HN saysCommenters call the scraping-like-the-big-boys article their favorite and forbidden knowledge, and praise it as a great blog on bot detection.
32HN people shared it
19linked it in comments
17stories submitted
1.2Kstory points
2 months agolast shared
2016people per year2026
Most-shared posts
- 1Typosquatting programming language package managers14 people · 494 points · first shared 2016 · 143 comments on HN
- 2Scrape like the big boys4 people · 697 points · first shared 2021 · 189 comments on HN
- 3Typosquatting in Programming Language Package Managers [pdf]4 people · 3 points · first shared 2016 · discussion on HN
- 4Homepage3 people · first shared 2022
- 5TCP/IP Fingerprinting for VPN and Proxy Detection2 people · 4 points · first shared 2021 · 1 comments on HN
- 6Fuzzing the WPA3 Dragonfly Handshake1 people · 3 points · first shared 2019 · discussion on HN
- 7Programming to Improve Your Life1 people · 3 points · first shared 2019 · discussion on HN
- 8BotOrNot (2021)1 people · 3 points · first shared 2021 · discussion on HN
- 9Breaking the Google Audio ReCAPTCHA with Google's Own Speech to Text API1 people · 2 points · first shared 2021 · 1 comments on HN
- 10On High-Precision JavaScript Timers1 people · 2 points · first shared 2021 · discussion on HN
- 11Achieving Your Dreams1 people · 2 points · first shared 2025 · discussion on HN
- 12The Rise of AI and Its Implications for Entrepreneurs1 people · 1 points · first shared 2026 · discussion on HN
- 13Browser based port scanning1 people · first shared 2022
- 14Avoid puppeteer and playwright for scraping1 people · first shared 2022
- 15Web scraping puppeteer aws lambda1 people · first shared 2021
What HN people wrote when they linked it
reincoder · 2023-09-14
https://incolumitas.com/
This is my all-time favorite article: https://incolumitas.com/2021/11/03/so-you-want-to-scrape-like-the-big-…
I used to do freelance web scraping, and that article felt like some kind of forbidden knowledge. After reading the article, I went down the rabbit hole and actually found a Discord server that provided carrier-grade traffic relay from a van which contained dozens of phones.
For the questions..... we have to kinda wait a bit, someone from our engineering team might come here and reply.
By the way, as I have you here have you considered converting the CSV files to MMDB format? I was planning to do that with our mmdbctl tool later today.
https://github.com/ipinfo/mmdbctl
incolumitas · 2023-09-14
Big fan of what articles? On https://incolumitas.com/ or on https://ipapi.is/?
Great idea with latency triangulation, I used latency information for a lot of things, especially VPN and Proxy detection.
But I didn't assume you can obtain that accurate location. I am honestly impressed. But latency triangulation with 600 servers gives some very good approximation. Nice man!
Some questions:
- ICMP traffic is penalised/degraded by some ISP's. How do you deal with that?
- In order to geolocate every IPv4 address, you need to constantly ping billions of IPv4's, how do you do that? You only ping an arbitrary IP of each allocated inetnum/NetRange?
- Most IP addresses do not respond to ICMP packets. Only some servers do. How do you deal with that? Do you find the router in front of the target IP and you geolocate the closest router to the target IP (traceroute)?
benmmurphy · 2022-03-10
> I think there’s literally no way for the backend to tell that the request isn’t sent by my browser and is actually being sent by a random Python program.
Oh my sweet summer child. Unfortunately, there is a whole industry built around this. This a great blog discussing different detection methods: https://incolumitas.com/
JDye · 2025-12-29
I'm testing using our residential proxies.
It's a super cool tool, I've been wondering about an open source tool doing this since reading about the technique in one of Nikolai Tschacher's blog posts years ago (https://incolumitas.com/pages/about/).
There's a few ways to work around this, but I think it's one of the best signals available to detect low-effort/common proxy providers.
Mahn · 2016-06-08
> In the thesis itself, several powerful methods to defend against typo squatting attacks are discussed. Therefore they are not included in this blog post.
http://incolumitas.com/data/thesis.pdf section 5 "Practical implications". Just wanted to point out that in case you skipped it it's worth a read, some interesting proposals there that are worth discussing with package manager maintainers.
I particularly like the preemptive approach of auto-blacklisting common typos by simply monitoring the number of times a specific unexisting package is requested over time (5.10). So if a lot of people regularly attempt to install the unexisting package "reqeusts", it could signal that it's a common typo and should be blacklisted to prevent malicious use in the future. False positives could always be sorted out manually by communicating with the package manager maintainers.
smallerfish · 2022-11-22
Sort of, and only with a lot of effort: https://incolumitas.com/2021/01/10/browser-based-port-scanning/
azalemeth · 2022-07-23
It's easier to detect that someone is using a SOCKS tunnel – from memory, one way it might be exposed is if the packet TTL is incongruous [1] as I don't think SOCKS rewrites those.
At the height of the pandemic I travelled to Denmark for work (on a clinical trial) and had a UK negative covid test to report to the UK government that I hadn't got around to – whose website geo-blocks people reporting covid tests from outside a UK IP address (even if, e.g. you'd just left it and wanted to report a negative test taken the day before). A SOCKS proxy was detected and I got a "we cannot verify you are in the UK" message. A wireguard VPN worked fine.
[1] https://incolumitas.com/2021/03/13/tcp-ip-fingerprinting-for-vpn-and-p…
Ian_Kerins · 2022-01-12
This has a lot of good info on how to cloudflare and others work, and more creative ways to bypass them if the easier options don't work https://incolumitas.com/2021/05/20/avoid-puppeteer-and-playwright-for-…
moviuro · 2021-03-02
closed · 2018-10-27
Were the research / data leakage packages from the student who did their thesis on the security of pypi etc..?
https://incolumitas.com/2016/06/08/typosquatting-package-managers/
sdiepend · 2017-09-15
Related to this? http://incolumitas.com/2016/06/08/typosquatting-package-managers/
edraferi · 2017-09-15
I appreciate the proactive approach.
Is your project the author of the packages identified by NBU? If so:
(1) Why is the tracking pingback obfuscated?
(2) Why does the code include a cheeky hello instead of a link to https://www.pytosquatting.org/ ?
(3) Why is there not a visible warning when installing one of these packages?
=================
edit:
Reading through the linked blog post [0], it appears these researchers used different code that DID provide visible warning and an cleartext pingback. It also collected command history and hardware information.
[0] http://incolumitas.com/2016/06/08/typosquatting-package-managers/
hannob · 2017-09-15
Ok, here's some ugly backstory on this:
This problem has been known for a while, yet both the pypi devs and the python security team decided to ignore it.
Last year someone wrote his thesis describing python typosquatting and standard library name squatting:
http://incolumitas.com/2016/06/08/typosquatting-package-managers/
However after that the packages used in this thesis - the most successful one being urllib2 - weren't blocked, they were deleted. Benjamin Bach was able to register urllib2 afterwards. Benjamin and I decided that we'd now try to register as many stdlib names as possible.
See also:
https://www.pytosquatting.org/
tardygrad · 2017-08-13
Having frivolous modules (ala left-pad) brings down the quality of the ecosystem as a whole.
Even if you don't use them directly one of the modules you do include could include them, and they're pretty much only a source of security holes.
NPM in particular could do with much more policing. Between module hijacking, typosquatting, and a lack of security audits of any sort, fewer packages on NPM is a good thing in my opinion.
http://www.drinchev.com/blog/alert-npm-modules-hijacked/
http://incolumitas.com/2016/06/08/typosquatting-package-managers/
wolfgang42 · 2017-08-01
This attack has been previously described in the paper "Typosquatting package managers":
Paper: http://incolumitas.com/data/thesis.pdf
Blog post: http://incolumitas.com/2016/06/08/typosquatting-package-managers/
Discussion: https://news.ycombinator.com/item?id=11862217 https://www.reddit.com/r/netsec/comments/4n4w2h/
The paper also discusses possible mitigation measures, including prohibiting registering new packages within a certain Levenshtein distance of existing packages and using additional namespacing.
micaksica · 2017-06-07
There was a thesis on this a year ago:
http://incolumitas.com/2016/06/08/typosquatting-package-managers/
After the "left pad" debacle, npm provided protection from typosquatting a deleted module:
http://blog.npmjs.org/post/141577284765/kik-left-pad-and-npm
lima · 2016-08-08
This.
Installing modules from npm is dangerous enough. Nice for education or playing around, unsuitable for a serious developers' workstation.
Related: http://incolumitas.com/2016/06/08/typosquatting-package-managers/
danjoc · 2016-07-04
tantalor · 2016-06-08
The doc (http://incolumitas.com/data/thesis.pdf) does have a short section on ethics, but IMHO it completely misses the point of the ethical concerns in running unauthorized, non-sandboxed code on devices you don't own. Instead it justifies the research by saying the threat cannot be shown unless the vulnerability is exploited, which is true, but that fact does not justify the research.
The acknowledgements mention 2 of the university advisers and a PyPi admin consented to the "notification program".
Still, people with good intentions have been prosecuted and convicted for less. I would be very concerned for this student.