DataDome Blog

There are risk services out there. https://sift.com/ Is one you call out to that gives you a risk score. https://datadome.co/ can sit within your cdn layer that does risk assessment.

Basically, we are using two sort of technics : technical and behavior. Technical : if the UserAgent claim to be a regular browser (let say Chrome 43) we will check on network level if the client implement http protocol like Chrome 43 usually do and on the JS side if the Javascript render is correct for Chrome. In case it's a real Chrome, we will check if the Browser is controlled by automation Tool. Behavior : we will check if the path of requests is regular according to the website usage. Disclaimer: I'm working at https://datadome.co, a bot protection tool.

DataDome | NYC / Paris | Full Time | Onsite or Remote https://datadome.co/ https://docs.datadome.co/ Bot hunters is a full time job! With our SaaS cybersecurity technology, we detect and manage non-human traffic in real-time to protect our clients’ websites and APIs. Named as a Strong Performer by Forrester, DataDome is a growing cybersecurity pioneering in bot detection and protection for content publishers, eCommerce websites and advertisers across the world. We handle and protect more than 4 billions requests daily below 3ms (99p). Our stack: - Stream data processing: Kafka / Flink / Elasticsearch - Real time detection engine: Java / Scala - Webservers modules (nginx, haproxy…): C++ / C#, nodeJS, lua - Infrastructure: 1000 VMs (AWS/GCP/Azure/baremetal) - Dashboard: Angular7 / Symfony3 We’re expanding our team and are looking to hire: - Cyber Security Analyst / Threat researcher - Senior Software Engineer : Java, Scala, C++ - Data Engineer / Data Scientist : Scala, Java, Python - DevOps/SRE Engineer - FullStack Web Engineer : Angular7 / Symfony3 Join us: [email protected] or https://www.linkedin.com/jobs/datadome-jobs-worldwide

DataDome | NYC / Paris | Full Time | Onsite https://datadome.co Bot hunter is a full-time job! With our proprietary SaaS cybersecurity technology, we detect and manage non-human traffic in real-time to protect our clients’ websites and APIs. Named as a Strong Performer by Forrester, DataDome is a leading cybersecurity company pioneering in bot protection for eCommerce, marketplace and classified websites across the world. We handle and protect more than 4 billion requests daily below 3ms (99p). Our stack: • Stream data processing: Kafka / Flink / Elasticsearch • Real-time detection engine: Java / Scala • Webservers modules (nginx,haproxy…): C++ / C#, nodeJS, lua • Infrastructure: 200VM (AWS/Azure/baremetal) • Dashboard: Angular6 / symfony3 We’re expanding our team and are looking to hire: • Solution Architect (NYC) • Technical Lead (NYC) • Data Engineer / Data Scientist : Scala, Java, Python (Paris) • FullStack Web Engineer: Angular7 / Symfony (Paris) Join us: [email protected]

> The IPs of these bots here actually do not come from datacenters or VPNs most of the time; the overwhelming majority come from residential and mobile networks. So I started searching for what these residential proxy networks actually are. https://datadome.co/bot-management-protection/how-proxy-providers-get-…

>large blocks of IP addresses This could be at least partially solved by starting legal and cybersec (bulk blocks, flagging SDKs as malware) action against botnets for rent[0], forcing their SDKs out of app stores[1]. 0 – https://spur.us/residential-proxies-the-legal-botnet-that-nobody-talks… 1 – https://datadome.co/bot-management-protection/how-proxy-providers-get-…

People are paying $500 for bots used to buy the latest Nike/Adidas/... limited edition sneakers. Or videocards a few years ago (for crypto mining). It's a whole industry. > If we consider a user base of ~175 users, and a minimum bot price of 200 euros (175 users x 200 euros), then the bot developers made at least 35K euros (~$37K USD) in initial bot sales. https://datadome.co/threat-research/inside-sneaker-bot-business/

I'm not aware of a benchmark, but puppeteer-extra-plugin-stealth can be detected: https://datadome.co/bot-management-protection/detecting-headless-chrom… Crawlee does appear to do the basic checks though, like checking navigator.webdriver: https://github.com/apify/crawlee/blob/master/test/browser-pool/browser… Last time I checked (over a year ago) I couldn't find any public code to make Chrome/Firefox properly undetectable. That said, going to extreme lengths to be undetectable is rarely necessary, because some sites will serve up CAPTCHA's to real people on clean uncompromised residential connections anyway.

If your IPv4 is shared, it might be because of the other users doing bad things on the same IP. If you have a compromised device on your network, it might be being used as a proxy to attempt logins, or for other uses as a "residential proxy": https://datadome.co/bot-detection/how-proxy-providers-get-residential-…

Meant to post something like this before. If it’s helpful here’s a pretty decent article on current bot mitigation issues: https://datadome.co/bot-management-protection/bot-detection-how-to-ide…

This (agent detection) is now a kind of emerging space. Obviously it'll get much more important, too. Other products in the space: - Foil (https://usefoil.com/), I'm biased, a friend is building this - Kasada https://www.kasada.io/ - DataDome (https://datadome.co/) - Castle (https://castle.io/) - Fingerprint (https://fingerprint.com/) - HUMAN (http://humansecurity.com/) - Google Cloud Fraud Defense, which is basically the updated reCaptcha (https://cloud.google.com/security/products/fraud-defense?hl=en) - this, Cloudflare Precursor It seems like some of the main reasons people care so far are: - Preventing automated credential stuffing - Preventing bots from creating a bunch of fake accounts (eg free trial abuse, which can also lead to high twilio SMS bills!) - Reducing payment fraud - Blocking LLM scraping - Blocking automated scalpers (!) eg for tickers or sneakers I'm curious to see which use cases end up dominating as the reason companies care about this. And I'm hopeful that my agents will still have good ways for me to browse and do things on the web on my behalf - eg detect agents and route them to an agent path, rather than blocking them. (I'm interested in tools for detecting AI agents and seeing how this shifts as bot traffic goes way up.)

Here are some resources: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers The article states that these may bypass `robots.txt`, so you may not even bother with that. There are people who use facebook as a proxy for scraping with the preview functionality. https://datadome.co/threat-research/how-facebook-was-used-as-a-proxy-b… >With technical detection only, the fraudulent requests would have been indistinguishable from legitimate API calls, since they had the Facebook user agent and IP address. However, our heuristic analysis uncovered that certain parameters, unlikely to be used by humans, were overrepresented in the URLs that Facebook requested. Certain use cases were also unlikely: for example, a human user would typically share a link to a specific ad, not to a page of search results. According to that article, looking at the requests and discerning patterns of bogus traffic would be preferable to more drastic approaches such as blocking or rate limiting indiscriminately, and risking to impact legitimate traffic.