Cliqz Blog
0x65.dev · Company blog · on HN since 2019
The engineering blog of Cliqz, a European search and privacy company, describing how it built a search engine from scratch.
What HN saysCommenters recommend its in-depth series on building a large-scale search engine, calling it interesting and impressive, though one found the index weak.
42HN people shared it
31linked it in comments
37stories submitted
2.9Kstory points
4 years agolast shared
2019people per year2026
Most-shared posts
- 1Homepage11 people · first shared 2019
- 2Building a Search Engine from Scratch10 people · 488 points · first shared 2019 · 142 comments on HN
- 3The world needs more search engines7 people · 795 points · first shared 2019 · 519 comments on HN
- 4Human Web – Data Collection Without Privacy Side-Effects7 people · 72 points · first shared 2019 · 17 comments on HN
- 5A New Search Engine6 people · 173 points · first shared 2019 · 102 comments on HN
- 6Privacy or Data, a Convenient False Dichotomy6 people · 77 points · first shared 2019 · 26 comments on HN
- 7HPN – Anonymous Rate Limiting Using Zero Knowledge Proofs6 people · 32 points · first shared 2019 · 1 comments on HN
- 8Pivoting from conventional search to search-as-you-type: Our story with Mozilla4 people · 72 points · first shared 2019 · 12 comments on HN
- 9Why we Forked Firefox and Not Chromium3 people · 500 points · first shared 2019 · 122 comments on HN
- 10Not all adblockers are born equal3 people · 20 points · first shared 2019 · 1 comments on HN
- 11Competition is just one click and 27 Billion Dollars away3 people · 18 points · first shared 2019 · 1 comments on HN
- 12Blocking tracking without blocking trackers3 people · 8 points · first shared 2019 · discussion on HN
- 13Dissemination cliqz3 people · first shared 2019
- 14The Architecture of a Large-Scale Web Search Engine, Circa 20192 people · 171 points · first shared 2019 · 22 comments on HN
- 15Personalized news recommendations and privacy are not mutually exclusive2 people · 108 points · first shared 2019 · 15 comments on HN
What HN people wrote when they linked it
throwaway292893 · 2021-06-22
You know exactly what you're doing. It wasn't a big deal then, and it shouldn't be a big deal now. It's cancel culture and you're participating in it.
This is an amazing release and I'm glad Brave revived the technical marvel that was Cliqz, you can see that on display in their blog: https://www.0x65.dev/
But no, you decided to bring up bullshit drama. You're not generally curious, and even if you were, it's not really on topic. It's negative and non-constructive.
You fool no one when you hide behind your "I was just asking a question" defense. You knew ahead of time what your question would provoke.
ramesh1994 · 2021-06-22
I would also highly recommend the blog post series [1] from Cliqz talking about the tech behind the search.
[1] - https://0x65.dev/
heipei · 2020-04-30
For what it's worth, their engineering blog about how to build a large-scale search engine more or less from scratch was really interesting: https://0x65.dev/
Zhyl · 2021-03-03
For those who missed it, Cliqz did a very in-depth series of blog posts on their tech [0]. Basically, it was "word2vec: The Search Engine".
It was actually very impressive in terms of semantic understanding, but the index was pretty awful. I either got exactly what I was looking for, or no hits at all. I can imagine that they'd get massive value by focusing on more information-rich natural language sources and de-prioritising images or structured/knowledge queries for the time being. Some good partnerships to provide instant answers a-la Duckduckgo's goodies would be excellent.
I also can't express how much better the Brave brand will be for them - even 'Tailcat' is a much better name. Cliqz sounds too much like an off-brand, gutter ad seller to ever have credibility, and I say that as someone who was fairly impressed with their tech.
I am worried that they're currently pitching this as a general purpose search engine at the moment when it will be a while before it can even pretend to compete with Google. I would advise for them to find a niche where they can whip Google into next week, and then expand on scope from there.
[0] https://www.0x65.dev/
searchableguy · 2021-04-08
Most of them were already in my list. :D
I was looking for more startup oriented engineering blogs like cliqz search engine one (it's closed now): https://0x65.dev/
My list: https://airtable.com/shrkMwyhCdgVPzzbY
kapsteur · 2021-03-03
The Cliqz Tech blog : https://0x65.dev contains many informations about how their constructed their private search engine
solso · 2021-03-03
There was no tracking on Cliqz, nor it will be any in Brave. To know more about the underlying tech of Cliqz there are interesting posts at https://0x65.dev, some of them covering how signals are collected, data, but no tracking.
I did work at Cliqz and now I work at Brave. I can tell for a fact, that all data was, is and will be, record-unlinkable. That means that no-one, not me, not the government, not the ad department can reconstruct a session with your activity. Again, there is no tracking, full anonymity, Brave would not do it any other way.
pythux · 2020-06-16
To add to the excellent answer from Gorhill, you might find this read interesting: https://0x65.dev/blog/2019-12-20/not-all-adblockers-are-born-equal.htm…
It is a deep dive into how modern content blockers work and what kind of rules they have to enforce (they are usually much more complex than simple domains, though).
As a side node, I recently ran a small experiment and found out that around 90% or blocking from Easylist/EasyPrivacy/uBO lists can be done based on domain information only. But of course this leaves a lot of corner cases where sites might break or ads might still show, and ultimately more granularity is needed to address these cases.
__ka · 2020-05-01
Here's a list of in-depth blog posts of the Cliqz journey and tech:
https://www.0x65.dev/
netankit · 2019-12-06
[Disclaimer: I work at Cliqz]
You are on point. Recency is a challenging problem in multiple ways for search engines. Not just limited to discovering new content, but also how does one index it? How does one balance out when you have for the same query "very new", "new", "slightly old" and "really old" results during ranking. This involves both news as well as new webpages surfacing on the web.
On top of this, we have to remember that this is a fully autonomous real time system which requires solving some of the most difficult engineering challenges at scale and at the same time being mindful of the latency and quality constraints.
At the end of the day, it's all about the final user experience that we ship. We are very much mindful of the same. We will be publishing more details about Cliqz search, on our blog https://0x65.dev/ in the coming days, so stay tuned.
tschakkaMarc · 2019-12-03
And here are all details how we remove all personal information with a technology we call Human Web: https://0x65.dev/blog/2019-12-03/human-web-collecting-data-in-a-social…
We love to get scrutinized and get feedback on this - we’re very serious about privacy.
yamtown · 2023-04-27
https://0x65.dev/blog/2019-12-05/a-new-search-engine.html
Kudos if you can find their 2023 crawler in logs....
jakub_g · 2022-03-27
This article has insights from folks who actually tried:
https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
TL;DR you need a lot of money and users (catch-22). Also, massive scale and massive noise.
You can also search HN for 'cliqz' for discussions about Cliqz shutting down.
belter · 2021-06-29
Yeap...
"Brave acquires search engine to offer the first private alternative to Google Search and Google Chrome on both mobile and desktop"
https://brave.com/brave-search/
"A New Search Engine" (2019)
https://0x65.dev/blog/2019-12-05/a-new-search-engine.html
"Building a search engine from scratch" (2019 )
https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
toytoyr · 2021-06-23
Sounds like they built their index by collecting Google search queries done by the users of their browser extension. Suddenly Brave's "completely independent index" doesn't sound quite as impressive.
https://www.0x65.dev/blog/2019-12-05/a-new-search-engine.html
merlinscholz · 2020-11-17
I miss Cliqz. It was a new search engine, with its own crawler, almost completely from scratch. It even had a dev blog where they wrote articles on how to build your own search engine: https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
x32n23nr · 2020-10-11
At some point they did flirt with the idea, even invested in a search company, but then they changed their mind [0]
[0] https://www.0x65.dev/blog/2019-12-11/the-pivot-that-excited-mozilla-an…
MH15 · 2020-07-21
This article on their custom search engine was popular on HN a while back https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
ash · 2020-01-02
Some more context: https://0x65.dev/blog/2019-12-04/human-web-proxy-network-hpn.html
sammacbeth_ · 2019-12-18
Note that Mozilla trusted us enough to invest in Cliqz, and we passed their security and privacy audit before they started their experiment with our search in the dropdown. At this time we did also built prototypes together for webextension APIs that would be able to tweak the url bar's behaviour.
Unfortunately, as we mentioned in a previous post [1], Mozilla is not totally independent when it comes to the presentation of search in Firefox due to their existing search deals, so they couldn't continue with the integration. This is why we needed our fork.
Disclaimer: I work for Cliqz
[1]: https://0x65.dev/blog/2019-12-11/the-pivot-that-excited-mozilla-and-go…
wpietri · 2019-12-14
I hadn't heard of it, but apparently Cliqz is a search engine [1] and browser [2] built by a German media company [3].
[1] https://0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-world-needs…
[2] https://en.wikipedia.org/wiki/Cliqz
[3] https://en.wikipedia.org/wiki/Hubert_Burda_Media
ssubu · 2019-12-14
[Disclaimer: work at Cliqz]
We do not crawl the web in the traditional sense, our search was bootstrapped on query logs. It is the very reason we could succeed in building a search engine with minimal resources, in comparison to our competitors.We have written about this in a lot more detail here :
How we collect data : https://www.0x65.dev/blog/2019-12-03/human-web-collecting-data-in-a-so…
How we build the search using this data: https://www.0x65.dev/blog/2019-12-06/building-a-search-engine-from-scr…
Feel free to peruse these posts and ask questions!
gravadlax · 2019-12-08
We do have our own custom word (piece) embeddings that we have trained on <good query, bad query>-pairs. There are a few more details about it in https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
philippclassen · 2019-12-07
(Disclaimer: I work at Cliqz) We had problems with being blocked in the past.
In cases, where we got a chance to explain, they agree that it is a false positive and took us off the block list. At least, that happened so far in all cases that I'm aware of. However, there are so many lists that it is hard to keep track of them. Would be nice if you could provide some information which block list it is, so we can contact them.
The reason why we end on the blocklist is normally a misconception of our data collection system Human Web:
https://0x65.dev/blog/2019-12-03/human-web-collecting-data-in-a-social…
If someone does not want to send Human Web data, the feature can also be disabled through the UI. Same if you browse in a private window; Human Web is automatically disabled there. There is no need to configure blocking rules.
natpat · 2019-12-07
Seems like they have 24/25
> each day until this Christmas, we will explain one core piece.
https://0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-world-needs…
trisch · 2019-12-06
No, it's not enough and had poor coverage outside of USA. We have also answered this question (it appears to be popular) in today's post about technical details of our search https://0x65.dev/blog/2019-12-06/building-a-search-engine-from-scratch…
kkm · 2019-12-06
Hi,
Thank you for bringing this up. Although, this is not relevant in the context of the blog post, we are on that list by mistake. We do NOT collect any personal data in our browser: (more details e.g., here: https://0x65.dev/blog/2019-12-02/is-data-collection-evil.html and https://0x65.dev/blog/2019-12-03/human-web-collecting-data-in-a-social… and we go a long way to make sure not even implicit indentifiers go through. We believe we ended up on that list for a bad Firefox experiment and we will reach out to the maintainers, make our case.
Disclaimer: I work for Cliqz.
lamkhanhsong · 2019-12-03
This is the 3rd post from the advent.
day 1: Why the world needs more search engines https://www.0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-world-n…
day 2: Is data collection evil https://www.0x65.dev/blog/2019-12-02/is-data-collection-evil.html
chrmod · 2019-12-03
Check the post from previous day titled “Is Data Collection Evil?” https://www.0x65.dev/blog/2019-12-02/is-data-collection-evil.html
By now it should be obvious that no privacy oriented company (or in fact any other company) should treat user data as product.
gfawke5 · 2019-12-02
seems to be like day 2 of the advent.
from day 1: The world needs more search engines [1].
[1] https://www.0x65.dev/blog/2019-12-01/the-world-needs-cliqz-the-world-n…