Algolia Blog

In general that sounds more like a classical search engine to me. Here I would check out Algolia (https://www.algolia.com)

We have been using https://www.algolia.com/ completely as a replacement for ES. Pros: - Managed search engine - Great API / Developer experience Cons: - Cloud only makes it hard for local development - Expensive (I guess it depends on the usage)

Wagtail (our open-source Django CMS [1]) can generate a static HTML export of your site to the filesystem, Amazon S3 or Google App Engine, piggy-backing on the excellent django-medusa library [2]. This feature was commissioned by a security company who didn't want to risk an embarrassing exposure to new vulnerabilities in Drupal, Rails, Django etc. Search is an issue for static HTML sites. Swiftype [3] and Algolia [4] look like solid options - has anyone used these in production? [1] https://github.com/torchbox/wagtail [2] https://github.com/mtigas/django-medusa [3] https://swiftype.com/ [4] https://www.algolia.com

Disclaimer: I work at Algolia (https://www.algolia.com/), a hosted search as a service API. While I agree that building a great and relevant search experience with a Lucene-based engine requires lots of extra time and effort to get right, there are other non-TFIDF based solutions that provide a much faster path to great relevance with far less effort (https://blog.algolia.com/inside-the-algolia-engine-part-1-indexing-vs-… and it's possible to have semantic ranking without too much machine learning (https://blog.algolia.com/promote-search-results-query-rules/). Not to discount the value of machine learning - we're finding that for specific usecases ML can be a very valuable way to help surface more pertinent content for individuals based on their profile/preferences etc. (https://blog.algolia.com/personalization-announcement/). This may be along the lines of what you mentioned around "commoditizing" complex traditional search workflows. I'd be curious to hear more about what kind of use-cases you think are trickiest without SS.

Algolia | multiple roles | SF, Paris, NY, Atlanta, London | Onsite only, Full time Hey HN, at Algolia we are facing a major challenge of scaling our team to follow our growth and our series B investment. We are currently looking for multiple dev roles, but looking for other areas such as marketing and sales as well :) We are a hosted search API, that allows anyone to quickly build good search within hours. https://www.algolia.com/ Responsibilities depend on the role, but we normally have teams that are responsible for different areas like the dashboard, infrastructure, our open source libs etc. Technical requirements are as per role. You don't need to be a superstar, but it helps if you want to become really good and enjoy the environment. I'm biased, I work there, but I can say that the past year has been good to me both technically and life wise (I relocated to Paris). My co-workers are extremely nice and knowledgable people who seek to bring the best out of everyone. My colleague wrote an article about his experience - https://medium.com/@liamboogar/celebrating-one-year-at-algolia-bc22c8c… As usual, there are job perks but I'll let those be discussed if you apply -> https://www.algolia.com/careers

Looks really nice!. Feature-request: Would love auto-suggest as I type. one of my favorite example(s) is Algolia's doc search https://www.algolia.com/doc/

Correct, the OS’ OOM reaper will kick-in to try and protect core OS operations. So you don’t want to let it get to that stage - you’d typically want to keep at least 15% free memory for the OS to do its thing. Commercially available RAM today goes all the way to 24TB these days, which should be sufficient for a good number of search use cases. Beyond that you’d have to shard data across multiple clusters. Similarly with Algolia, they use 128GB RAM clusters, and recommend you keep your Algolia index size below 100GB: https://www.algolia.com/doc/guides/sending-and-managing-data/prepare-y…

More resources: documentation: https://www.algolia.com/doc/guides/algolia-ai/recommend/ launch blog: https://www.algolia.com/blog/product/introducing-algolia-recommend-the…

Very interesting use of Google's sparse hash. This library is used throughout the codebase: https://github.com/greg7mdp/sparsepp Also, after a quick look through the codebase, it really resembles how Algolia deals with indexes: https://www.algolia.com/blog/engineering/inside-the-algolia-engine-par… And distributed search: http://highscalability.com/blog/2015/3/9/the-architecture-of-algolias-… Amazing project, hopefully gets a lot of traction.

My team used typesense for a recent project. 16 million records combined with https://www.algolia.com/doc/guides/building-search-ui/what-is-instants… Really fast and worth the investment. It doesn't have all the enterprise features of ES or Solr but for basic search features it's great.

Yep - there are already major players in that space: https://www.algolia.com/

Would love to see index restricted API keys comparable to https://www.algolia.com/doc/api-reference/api-methods/generate-secured…

If you are building a core product, such as Spotify's Discover Weekly. Then no question, you need a dedicated team, and substantial commitment. Maybe even a PhD or two. But as an unabashed Angolia devotee, I think the value prop of InstantSearch is a no brainer. It's worthwhile looking at the product itself, as an almost textbook example of how to package services to enterprise customers. https://www.algolia.com/products/instantsearch/ https://www.algolia.com/enterprise/customers/

They built their own. The engine is baked into nginx modules. https://blog.algolia.com/?filter=engineering

Curious to know if you considered https://www.algolia.com/ as well?

Algolia had a great 8-part series of 'under the hood' blog posts: Part 1: https://blog.algolia.com/inside-the-algolia-engine-part-1-indexing-vs-… No affiliation, just thought they were really interesting.

Thanks for the links, Mark! (Disclaimer, I work for Algolia.) As Mark mentions on his summary page, the best place for that kind of information is our CTO's "Inside the Engine" series (8 parts). https://blog.algolia.com/inside-the-algolia-engine-part-1-indexing-vs-…

As Daniel Parker (acloud.guru) mentioned in the article comments, they went with Algolia [1] despite being AWS experts. He cited the uncertainty and complexity around AWS ES as the problem. [1] https://www.algolia.com/

This is great for something like a public website. What about a situation where I want some of my users to only be authorized to view some of my data based on a set of rules I set in my backend application. Can Agolia accomplish this? EDIT: Looks like they do support this.. https://www.algolia.com/doc/api-client/ruby/api-keys/#generate-key However, I think for a small project it will probably be extra work to keep these keys synchronized vs just doing a postgres search for my project

There's Algolia https://www.algolia.com/ This is what HN uses for search.

Companies like Algolia which provide a site specific search engine has been doing really well especially with the speed and relevancy where Google currently is not concentrating on. https://www.algolia.com/

Not sure it works across multiple properties but Algolia https://www.algolia.com/ is good. We use it for https://docs.gitlab.com/

I've used Algolia [1] and Swiftype [2] in the past for search on static websites, but recently experimented with lunr.js [3] for our company playbook (built with GitHub pages): http://playbook.wiredcraft.com/ I still want to separate the index in its own JSON file, but so far, so good. Search is fast and index is rebuilt automatically. [1]: https://www.algolia.com [2]: https://swiftype.com/ [3]: http://lunrjs.com

Algolia (YC W14) | San Francisco | Paris | Full-Time On-Site Frontend, Full-Stack, iOS Engineers; Site Reliability Engineer; Customer Solutions Engineer (Paris) Customer Solutions Engineer (SF) Algolia is "search at the speed of thought." A hosted search service, that hits the sweet spot between "wrestle for months with Elastic" and "just throw some subpar search up." We power the search for sites and apps like Genius, Periscope, and many, many others. Relocation and visa sponsorship (Paris) can potentially be offered for the right person. The hiring process differs between teams, but involves a small number (around 2) of video calls, followed by a take-home test, and a day on-site. More information and can apply here: https://www.algolia.com/careers. Also feel free to email jen (for SF) or marion (for Paris) @algolia.com.

I would recommend to read this post to have an idea: https://blog.algolia.com/inside-the-algolia-engine-part-3-query-proces… Textual relevance is a very complex domain and the Postgres's built in text search features is a simple keyword matching engine compared to Algolia engine that contains a lot of alternative matching. The mesure of textual relevance is also very different of what you have in the Postgres text search. At the end, this is not only about speed but mainly about relevance

If what you need is just search, you could try a SaaS, such as Algolia (https://www.algolia.com/)

Nice! Funny is that Algolia released something very similar some time ago: https://community.algolia.com/instantsearch.js/ It seems searckit took great inspiration from it: - component naming (search box, refinement list, hierarchical menu) - documentation organisation: + http://searchkit.co/ is using an imdb demo just as https://www.algolia.com/ homepage + https://ssetem.gitbooks.io/searchkit/content/ vs https://community.algolia.com/instantsearch.js/documentation/ (see "components" menu on the left) - react based (but only works if you have react though) - and more parts maybe @Author could you give some more details on the parts of instantsearch.js that were inspiring to your project? Thanks

Natural language search and answer generation _are_ completely separate. Search is often (but not exclusively) performed using cosine similarity over semantic vectors. Such vectors are produced using embedding models, which represent the meaning of the document via an arbitrary length vector called an embedding, and 768 is a common vector size for this. You calculate the embedding for all documents in your database ahead of time (during insertion), and you calculate the embedding for the user's query, then search for documents closest to the query using a similarity metric of choice such as cosine. Nothing prevents you from serving documents found this way directly, instead of using them to generate answers. Part of the Google search pipeline involves something like this. Many full-text search products also do this (Algolia is such an example https://www.algolia.com/blog/ai/what-is-vector-search/) LLMs and generation are used to synthesize the answer and tie it back into the question using a layer of soft judgment based on the LLMs prior knowledge. This does work out great in some contexts, less so in others, as you pointed out. But these components aren't coupled in any way.

Hi naiv, Nick Vlku, VP of Growth at Algolia, here. Wanted to dive in a bit more on Ben's answer below. To be honest, I was surprised at this limitation too when I first starting using Algolia. Upon some research, it turns out this is really complicated and Algolia errs on doing what yields the best experience for end users (even if it might require a little bit more work). Basically, we are trying to avoid the phenomenon when you apply a filter search (like "sort by price") and returning something that has nothing to do with your query or intent. For example, a customer searching for an "Eames chair" and then sorting by lowest price would return things like $2 postcards of Eames chairs. So, in order to capture the right kind of intent for that sort (and also allow our customers to further customize results), we require a separate replica of the data. (Maybe if somebody sorts for cheapest Eames chair, you would bump up a similar cheaper chair by another company instead of a postcard?) The good news is, for most of these cases, there is no additional cost. You basically can create a virtual replica which points to your same records with different ranking strategies. So no doubling up the records! See here for info on this: https://www.algolia.com/doc/guides/managing-results/refine-results/sor… It's a balancing act between "magical sorting just working" and "doing what looks right for the customer". We landed on this…

Search is a fascinating topic because it's such a fundamental problem and every search engine is based around the same extremely simple data structure (Posting list/inverted index). Despite that, search isn't easy and every search engine seems to be quite unique. It also seems to get exponentially harder with scale. You can write your own search engine that will perform very well on a surprisingly large amount of data, even doing naive full-text search. A search tool I came across a while back is a great example of something at that scale: https://pagefind.app/. For anyone who doesn't know anything about search I highly recommend reading this (It's mentioned in the blog post as well): https://swtch.com/~rsc/regexp/regexp4.html. Algolia also has a series of blog posts describing how their search engine works: https://www.algolia.com/blog/engineering/inside-the-algolia-engine-par… --- It's interesting that GitHub seems to have quite a few shards. Algolia basically has a monolithic architecture with 3 different hosts which replicate data and they embed their search engine in Nginx: "Our search engine is a C++ module which is directly embedded inside Nginx. So when the query enters Nginx, we directly run it through the search engine and send it back to the client." I'm guessing GitHub probably doesn't store repos in a custom binary format like Algolia does though: "Each index is a binary file in our own format. We put the information in a specific…