OpenSource Connections Blog

BTW, I highly recommend the Relevance Slack as a good space to discuss this and search-related topics generally. It's run by OpenSource Connections, but it's free and anyone can join, even folks who compete with them. It's the best community I have found for search developers. https://opensourceconnections.com/blog/2021/07/06/building-the-search-…

There is a Slack group related to search at http://www.opensourceconnections.com/slack Great community. I would love to join something from the start, but now is not the right time.

Great blog post on this subject: https://opensourceconnections.com/blog/2019/02/28/stop-worrying-solr-e…

I think a lot off preference towards humans is an issue of trust and a measure of your relationship with that person. Usually we get recommendations from friends and give them more thought and weight than some random ad we scroll by on Facebook. https://opensourceconnections.com/blog/2016/08/21/recommendations-syst…

I think this is a great comparison between Solr and Elasticsearch from a search relevance perspective: http://opensourceconnections.com/blog/2016/01/22/solr-vs-elasticsearch…

People have been predicting the death of keyword search for at least as long as I've been working in the field (23 years and counting!). We've seen outrageous marketing claims, buzzwords like "concept search", "insight engines", many new companies promising a step change in search quality, some of which are still in business but many who shone briefly then vanished. The concept of an easy to use, fire-and-forget, scalable search engine that gives you great relevance out of the box isn't new. Yet the bag-of-words model remains the standard across the sector, it's well understood with many powerful and scalable open & closed source implementation options. To make search work in practice however is hard. It's as much about process and people as it is about technology: many companies aren't even measuring search quality, recording search issues correctly or have an active search team (bigger than one poor overworked search person). No matter how clever the tech, these problems aren't going away: they're compounded by bad source data quality, misunderstandings of user search intent and bad search UX. Martin White, author of many books on search, describes search as a 'wicked problem'. Getting all these parts working in harmony so you can truly own your search is what we do here at OSC and it takes time, investment and commitment. I think Vectara is very interesting and the people involved have impressive track records (there's also some other great engines like Vespa, Pinecone,…

https://opensourceconnections.com/blog/2023/02/27/hybrid-vigor-winning… https://jina.ai/news/hype-and-hybrids-multimodal-search-means-more-tha… are two I've been looking at Basically, get the BM25 results and normalize them to be between 0 and 1, then take the (potentially weighted) average of them and the cosine similarity results (already between 0 and 1) to get the final ranking.

Does anyone know of the"sqlite connector" mentioned in this post ? https://opensourceconnections.com/blog/2013/05/06/does-foundationdb-be… It would be really cool to find it, if it's still out there.

Don’t forget JMESPath, as built into AWS CLI: http://jmespath.org/ With language libraries or as jp CLI: http://jmespath.org/libraries.html Example AWS CLI tricks: https://gist.github.com/magnetikonline/6a382a4c4412bbb68e33e137b9a7416… https://opensourceconnections.com/blog/2015/07/27/advanced-aws-cli-jme…

tl;dr is that those are different approaches to weighting documents in order to return the most relevant ones for a query. For an intro to the problem space, see https://opensourceconnections.com/blog/2014/06/10/what-is-search-relev… If you want a lot more detail, check out the book Relevant Search. https://www.manning.com/books/relevant-search

Related, Lucene (and thus Elasticsearch) moved to BM25, which can be understood relative to the traditional TF*IDF here: http://opensourceconnections.com/blog/2015/10/16/bm25-the-next-generat…

HyperLogLog is an algorithm that allows you to find an estimate of the distinct elements of a dataset with very little memory. Here's a simple explanation: http://opensourceconnections.com/blog/2015/02/04/its-log-its-log-its-b… HyperLogSandwich solves a similar problem. From the README: > Unlike the HyperLogLog, which estimates the cardinality of unique items in a set, and unlike a CountMinSketch, which estimates the frequency of a specific item, the HyperLogSandwich estimates the number of unique items that have occurred for any frequency. Real world examples include: - How many people viewed my Tweets 4+ times this month? - How many users watched this video 2 times this week? - How many users visited my website 3 times during an arbitrary time range? - How many log exceptions happened 10+ times last week?

At my company, we've been beating our heads to the wall[0] in getting multi-term synonyms to work correctly in SOLR. e.g. fruit extractor => fruit juicer, citrus juicer Does anyone experienced enough have a clue if SOLR 5 can help with that? [0] - http://opensourceconnections.com/blog/2013/10/27/why-is-multi-term-syn…

The talk "Database History from Codd to Brewer and Beyond" by Doug Turnbull while not representative of entire database history. Gives pretty good motivations for why things are the way they are. Talk link: https://www.youtube.com/watch?v=HiXoEQzf-Zg Blog link: http://opensourceconnections.com/blog/2013/12/11/codds-relational-visi…

Yes! This: http://www.opensourceconnections.com/2013/12/11/codds-relational-visio…

In 2015, they evaluated Apache Solr and Elasticsearch and decided to build their own (Firefly). They said, other solutions did not scale. So, instead of contributing to scaling (like Apple and Bloomberg and Cloudera did), they went the other way. Now, they seem to be doing it again (at least they are using Tika). In a meanwhile, Solr implemented most of the features they are describing in their architecture document. Specifically: 1)General scaling: https://lucene.apache.org/solr/guide/7_5/introduction-to-scaling-and-d… (using ZooKeeper and SolrCloud) 2) Search Ranking and click-data training: https://lucene.apache.org/solr/guide/7_5/learning-to-rank.html (Contributed by Bloomberg) 3) Offline builds with substitution into production: https://lucene.apache.org/solr/guide/7_5/collections-api.html#createal… 4) Near-Real-Time: https://lucene.apache.org/solr/guide/7_5/near-real-time-searching.html 5) Sharding specifically: https://lucene.apache.org/solr/guide/7_5/shards-and-indexing-data-in-s… 6) Extraction pipeline, they are doing all together. We have: a) pre-Solr extraction (usually done in a stand-alone client, though we do include Tika and DataImportHandler for quick start), b) in-Solr pre-schema processing with Update Request Processors https://lucene.apache.org/solr/guide/7_5/update-request-processors.htm… c) Actual per-field text processing pipelines, separate both for index and query (they call query part later "query…

(I'm working at Algolia) The answer I'm used to provide when people ask me this question is the one our CTO posted on Quora a few years ago: https://www.quora.com/How-does-Elasticsearch-relate-and-or-compare-to-… tldr; Algolia's focus is on consumer-grade & user-facing search (searchbar like you would have on Amazon, IMDB, Twitch, Medium.com, quora, ...) while ES's sweet spot is on indexing billions of documents (or lines of logs) and provide rich dashboarding & aggregation capabilities (like a monitoring dashboard with their ELK stack). Recently, ES released a beta of "App Search" which seems to target more and more the user-facing search bars. Here are also a few resources to help get you an idea: * from Doug Turnbull, author of "Relevant Search" & ES expert: https://opensourceconnections.com/blog/2016/06/01/thoughts-on-algolia/ * from an Algolia & ES user: https://medium.com/@matayoshi.mariano/elasticsearch-vs-algolia-96364f5… * from the Algolia FAQ: https://www.algolia.com/doc/faq/why/what-makes-algolia-different-than-… * from the Algolia blog: https://blog.algolia.com/?s=comparing+algolia+elasticsearch