Pinecone Blog

Pinecone | Software Engineer | New York, NY (NYC) | Tel Aviv | Full-time | https://grnh.se/f13fe7453us Pinecone (https://www.pinecone.io/) is building a vector database for machine learning applications. We're a distributed team with offices in New York and Tel Aviv. At Pinecone you will work with world-class scientists and engineers who have built large scale ML applications and platforms (including Amazon SageMaker) at leading companies and cloud providers. We are passionate about building great solutions by pushing the boundaries of science and technology. Apply here: https://grnh.se/f13fe7453us

Your ideas seem really cool. I'm sad that you chose a name so close to another great startup that does vector databases and search. https://www.pinecone.io/ You're headed into a place where some customers may confuse you.

> 1) search engines will be replaced with monolithic large language models This is already well under way. It's called vector search[1]. Google, Bing, Facebook, Spotify, Amazon, etc etc already use this to power their search and recommender systems. There are even a bunch of companies popping up (I work for one[2]) that let everyone else get in on the fun. Check out this video with the creator of SBERT / SentenceTransformer explaining how vector search is used in combination with language models to power semantic search: https://youtu.be/7RF03_WQJpQ [1] https://www.pinecone.io/learn/what-is-similarity-search/ [2] https://www.pinecone.io

Pinecone | Software Engineer: Full-Stack Frontend | New York, NY (NYC) & San Francisco Bay Area, CA (SF) & Tel Aviv, Israel | Full-time | https://grnh.se/a6f4b3203us Pinecone (https://www.pinecone.io/) is building similarity search at scale, providing vector indexing as a service. We're a distributed team with offices in New York, San Mateo, and Tel Aviv. At Pinecone you will work with world-class scientists and engineers who have built large scale ML applications and platforms (including Amazon SageMaker) at leading companies and cloud providers. We are passionate about building great solutions by pushing the boundaries of science and technology. Apply here: https://grnh.se/a6f4b3203us

I can't wait to see the vector DBs we're going to have 5 years from now when everyone needs to serve their embeddings. It's clearly early but frothy right now. Also check out this similar co I ran into: https://www.pinecone.io/ (the CEO, EL, has some classic sketches and feature hashing papers).

I'd say also SMO (social media optimization) Any "link juice" on Google may or may not be real, but traffic is real. I'd point to https://www.pinecone.io/ as a company that was highly effective at building awareness of their product long before vector databases became hot with a blog. Their blog now is just another navel-gazing, self-centric 'enterprise' blog, which is sad.

If you are just getting into this tech, like me, then this article leads down the rabbit hole of different index types supported in different vector stores. Most easy-to-use vector stores only use HNSW indices, which is "good enough". A few offer different types of indices. Milvus[1] and FAISS[2] are a couple that support IVF indices. It's great to see another lightweight tool that does this with GPU acceleration. 1. https://objectbox.io/vector-database/ 2. https://www.pinecone.io/learn/series/faiss/vector-indexes/

Depends what you mean by taking all your articles. If it's scraping no, but if you provide text content and urls as key/values pairs yes. SageMaker and VertexAI are the AI services of AWS and GCP respectively, and they both offer embedding generation and vector databases (the two key pieces necessary for embedding search). There are a bunch of smaller companies offering vector search as a service too, example pinecone to name just one: https://www.pinecone.io/

Not to be confused with https://www.pinecone.io/ the vector database, which is also trending.

Might want to think about the name. Pinecone is a relatively recent startup specializing in vector databases. https://www.pinecone.io/

This is really crazy. Are we running out of words to name companies: * https://pinecone.io/ - Long-term Memory for AI * https://pynecone.io/ - Frontend. Backend. Hosting. Pure Python. Just one letter difference 'i' -> 'y'. In other context I would suspect phishing.

I was playing around doing the same thing today but using Pinecone [1] as a vector store. Some other folks were experimenting with building key-value stores at the Scale AI hackathon this weekend [2] [1] https://www.pinecone.io/ [2] https://twitter.com/sjwhitmore/status/1617318051455840258

It makes a request to the Pinecone API to search through vector embeddings. [1][2] [1] https://github.com/chrislee973/bible-semantic-search/blob/ab5e532228c2… [2] https://www.pinecone.io/

How about a vector oriented 'database' instead? Pinecone(https://www.pinecone.io/) does both exact and approx search and it's fully managed so you don't have to worry about reliability,availability etc. PS: I work there

One from the Qdrant team here. We genuinely recommend starting with whatever you already have in your stack to prototype or produce applications with vector search. From the beginning, one should probably not start a new project with a complex micro-service architecture. The same as you can start with a full-text search using just Postgres or whatever you use as the main DB. However, as the requirements are getting higher and more complex, you probably should switch to a dedicated search engine to avoid the monolithic persistence anti-pattern. https://learn.microsoft.com/en-us/azure/architecture/antipatterns/mono… Regarding the dedicated solution, the difference is always in the details. That detail might significantly impact if you are looking for precision or working with large data. First of all, filtering the vector similarity search is quite tricky, and it does not work with pre and post-filtering because the graph's connectivity gets lost. That is why we introduced filterable HNSW at the very beginning. https://qdrant.tech/articles/filtrable-hnsw/ Here is another quite good explanation by James Briggs: https://www.pinecone.io/learn/vector-search-filtering/ Going further, it is not only about being able to search. It is also about being able to scale. Optionally, use disk for cold data as more affordable storage and keep hot data in expensive RAM. Or using one of the built-in compression functionalities. The recently introduced Binary Quantization…

Great post! We've got some additional resources for folks looking to better understand Retrieval Augmented Generation (RAG) and even see it in action - in this example we demonstrate a potentially very dangerous hallucination (that has to do with driving) and how to fix it using RAG: https://www.pinecone.io/learn/retrieval-augmented-generation/ If you're curious to actually try out the difference between an LLM without domain-specific context and an LLM that is using RAG, you can try our live demo here: https://pinecone-vercel-starter.vercel.app/ And if you'd like to fork and make your own tweaks to the above demo ^ chatbot, in order to, for example, swap in your own company logo and extend it for your purposes, you can find our Vercel template here: https://github.com/pinecone-io/pinecone-vercel-starter In our opinion, RAG is indeed an effective technique partly because you don't need to be a machine learning expert in order to implement it in your Generative AI applications.

I honestly hope they use it to improve their documentation. I consider myself and pretty adept developer but without much background in AI and was looking for a solution for building out a recommendation engine and ended up at Pinecone. Maybe I'm not the target audience but after spending some time poking around I couldn't honestly couldn't even figure out how to use it. Even a simple Goole Search for "What is a Vector Database" ends up with this page. https://www.pinecone.io/learn/vector-database/#what-is-a-vector-databa… Pinecone is a vector database that makes it easy for developers to add vector-search features to their applications Um okay... what's "vector-search"? For that sake what the eff is a "Vector" to begin with? Finally after getting about a third down the page we start defining what a vector is.... Maybe I'm not their target audience but I ended up poking around for about an hour or two before just throwing up my hands and thinking it wasn't for me. Ended up just sticking with Algolia, since we had them in place for Search anyway...

Pinecone is a vector database: https://www.pinecone.io/

Check out https://www.pinecone.io/

Being from 2017 the article misses some of the coolest advances in semantic search, which is now pretty easy and lets you search in (almost) the same way they would ask a shop assistant when looking for something specific - "do you know where the thing with the cool circles and pointy bits is?" (maybe being a little more specific...) Google do this and they're very good at it, but a lot of companies need their own search capabilities - think about those internal help pages. They usually seem super outdated compared to the semantic search capabilities of Google. In the end there's only a few components to it, you use some NLP model to create what are called 'dense vectors'. Then you put all these dense vectors into an 'index' which is optimized for fast search (that comes under the umbrella of ANN search). Then given a new query you just compare that to the items in the index and return the most similar results. I covered the search part of it (https://www.pinecone.io/learn/) with Pinecone, who provide managed-search - although here we mainly focus on Faiss (a great engine from Facebook AI). However, we're also looking to create some content covering the first half too, which is 'how to build dense vectors' using models like BERT, we have one post so far on that (https://www.pinecone.io/learn/dense-vector-embeddings-nlp/)

Hi All, it's true that LSH (locality sensitive hashing) and DR (dimension reduction) are very related. They both derive their correctness from the same strong concentration phenomenon. But, they are used/designed for different use cases. DR is mostly concerned with preserving distances such that c * D( f(x), f(y) ) < d(x,y) < C*D( f(x), f(y) ) Here, f(x) is the mapping of x to a lower dimensional space (or any other object, really). d(-,-) and D(-,-) are the original distance measure and new distance measure respectively. The smaller the ratio C/c the better. Clearly, the computational overhead of applying f and D are also important. LSH, on the other hand, is mostly concerned with probabilistically filtering and searching for nearby points efficiently. Here are some (old) class notes from a course I gave 2013https://edoliberty.github.io//classnotes/datamining2013/pdfs/12_approx… While LSH is mathematically beautiful, it is no longer considered the gold standard for ANN. There are many open source algorithms that outperform LSH on most datasets. (I founded a vector search company: https://www.pinecone.io)

Paper can be found here: https://www.pinecone.io/research/unveiling-dime-reproducibility-scalab…

This post has some more technical info: https://www.pinecone.io/blog/integrated-inference/ Makes a lot of sense to me to combine embedding, retrieval and reranking — I can imagine this being a way that they can differentiate themselves from the popular databases that have added support for vector search

There's more technical detail here: https://www.pinecone.io/blog/integrated-inference/ > Is this announcement that pinecone is adding their own? TLDR: they trained their own embeddings model and rely on Cohere for ranking. Pinecone (the database) uses this model automatically to generate and store embeddings. > I assumed that a specific flavour of LLM was needed, an “embedding model” to generate the vectors. You're mostly right, with one caveat: embeddings models aren't really LLMs in that they're not very large: they just map semantic meaning to numerical space. > Is it better or worse than the models here: https://ollama.com/search?c=embedding For example? This is the golden question. As far as I know, there is no appropriate benchmarking/eval data about this. I think the real value is the first-class integration between their model and their service.

To be fair, the pricing is still available in this case: https://www.pinecone.io/pricing/ (though the "Unlimited XXX" with the actual price below in gray might be considered misleading)

Thanks for the answer. Wikipedia has a lot of tables so I was wondering if content-aware sentence chunking would be good enough for Wikipedia. https://www.pinecone.io/learn/chunking-strategies/

I don't have plans for compressed vector storage. There is support for int8/binary quantization, which can reduce the size of stored vectors drastically, but impacts performance quite a lot. I'd like to support something like product quantization[0] in the future, though! No plans for using usearch or another vector search library, either. I want to keep dependencies low to make compilingeasy, and I want full control for how vectors are stored on-disk and in-memory. [0] https://www.pinecone.io/learn/series/faiss/product-quantization/

seeing comments about using pgvector... at pinecone, we spent some time understanding it's limitations and pain points. pinecone eliminates these pain points entirely and makes things simple at any scale. check it out: https://www.pinecone.io/blog/pinecone-vs-pgvector/

Vector databases don't contain ML models. There is nothing that is learned. Here is a typical algorithm: https://www.pinecone.io/learn/series/faiss/hnsw/ It is all about performance; latency and recall.

Throwing one more - https://www.pinecone.io/learn/vector-database/