ParadeDB Blog

I agree with the author - introducing a vector database often isn't worth the extra complexity. Personally, I can vouch for ParadeDB: https://www.paradedb.com/ It adds extra extensions to PostgreSQL which enable vector indexing, full text search and BM25. Works great and developers are helpful! The major difference is that you must generate the embeddings by yourself, but I consider it an upside - to each their own :)

ParadeDB (YC S23, https://paradedb.com) | Database Internals Engineers | Remote or hybrid in San Francisco | Full-time ParadeDB is a Postgres extension that delivers state-of-the-art search in Postgres. Our customers, including Alibaba Cloud, BILT Rewards, Modern Treasury (YC S18) and many others, rely on us to deliver them the performance and feature set of dedicated vector databases/Elasticsearch natively in Postgres, to simplify their stack and day-to-day operations. If you like systems, this might be for you. We're a team of 11, Series A company based in SF but distributed across North America. We're open source at: https://github.com/paradedb/paradedb. Our technical blog: https://www.paradedb.com/blog More info on roles: https://paradedb.notion.site/

I think you forgot to add the links Anyway tantivy is great! I love pg_search https://www.paradedb.com/blog/introducing_search (which appears to be built by another company, but on top of tantivy, which is a great feature of open source) Now, I am worried about development being stalled after this acquisition. How does further developing tantivy in the open helps Datadog's bottom line?

> Recently, the most interesting rift in the Postgres vs OLAP space is [Hydra](https://www.hydra.so), an open-source, column-oriented distribution of Postgres that was very recently launched (after our migration to ClickHouse). Had Hydra been available during our decision-making time period, we might’ve made a different choice. There will likely be a good OLAP solution (possibly implemented as an extension) in Postgres in the next year or so. There are a few companies are working on it (Hydra, Parade[0], tembo etc.). 0 - https://www.paradedb.com/

> 0 - https://www.paradedb.com/ this looks like repackaging of datafusion as PG extension?..

That's true for the kernel, How about extensions such as ParadeDB BM25 https://www.paradedb.com/ + PGroonga https://pgroonga.github.io/ + PG Bigm https://github.com/pgbigm/pg_bigm ?

Blog post author and one of the pg_bm25 contributors here. Super excited to see the interest in pg_bm25! pg_bm25 is our first step in building an Elasticsearch alternative on Postgres. We built it as a result of working on hybrid search in Postgres and becoming frustrated with Postgres' sparse feature set when it comes to full text search. To address a few of the discussion points, today pg_bm25 can be installed on self-hosted Postgres instances. Managed Postgres providers like RDS are pretty restrictive when it comes to the Postgres extension ecosystem, which is why we're currently working on a managed Postgres database called ParadeDB which comes with pg_bm25 preinstalled. It'll be available in private beta next week and there's a waitlist on our website (https://www.paradedb.com/).

I'm trapped on Azure at work and we're constantly waiting for Azure pg to catch up with modernity. For example, you cant use this: https://www.paradedb.com/blog/hybrid-search-in-postgresql-the-missing-… Also for example, you dont get ultra-wide high dimensionality vectors. It is nice they are open sourcing pg_durable, but how about adopting table stakes I'd get with AWS?

Funny argument on the query languages in hindsight, since the latest release (https://www.paradedb.com/blog/paradedb-0-20-0 but that was after this blog) just completely changed the API. To be seen how many different API versions you get if you make it to 15 years ;) PS: I've worked at Elastic for a long time, so it is fun to see the arguments for a young product.

It's been shown in Postgres https://www.paradedb.com/blog/lsm_trees_in_postgres

It used to use DataFusion. https://www.paradedb.com/blog/iceberg_lakehouse

Thanks for the questions! After reading I don’t get how locks held in memory affect WAL shipping. WAL reader reads it in a single thread, updates in-memory data structures periodically dumping them on disk. Perhaps you want to read one big instruction from WAL and apply it to many buffers using multiple threads? We currently use an un-modified/generic WAL entry, and don't implement our own replay. That means we don't control the order of locks acquired/released during replay: and the default is to acquire exactly one lock to update a buffer. But as far as I know, even with a custom WAL entry implementation, the maximum in one entry would still be ~8k, which might not be sufficient for a multi-block atomic operation. So the data structure needs to support block-at-a-time atomic updates. I guess your implementation generates a lot of dead tuples during compaction. You clearly fighting PG here. Could a custom storage engine be a better option? `pg_search`'s LSM tree is effectively a custom storage engine, but it is an index (Index Access Method and Custom Scan) rather than a table. See more on it here: https://www.paradedb.com/blog/block_storage_part_one LSM compaction does not generate any dead tuples on its own, as what is dead is controlled by what is "dead" in the heap/table due to deletes/updates. Instead, the LSM is cycling blocks into and out of a custom free space map (that we implemented to reduce WAL traffic).