DoltHub Blog

Hi there, I wrote a blog about using Cypress.io to test our web application (https://www.dolthub.com). We recently started supporting tests for login and private paths. The blog compares two login options we considered for best testing the workflows on our website. You can check out our suite of Cypress tests here: https://github.com/dolthub/dolthub-cypress

I'm the CEO of a company that's focused on version control for structured data ie. SQL tables and JSON documents. We share the "version control for everything" vision and think an agentic world needs it. https://www.dolthub.com/

I have tried this before: https://www.dolthub.com/ It was a lot of work and had poor performance with a lot of complications. I am not using it in my latest projects as a result.

The database actually is named "Dolt." It's their own take on "git." https://www.dolthub.com/

@ThePrimeTimeagen trigger ... BRazil mentioned ... New Go Billion Row Challenge w/ Great Optimizations | Prime Reacts - https://www.youtube.com/watch?v=SZ1PDS7iRU8 - https://r2p.dev/b/2024-03-18-1brc-go/ Spoiler: SwissMap: A smaller, faster Golang Hash Table - https://www.dolthub.com/blog/2023-03-28-swiss-map/ - https://github.com/dolthub/swiss Want to see Paul Allen's implementation in Rust, Zig, Mojo ... and even Bun.sh, Deno.com

Isn’t Dolt one of the initiatives for this? https://www.dolthub.com/ Also not so long time ago, I saw a post on HN sharing a platform that does just this but for hospital data. can’t find tho

A bit surprised to see there is no mention of Liquibase, Flyway or others. Not to mention there is also "https://www.dolthub.com/". This was just for git.

https://www.dolthub.com/ So I get what you're saying, but at the same time that means your business data can't ever include formulas. I've worked in areas where the formulas were business data (i.e. modeling). You could hard-code all your models/correlations/aggregation functions. But then it becomes much harder to make UIs that do things like "find all formulas that involve this thermometer reading" or allow a technician to make temporary edits without risking them screwing up your entire application.

If anyone from DoltHub team is reading this, please consider fixing [0] the RSS/Atom feed link [1]. I love your blog posts, but I am not able to subscribe to them in my feed reader. Your posts on SwissTable and Prolly Trees are my fav ones. [0] - https://validator.w3.org/feed/check.cgi?url=https%3A%2F%2Fwww.dolthub.… [1] - https://www.dolthub.com/blog/rss.xml

Our implementation of the Merkle DAG data structure is optimized for tabular data since Dolt is a SQL database, but a lot of it generalizes to other kinds of data. For example, we put a lot of work into making sure that chunk sizes had a predictable distribution, since very large chunks kill query performance: https://www.dolthub.com/blog/2022-06-27-prolly-chunker/ We also have a lot of work to do with preserving structural sharing across schema changes, which we're not great at yet. https://www.dolthub.com/blog/2024-01-19-structural-sharing-with-schema…

You can already use Dolt in embedded mode. Check this blog post from them: https://www.dolthub.com/blog/2022-07-25-embedded/. I use it like that and it works great.

I just learned of Neon a few weeks ago. From the looks of it, Neon supports branching, but it doesn't support merging. Xata supports both branching and merging, however it only applies to the schema. Dolt (and eventually DoltgreSQL) handles everything. Branching, merging, diffing, cherry-picking, commits, and more. Working with both the schema and data. On top of that, we also have DoltHub (https://www.dolthub.com/) that's analogous to GitHub, and DoltLab (https://www.dolthub.com/#doltlab) that's analogous to GitLab. We are targeting the entire ecosystem from the bottom up.

It would be nice if we finally could have a database that we could use git on to track changes. Dolt looks very promising in this area but only works with MySQL, would be nice to have a SQLite solution. https://www.dolthub.com/

> I'd love to see what this map looks like filtered for specific services. Sure. I did something like that with our insurance data. Did a quick folium visualization. You can start from here: https://www.dolthub.com/repositories/dolthub/hospital-prices-allpayers… A limitation of our dataset is it excludes percentage-based contracts. So those won't show up. Anyways, if you have the hospital NPI you want, you can look up all of its dollar-amount negotiated rates in this table. You can link that to the `hospital` table, geocode however you want, and make a map of the different rates. Note, however, that the insurance data published here is kind of a mess and, I think, contains many many mistakes (originating in the source data.) BTW, I'm not sure if this insurance-derived hospital data exists anywhere else. I believe it doesn't.

You could post it on DoltHub… that would give you a web interface with SQL support, version tracking and a way to receive community edits. https://www.dolthub.com/

People who find this interesting might also like DoltHub: "Dolt marries two familiar concepts, Git and MySQL. The first and only SQL database that supports clone, branch, and merge." https://www.dolthub.com/

You should check out https://www.dolthub.com/ then. They are working on something very similar.

This is really cool! I would have liked to have incorporated this into my vaccine appointment slot finder tool a few months ago. I like using git commits for change tracking too. Seems not dissimilar (though not identical) to what they're doing at Dolt (https://www.dolthub.com/).

There's a trend of versioned databases that could address these issues. I've seen at least Dolt [1] for relational: "Dolt is a data format and a database. The Dolt database provides a command line interface and a MySQL compatible Server for reading and writing data." ... and TerminusDB [2] for graphs: "TerminusDB is an open-source knowledge graph database that provides reliable, private & efficient revision control & collaboration." I don't know if Dolt supports stored procedures, but in any case versioning a database would be an amazingly powerful and convenient thing, if it can be done without sacrificing too much DB performance. Worth investigating and has the potential of changing the way we write apps. -- 1: https://www.dolthub.com/docs/tutorials/installation/ 2: https://terminusdb.com/

Is something like dolthub [0] what you have in mind? [0] https://www.dolthub.com/

Author here. I'm a software engineer at Liquidata (https://liquidata.co), the company behind Dolt (https://github.com/liquidata-inc/dolt) and DoltHub (https://dolthub.com). We're building the future of data distribution and collaboration; come check out what we're up to! I wrote this article as part of a series on the technology behind DoltHub. In this installment, I explore how GraphQL rescued us from a nasty pit of sadness we got stuck in calling our gRPC API directly from our front end. Hope folks find it interesting. Happy to answer any questions!

"Dolt" https://www.dolthub.com "Gun" https://gun.js.org (Bonus points for gun-shaped logo.)

https://dolthub.com is the cool kid right now. There is pacaderm, git lfs, IPFS. Really what we need is version control for data, it's not just an ML data problem. It's a little different though, because you would like to move computation to data, rather than the other way around

https://www.dolthub.com/ ? Never used it myself, but it's one of the many "git for data" projects I've seen go past. I'm interested that none of them meet the authors need.

When I read the headlines, I thought it was these guys: https://www.dolthub.com

There are purpose-built databases for this purpose such as Dolt (https://www.dolthub.com/), itself purportedly derived from noms (https://github.com/attic-labs/noms). Because it's baked into the database, you can do complex things like diffs and merges, but it requires that you run everything on that RDBMS vs. something standard. Generic diffing from two arbitrary data states of any database is pretty challenging, even if you have all the transaction history.

That assumes your database cells aren't themselves revision-controlled (e.g. Dolt: https://www.dolthub.com/) But also, this is the same argument people generally give against using DB stored procedures. And, despite using sprocs often myself, I generally agree with them — the tooling around sprocs really sucks. Sprocs are "DB objects as state" (like rows in a table are state); but they could be a lot more than that. My secret dream, as an infrastructure engineer and DBA, is that, on boot, each version of each application that uses an RDBMS could "register" or "zero-install" a client schema with said RDBMS — i.e. a namespace of persistent virtual DB objects, unique to that build of that application — but where the underlying physical DB objects are not necessarily unique, instead being shared as needed between builds and even between applications. (Think: what would happen if you put the Plan9 designers in charge of the evolution of the SQL standard.) Such a "client schema" would specify, among other things, the full source code of any sprocs the application expects to be able to call. These would be able to be internally deduplicated — e.g. registered by content-hash into an sproc store on the DB (compare/contrast: Redis EVALSHA), and then "symbolically linked" to a particular name in the client schema. Ideally, in applications that use client schemas, every SQL query that's a static literal at app build time, would be compiled into a boot-time registration of, and…

> Somebody always wants to undelete something > or examine it to see why it was deleted > or see who changed something, or blah blah blah People for whom this resonates should look into Dolt: https://dolthub.com/. It's a mysql-compat database that is versioned and forkable - basically imagine git and mysql had a baby. Every transaction creates an entry in the commit log that lets you see when, how, and why the database changed. And just like git you can `dolt checkout <hash>` to see what the data was like at some point in time, or `dolt checkout -b temp-branch <hash>` to make exploratory changes to some historical version, or `dolt revert` to revert a transaction... etc. There is a lot more power that comes with making the entire database versioned and forkable by default. For example it makes it much easier to recover from catastrophic bad code pushes, etc. note: Dolt was forked from Noms, my previous project, but I don't work for Dolt or have a stake. Just a fan.

See also https://www.dolthub.com/

Well there is Dolt: https://www.dolthub.com/