Dat Project Blog

Interesting project, particularly with the choice of IPFS and DCAT -- something I'll have to look into. There have been other efforts to handle mostly file-based scientific data with versioning in both distributed (Dat https://blog.datproject.org/tag/science/) and centralized ways (DataHub https://datahub.csail.mit.edu/www/). Juan Benet visited our research center to give a talk about IPFS a few years ago. Really fantastic stuff. I'm the creator of DVID (http://dvid.io), which has an entirely different approach to how we might handle distributed versioning of scientific data primarily at a larger scale (100 GB to petabytes). Like Qri and IPFS, DVID is written in Go. Our research group works in Connectomics. We start with massive 3D brain image volumes and apply automated and manual segmentation to mine the neurons and synapses of all that data. There's also a lot of associated data to manage the production of connectomes. One of our requirements, though, is having low-latency reads and writes to the data. We decided to create a Science API that shields clients from how the data is actually represented, and for now, have used an ordered key-value stores for the backend. Pluggable "datatypes" provide the Science API and also translate requests into the underlying key-value pairs, which are the units for versioning. It's worked out pretty well for us and I'm now working on overhauling the store interface and improving the movement of versions between servers. At our…

HN is pretty niche. In some countries like the phillipines, most internet users stay on Facebook for almost their entire session. Even so their internet is filtered through Facebook Basics (and one could make the argument our internet is filtered through google). Even when looking at reddit in the western world, most people don't click the links, they stay on reddit. I would say we did get something wrong by not regulating enough and allowing companies to monopolize the internet and further to act as private intelligence agencies with no legal liability for the data stored. I take issue with the author mentioning blockchain though. Im not going to derail the thread but everyone should look into datproject as the future: https://blog.datproject.org/2018/01/16/dat-privacy-models/

Duncan Keall is just talented. He also did the Scuttlebutt Protocol guide (https://ssbc.github.io/scuttlebutt-protocol-guide/). Here is a blog post describing how it was made: https://blog.datproject.org/2019/01/21/how-dat-works/

Props to Duncan Keall for creating the diagrams [1][2]. > Duncan is a technical writer and illustrator interested in internet infrastructure, peer-to-peer protocols and net art. [1] https://blog.datproject.org/2019/01/21/how-dat-works/ [2] https://github.com/datprotocol/how-dat-works/commits?author=vtduncan

In this recent essay, Darius Kazemi suggests that Scuttlebutt and Dat (as well as Activitypub) should be regarded as complementary efforts rather than as re-decentralization silver bullets: https://blog.datproject.org/2019/03/22/three-protocols-and-a-future-of…

> businesses need to make money This is another reason to prefer federation: your social graph is not owned by (at the mercy of) a business whose interests may not even be aligned with yours (bad advertising is only one of the many consequences of this). > extreme I'd not call "there's something better" an extreme approach. The maintenance burden is what makes federated services difficult to operate. That's the issue right now, but people are working to resolve that too. > https://blog.datproject.org/2019/03/22/three-protocols-and-a-future-of… > https://news.ycombinator.com/item?id=19466929

Cool, thanks :) Bittorrent with streaming interface actually works pretty well. Edit: Re Dat, this is interesting, and promising: https://blog.datproject.org/2017/12/10/dont-ship/

It's been almost 5 years at this point somehow! Check out the first commit: https://github.com/datproject/dat/tree/464679267049899eafa345125a0f221… Bit more on the funding history, etc. : https://blog.datproject.org/2017/09/15/dat-funding-history/

I believe the way Dat works is if you don't have the public key for a Dat Repository (AKA the Dat URL), then you'll never find it even if other people are sharing it because peers will only respond if they are asked for that specific public key. However, as soon as you have that public key, then you can see the IP addresses of everyone sharing it. This seems like a privacy problem with the Internet itself that the Tor network is attempting to at least help solve. - https://docs.datproject.org/security - https://blog.datproject.org/2016/12/12/reader-privacy-on-the-p2p-web/

A few other CRDT / collaborative text editor links I've collected: * https://blog.datproject.org/2019/03/05/caracara-react-dat-automerge/ * https://conclave-team.github.io/conclave-site/ * https://github.com/noffle/hyperpad * https://xi-editor.io/xi-editor/docs/crdt.html * https://github.com/atom/teletype-crdt * https://github.com/atom/xray * https://github.com/MicrosoftDocs/live-share