libtorrent Blog

Great suggestions! Actually, one of the inspirations for PicoTorrent came from this blog entry, http://blog.libtorrent.org/2012/01/seeding-a-million-torrents/. I haven't had a chance to test it with a million torrents yet, but my development environment has a few thousand at least, so easy bulk operations are a requirement :)

Is there a detailed overview about which functions in libuv (Nodejs) rely on blocking primitives and thus are using the thread pool to work async? From [here](http://docs.libuv.org/en/latest/design.html), it sounds like all file IO is always based on blocking primitives, and native async file IO primitives are not used, although such async file IO primitives do exists and were tried out in libtorrent (http://blog.libtorrent.org/2012/10/asynchronous-disk-io/). The result of that experiment however was mostly that the thread pool solution was simpler to code (I guess). From the libuv design doc, the overview is: * Filesystem operations * DNS functions (getaddrinfo and getnameinfo) * User specified code via uv_queue_work() I wonder whether this is really the best solution of if some combination of a thread pool and native async disk IO primitives could perform better.

trere is the bittorrent v2 standard: https://blog.libtorrent.org/2020/09/bittorrent-v2/ but unfortunately most foss torrent clients do not support it, partly because at release libtorrent 2.0.x had poor io performance in some cases so torrent clients reverted to the 1.2.x branch

BEP46 aka Dynamic Content Updates and Decentralization is, if I recall correctly, rolled into BEP-52 , aka The BitTorrent Protocol Specification v2 which has been the standard since 2020 (or so) and on paper since 2017 (ish). ( again, IIRC ) Question is, of course, how many release groups, public and private trackers are supporting this feature? https://medium.com/@kyodo-tech/bittorrent-protocol-v2-and-dynamic-cont… https://www.bittorrent.org/beps/bep_0046.html https://www.bittorrent.org/beps/bep_0052.html https://blog.libtorrent.org/2020/09/bittorrent-v2/

This tool should be upgrade to use Bittorrent v2 new functions. https://blog.libtorrent.org/2020/09/bittorrent-v2/ Especially merkle hash trees which enable : - per-file hash trees - directory structure

I don't think you understood the issue. Maybe read it again, or check the change in BitTorrent v2 which provides something similar as per-file trees https://blog.libtorrent.org/2020/09/bittorrent-v2/ (v2 support in clients in the wild is still very low though)

The biggest thing is that in IPFS each chunk has a CID, and that CID is globally unique and sharable from other CIDs. You can think of each CID as a mangent link, and each CID can point to more CIDs. Where as in a torrent the peer tracking is done at the whole torrent level, instead of the chunks inside the torrent. Torrent V2[0], which has poor adoption thus far, should also allow for a similar ability. To my knowledge this is not being taken advantage of yet. [0]: https://blog.libtorrent.org/2020/09/bittorrent-v2/

>BitTorrent, namely, has fixed chunks and data overlaps in them from adjacent files BitTorrent 2 fixes that: https://blog.libtorrent.org/2020/09/bittorrent-v2/ (hash trees section)

Was libtorrent v2 [0] finally adopted by enough clients and trackers to be used ? I never encountered a magnet link including the "urn:btmh:" (instead of "urn:btih:" for v1), specific to hybrid or full libtorrent v2 torrents. [0] https://blog.libtorrent.org/2020/09/bittorrent-v2/

bittorrent v2 is a bit different and more similar to IPFS: https://blog.libtorrent.org/2020/09/bittorrent-v2/

BitTorrent v2 also includes a per file hash tree https://blog.libtorrent.org/2020/09/bittorrent-v2/ so I assume that if you end up with the same file in multiple torrents, you'll be able to grab the files from seeds of these other torrents.

People are working on BitTorrent yes! libtorrents blog made a blog post quite recently (September 7th, 2020) - https://blog.libtorrent.org/2020/09/bittorrent-v2/

Node.js uses libuv for async file operations -- And libuv has its own internal thread pool to make file ops appear fully async to a node program. From the libuv docs [0]: Unlike network I/O, there are no platform-specific file I/O primitives libuv could rely on, so the current approach is to run blocking file I/O operations in a thread pool. For a thorough explanation of the cross-platform file I/O landscape, checkout this post [1]. [0] http://docs.libuv.org/en/v1.x/design.html#file-i-o [1] https://blog.libtorrent.org/2012/10/asynchronous-disk-io/

http://davmac.org/davpage/linux/async-io.html https://blog.libtorrent.org/2012/10/asynchronous-disk-io/

Let me rephrase it: there is nothing on any version of UNIX that supports an asynchronous file I/O API that integrates cleanly with the file system cache -- you can do signal based asynchronous I/O, but that isn't anywhere near as elegant as having a single system call that will return immediately if the data is available, and if not, sets up an overlapped operation and still returns immediately to the caller. This isn't a terrible recap of async file I/O issues on contemporary operating systems: http://blog.libtorrent.org/2012/10/asynchronous-disk-io/

Of course, a global attacker can always muck with the system. I was talking about single independent peers behaving badly on their little scale, such as returning wrong or no information, which I believe is more likely to happen. Also, the guys at Bittorrent are pushing for a limitation on how you can form your own ID based on your external IP [0], which should make it much more costly to perform a large-scale attack. Thank you for being pedantic :) [0] http://blog.libtorrent.org/2012/12/dht-security/

No, I don't have a demonstration since nothing like this was ever widely adopted, but it is an opinion shared among the bittorrent community: "One of the key algorithms in bittorrent is the rarest-first piece picker. It is vital to bittorrent’s performance that the piece picker fulfills both of these requirements: The rarest piece is picked (from the client’s point of view of the swarm) If two or more pieces have the same rarity, pick one of them at random The reason to pick a random rarest piece is to always strive towards evening out the piece distribution in the swarm. Having an even piece distribution improves peers’ ability to trade pieces and improves the swarm’s tolerance to peers leaving." http://blog.libtorrent.org/2011/11/writing-a-fast-piece-picker/

The .torrent file is separate to the torrent metadata which you need to fetch for magnet links. This is mentioned a bit later in that section: > The .torrent file size is not smaller for a v2 torrent, since it still contains the piece hashes, but the info-dictionary is, which is the part needed for magnet links to start downloading. Also it seems that it'll be easier to detect peers which are sending bad data and correct it. I guess this is because previously you could fetch a piece from more than one peer but because the hashing was done at the piece level you didn't know which part of the block was wrong (meaning you'd have to redownload the whole thing). But now you can -- with a bit of tracking -- tell which block of a piece was sent from which peer and then redownload it from another (likely blocking the original peer), because you know which block hash failed to validate. (EDIT: This is correct -- [1] explains the issue and how torrent clients have worked around this problem in v1 torrents.) Merkle trees have some other benefits (you can cache parts of the tree in such a way that repeatedly checksumming data where lots of leaves are unchanged is cheaper, and you can efficiently prove that a given leaf was part of the hash), but I don't know if that's going to be useful for BitTorrent. [1]: https://blog.libtorrent.org/2011/11/smart-ban/

>I did this for two reasons, the first being that I cannot, for the life of me, figure out how to use asyncio to do local file IO and not a network request, but maybe I'm just an idiot. I don't think you are an idiot I just think you didn't search well enough. There is a reason there is no local file IO with asyncio. Check these links for more info: * https://stackoverflow.com/questions/87892/what-is-the-status-of-posix-… * http://blog.libtorrent.org/2012/10/asynchronous-disk-io/ From what I understand the way libuv (what node.js uses) gets around OS limits is with a thread pool. Also, this documentation might be helpful: * https://docs.python.org/3.5/library/asyncio-dev.html#handle-blocking-f… My experiences don't seem to mirror your frustration. I found asyncio quite useful for tinkering with a simple web scraper that hit multiple sites at once (each with a different response time) and munged all the data together into one data set. Thanks for the write-up!