beepb00p

I stopped worrying and in most cases just write configs for my tools in python. Then you can just import/exec it and you're done. I can use all the operations/primitives I already know in Python: string interpolation & operations, loops, Pathlib, imports etc. I can use mypy and all the other existing linting tools to make sure my configuration is correct without having to write a custom linter (and basically reimplement 10% of mypy). Shameless plug if you wanna read a longer analysis: https://beepb00p.xyz/configs-suck.html A great example is pyinfra https://github.com/Fizzadar/pyinfra#readme Think Ansible but instead of YAML you write Python. It provides a set of primitives/DSL and some rules you need to adhere to, but otherwise you just write regular python code.

Matt, did you see Promnesia? https://beepb00p.xyz/promnesia.html There are a lot of converging ideas out there, for "self-hosted NSA of yourself" type software which gather your own data from services you use and index it / make it more accessible to users. Would love to see something serious come out of it. I followed Timeliner early on and really like the idea, but yeah it's a bitch to get traction for this sort of thing. You need people willing to implement backends for basically everything, then maintain them.

My version of this is called Dogsheep, and involves building tools that suck my data from various sources into SQLite databases so I can query it all using Datasette. https://dogsheep.github.io/ I enjoyed reading this author's essay on why he doesn't think loading everything into relational databases is the best approach: https://beepb00p.xyz/unnecessary-db.html I think I have a workarounds for most of the issues he describes there. One example: I mostly create my base table schema automatically based on the shape of the incoming JSON, and automatically add new columns if that JSON has keys I haven't seen before. Very thought-provoking writing (and software) here.

https://beepb00p.xyz/ - Dmitrii writes a lot about quantified self and building personal knowledge bases.

No global solution I've seen. Best overview of the landscape I've seen so far is here [0] Personally I've been using a combination of Roam Research [1] and Readwise [2]. Used Notion [3] for a while but it's very hard to tag data. 0 - https://beepb00p.xyz/annotating.html 1 - https://roamresearch.com/ 2 - https://readwise.io/ 3 - https://www.notion.so/

Let’s build it: The Sovereign Individual (1997) + The Sovereign Consumer (2018) + AI Community of people that practice and develop tools for achieving digital sovereignty, digital immortality and autonomy. Roadmap of products: • social co-pilot | social os for building and maintaining relationships with best people • life OS — ai-native workspace with long-term tasks to be executed by ai, gamified, execution-free. • self-hosted automated LLM-based recomendation system (aka beeper.com + Loyal AI) with MCPs and A2A protocol. • self-hosted self-sovereign identity (ref. timelinize.com + https://beepb00p.xyz/hpi.html Roadmap of problems to solve: • solving loneliness and isolation for people (stop losing people and messages from) • solving social engineering for senior people • solving productivity for power users through automations References: • socialinbox.dev • lifeweeks.app • timelinize.com • beepb00p.xyz/hpi.html • beeper.com • reddit.com/r/DataHoarder • reddit.com/r/selfhosted/s/n8uxvYYa1V • https://postplatforms.org • book: The Sovereign Individual (1997) + Sovereign Consumer (2018) • small web / small technology https://ar.al/2020/08/07/what-is-the-small-web/ • indieweb.org 1000 people sovereign community • #quantified self • independence part of diagram.website

If this is interesting to you, you should check out the interesting work that karlicoss and others have done with "Human Programming Interface" [0] / [1]. I've been kicking this idea around for quite a few years and have gone through multiple iterations before finding HPI and tossing out or adapting all my work in favor of building off theirs. Mine is a bit more service / cloud oriented in how it runs (shocker... at one point I made it into a product) while HPI is heavily local-first for obvious good reasons. HPI is a great platform to build your own stuff off and benefit from all the work that has already been done because imo building a good foundation is the hardest part. Sean Breckenridge's HPI-API is super interesting and useful, could likely be worked into this search engine concept, quite sure Sean actually has both newsboat and Firefox modules already made. I wrote modules of my own and made an authentication wrapped HPI-API and a GraphQL instance but currently in the middle of an infra move so nothing super cool to show off. At one point I had a dashboard powered by it but we all know how these one-off internal-use-only projects end up. :) I think the most interesting thing I've written for HPI is my ActivityWatch syncing. It's been happily churning away for years and during my infra move, I discovered that it had about 20GB of activity data stockpiled. Multiple years of down to the second cataloguing of everything I did and saw on the computer. Lots of…

James is one of the world's great techno-adventurers, & getting to para-socially share in wild adventures like this makes living on Spaceship Earth more lovely & lively! James has also done cool projects like sweet.js macros, helped kick off Firefox devtool's transition to react (iirc), oh and lead the basically industry standard JS formatter Priettier project. I'm forgetting a dozen other things over the years but it's always been fun. Just a heads-up, the File System Access API[1] is underway in Chrome, which potentially removes nearly all of the absurdity here. It has other benefits too. A web page using this could write a .sql file on to your drive, that other programs could then access. One of the other bright stars in my world is Karli Koss, who has an extensive personal data-extraction setup for a ridiculously colossal variety of services & devices[2]. A vast amount of this massive massive data-gathering framework is just reading sqlite databases of the various devices and apps. If the web can help participate more actively, can let apps write sql files to store state: so much the better I say. Help externalize your state beyond the browser, please! [1] https://wicg.github.io/file-system-access/#api-filesystemwritablefiles… https://caniuse.com/native-filesystem-api [2] https://beepb00p.xyz/myinfra.html

My general use of Pinboard goes like this: Several times a day: Bookmark a thing (usually marked as 'unread'). Every week or so: Look back at my unread bookmarks. Read, or just mark read because I want to keep a bookmark but I don't want to actually read. A few times a month: Use Pinboard to look something up that I remember seeing, or I don't - but hope I did actually come across already. Full text search helps here. I wonder if Maciej would consider getting some browser extensions and mobile apps developed? I've probably paid the same for apps over the years as I have for Pinboard itself [1], and they've generally been terrible. I'm certainly willing to pay more than I already am for Pinboard if it comes with some decent tools. Something like Promnesia [2] and note taking being integrated would make it the tool I use the most all day, so this would be well worth it. [1] I'm a paying Pinboard user since 2011 and have been paying for full-text indexing since it became available. [2] https://beepb00p.xyz/promnesia.html

Yes, this siloing stops me from using annotation features "native" to each app and format as such as well. Instead, I prefer to exfiltrate information from the silos(apps, formats, etc) and put them into my note taking system. Then I can do highlights, annotations, etc. on my own terms and also get the benefits of centralization such as searching and linking(the OP has another post describing their own system, which is pretty cool[1]). Currently I'm using Notion, which is also a silo of its own, but it's one that gives me a lot of control over how I lay my information out(and an escape plan from). There are a lot of perspectives on this issue with data silos and walled gardens. But I'm of the opinion that it's a fairly bad state for all of us "end users". Computers to me are about infinite flexibility and malleability, but ironically the tools we have for annotation and remixing are in practice worse than what we have in the physical world. Reading a book in the physical world, I can converse with the author simply by jotting marginalia with my pencil. It's fluid, intuitive, and the medium of paper encourages it(in fact it can't help but be mutated by my use!: pages get bended, stained, torn, etc.). If I want to go further I can add post-it notes to mark interesting passages, I can xerox some pages and create subsections, if it's a magazine I can just tear them all out! That kind of flexibility just isn't available on a computer. I think it's worth thinking really hard…

I'm struggling to remember/find the exact story but like 6 months ago, some dev had built a guestlist or q&a & used some off the shelf Notion-y thing - maybe some form builder tool. There was seemingly IMHO a lot of protest the dude didn't make some kind of php script or something one off & "simple" for the job. But they'd used existing tools.to make a real data pipeline. And potentially could keep making new tools around similar pipelines. They had invested in some pipe building technology and it felt like no one was interested in giving credit for that. Seperately, Karlicoss has HPI as their personal data toolkit/pipeline, and a massive map of services & data systems they've roped into HPI (and HPI-near) systems. https://beepb00p.xyz/hpi.html https://beepb00p.xyz/myinfra.html

This is a horrible use case for Windows Recall. Even if we ignore all the privacy implications of having a third party screenshot you every 30 seconds and making the files world readable, it's a bad idea. Recall has lost a ton of useful metadata you already have - both URL visits and streaming are clearly discernible actions, both at the network stack level, and from your browser history. Throwing that away to trust an LLM to re-infer the same data is both reducing data fidelity and significantly increasing processing cost. If you want to see this done reasonably well, I'd suggest looking at e.g https://beepb00p.xyz/promnesia.html (which not surprisingly bears a strong similarity to what the article discusses) LLMs don't add much value here, outside of tightly locked down systems where screenshots are the only way of exporting.

Yes!! https://beepb00p.xyz/unnecessary-db.html really changed how I think about data manipulation, mostly with this principle.

This looks pretty interesting! I'll try it. I have added a link to dron and to your "In search of a better job scheduler" post (https://beepb00p.xyz/scheduler.html) to https://dbohdan.com/fcron#see-also.

I just started this: A collection of tools to pull in personal data from various sources, in the spirit of a "quantified self". I just got it to a state yesterday where I can do do this: https://raw.githubusercontent.com/VitamintK/data-fetishism/main/figure… Source code: https://github.com/VitamintK/data-fetishism Eventually I'd like to get it to the point of something like this: https://beepb00p.xyz/myinfra.html

Promnesia is meant for that (I didn't test it though): https://beepb00p.xyz/promnesia.html

Promnesia is a browser extension to help you develop a memex. https://beepb00p.xyz/promnesia.html

https://beepb00p.xyz/sad-infra.html This one had some impact on me because it revived the importance of taking ownership not only on my personal data, but the infrastructure which generates it, processes and stores it; and that the same infrastructure should be well thought of by me through a tinkering process which adds value to itself and to data the more I invest time on it. I felt very motivated to rethink my workflows, my opsec and the way I store any data, not only new data I create, but general content that find around the Internet and that I think could be of any use in the future. An honorable mention would be this one: https://staltz.com/the-web-began-dying-in-2014-heres-how.html which helped me look at how the web has been evolving from a different angle and, of course, triggered me to change some of my behaviors regarding privacy and the use of big tech derived products.

There's an explanation here: https://beepb00p.xyz/unnecessary-db.html (the link behind "Against unnecessary databases")

What are you talking about, this is a package for Emacs, which already is a web app: https://beepb00p.xyz/cloudmacs.html

Another principle virtue of Dhall (stemming from the type system) is that it's total, so you know no funny business can go on when computing the configs. Even if you use typescript for (unsound) static type checking, you're definitely not going to get totality out of it. If someone wanted dhall in JS, they could implement it, as has been done for Go, Ruby, Clojure, etc.[1] iiuc, this library is much more akin to the approach described in https://beepb00p.xyz/configs-suck.html [1]: https://docs.dhall-lang.org/howtos/How-to-integrate-Dhall.html#languag…

For annotating, you can try org-noter: https://github.com/weirdNox/org-noter If you want to get your annotations into org mode/emacs, look at this guy's work: https://beepb00p.xyz/orger.html I haven't yet tried it, but I think he does something similar.

I generally archive as pdf ONLY things really interesting me, in org-mode/org-roam managed notes as org-attachments, most other links are just noted with a not-that-good but the best I found combo: - https://addons.mozilla.org/en-US/firefox/addon/copy-as-org-mode/ and https://github.com/kuanyui/copy-as-org-mode - https://beepb00p.xyz/promnesia.html and https://github.com/karlicoss/promnesia and https://addons.mozilla.org/en-US/firefox/addon/promnesia/ - https://beepb00p.xyz/grasp.html and https://github.com/karlicoss/grasp/ and https://addons.mozilla.org/en-US/firefox/addon/grasp The first to quickly copy text from html, the second to see if something is already noted, the third to quickly archive the bookmark. In the past I've used Zotero witch works automatically VERY well (and on deduplicated storage does not consume so much disk space) but since it's a kind of walled-garden in the sense I can't really integrate it in anything else I decide to have a bit less features but hyper-superior integration with org-mode.

I was just going to say you should lookup this - https://beepb00p.xyz/myinfra.html - a page I bookmarked and have kept at the back of my head, and turns out it's you! I have a strong personal need for such a system. I have severe chronic pain that severely disrupts my executive function, and so it's hard to mentally self-direct - so having a visual map of data interconnectivity, where in part I could set view layers based on priority (hide or show different data depending on say if viewing "holistic view" vs. a "project" that's arguably data that's tagged with a project name, etc. I'm also wanting to write a book on my journal with health and healing, among other things system-society wide that influence health - and allowed my situation to exist and dis-ease progression to occur - and so having a singular system for me to access things like my bookmarks, notes in Notes app, emails filtered based on tags, my comments on HN/Reddit/etc. would really help in allowing me to start to organize and compile different data; perhaps a way to add a non-impacting link between different pieces of data, e.g. say there's a topic in the book related to food and/or water fasting - I could start to interlink all of that content first with links, then another step in the process perhaps with version control integrated - so I can archive a previous/unedited version, keeping the polished/updated version (or multiple data files amalgamated) but linking to their sources/prior versions,…

The holy grail [https://beepb00p.xyz/pkm-search.html#future] of this really resonated with me and fully mirrors what I've been thinking about the past few months. In my observations, it's input capture, information organization, and subsequent retrieval: Information Capture: Input Capture - You’re going to have all-encompassing tracking and recording of all activity, but want configurable privacy on the extent to which you want your daily conversations and observations of external things you encounter and are exposed to. Capturing input needs to be holistic and incorporate all properties of encounters and new information. Potential sources of input: Vision — point of view recording, see snapchat spectacles, etc as primitive examples. Audio (voice notes and multi-party conversations) - voice calls, video, etc. and other forms of audio transmission where there is more than a single party in the interaction. Digital interactions You will need to keep track of web pages you visit at what times Conversations you see on Twitter, etc. Properties and cues must be extrapolated from the information that is captured on input, in the case of audio, transcriptions are sufficient for transcription and retrieval purposes, however since video is a visual medium, it includes significantly more properties that need to be accounted for. The aim here is to identify sufficient data points (cues) that are subsequently represented in such a way that they are easy to search across things you…

I want to build a browser that captures everything I saw on the internet, allows me to search it, run graph algorithms (like PageRank). Improves navigation (by showing trails as tree instead of tabs). Heavily offline focused (Backend only for updates, maybe for analytics). Difference with rewind.ai: linkkraft does not have funding, i'm solo, no apps & image/video/audio recognition. Focus on web, trails, research and using web copies, selections/highlights as part of your notes & whiteboards. Preserving all possible graphs. My inspirations: https://pages.gseis.ucla.edu/faculty/bates/berrypicking.html, https://beepb00p.xyz/promnesia.html, Jeff Raskin (Global Search, Zoom UI) https://linkkraft.com/notes/backstory I've built a prototype with trails tree & HTML snapshoting. For each my step even inside SPA linkkraft creates HTML snapshot. https://arestov.github.io/linkkraft-notes/comparing/linkkraft-vs-chrom… https://arestov.github.io/linkkraft-notes/trails-tree-plus-offline-for… Across similar software ("too many tabs" solvers, snapshoters) combination of trails tree & snapshots that reinforces is unique in linkkraft. To make snapshot it takes current state of DOM, current state of styles (to capture CSSinJS stuff), take resources like font/images from network storage (Chrome Dev Tools protocol) and saves it as HTML on disk. I've been developing created frontend in-memory graph DB / state manager for 1) replication 2)…

If you’re interested in this ‘Personal Data Warehouse’ concept, you might also be interested in @karlicoss’s articles[0][1][2] about his infrastructure on saving personal data. There are various technical differences, for example @simonw preferring lightweight SQLite databases compared to @karlicoss preferring dumping the data and parsing-when-needed[3], but the purposes are similar enough that I think they should be mentioned. There were also some very constructive HN discussion[4] on these articles, where @simonw have also introduced Dogsheep before :-) [0]: Building data liberation infrastructure — https://beepb00p.xyz/exports.html [1]: Human Programming Interface — https://beepb00p.xyz/hpi.html [2]: The sad state of personal data and infrastructure — https://beepb00p.xyz/sad-infra.html [3]: Against unnecessary databases — https://beepb00p.xyz/unnecessary-db.html [4]: https://news.ycombinator.com/item?id=21844105