Marginalia (Viktor Lofgren)

The https://memex.marginalia.nu/ search engine runs basically on one desktop computer (if I recall, the creator made a post on this a week or so ago). How much of what google spends on "search" is strictly for search, vs business goals related to search? How much of that google spend is salary? What are those salaries? How do they compare with EU post-doc salaries?

My dude, them's fighting words. So I built a SSG in 50 lines of mostly idiomatic Java (one line is a bit squished, but 52 just doesn't sound as cool): https://github.com/vlofgren/50-line-java-ssg Does deal with partials, bit stuff like an automatic index requires maybe 5-10 additional lines. Took about an hour (plus 15 minutes to try to figure out how to get into my github account, which I haven't actively used in the last 15 odd years). My "real" Java SSG is used to generate https://memex.marginalia.nu/. Unfortunately I don't have the source published.

it is on the front page of their page: https://memex.marginalia.nu/projects/edge/supporting.gmi

Word vectorization + cosine distance, perhaps. See also this post by marginalia[1] (HN discussion[2]) which discusses the same thing. [1] https://memex.marginalia.nu/log/69-creepy-website-similarity.gmi [2] https://news.ycombinator.com/item?id=34143101

This sort of reminds me of a little post I read a while ago: https://memex.marginalia.nu/log/48-i-have-no-capslock.gmi I guess it's not entirely the same, but it has a similar idea of how text is morphed into near ilegibility by simplifying it so much.

The main issue is the conflating of usage and value. The persona/user group that consumes the Trending pages are likely users who themselves contribute to open source, and thus are part of a flywheel that drives more code contribution overall. (e.g. how 1% of users post on reddit vs 99% consume) Not all usage is made equally. This scenario reminds me of a funny story posted on HN a while ago: https://memex.marginalia.nu/log/48-i-have-no-capslock.gmi (TL;DR; if you take this "usage based" feature removal logic to it's conclusion, a lot of important keys would be removed from computers/keyboards).

i found these pages: about the search engine: https://memex.marginalia.nu/projects/edge/about.gmi about the encyclopedia: https://encyclopedia.marginalia.nu/wiki-clean.html

TLDR; There are lots of interesting tangents here, even if you don't want to run a web search engine. I'm in search of kindred spirits seeking to flush out the full power of the memex. I found the notes it quite interesting[1] I've gone through quite a bit of the related content, up to watching a bit about the Gemini protocol, which is interesting. 1 - https://memex.marginalia.nu/log/08-whatever-happened-to-the-memex.gmi

Apparently, if you omit all the we-got-you-and-lock-you-in-your-ad-induced-information-bubble, you can get quite far with even modest hardware: https://memex.marginalia.nu/projects/gemini-server.gmi > I put the machine together mostly for a search engine, Because I didn't want an actual rack making noise and heat in my living room, the server is made out of consumer hardware: > Ryzen 9 3900X > 128 Gb RAM > 4x4 Gb IronWolf ZFS > A bunch of SSDs index lookups

Though before basing life-and-death decisions on this, consider reading the "about" page first: https://memex.marginalia.nu/projects/edge/about.gmi > The purpose of the tool is primarily to help you find and navigate the strange parts of the internet. Where, for sure, you'll find crack-pots, communists, libertarians, anarchists, strange religious cults, snake oil peddlers, really strong opinions. and > If you are looking for fact, this is almost certainly the wrong tool.

> I'm thrilled someone is working on a search competitor to Google. Except the author goes to quite some lengths to explain that his search engine is not a competitor to Google, and is in fact exactly the opposite of Google in many ways: https://memex.marginalia.nu/projects/edge/about.gmi

I’ve been puttering away at making a search engine of my own (I should really do a Show HN sometime); let’s see how my experience compares with 18 years ago: Bandwidth: This is now also cheap; my residential service is 1 Gbit. However, the suggestion to wait until you’ve got indexing working well before optimizing crawling is IMO still spot-on; trying to make a polite, performant crawler that can deal with all the bizzare edge cases (https://memex.marginalia.nu/log/32-bot-apologetics.gmi) on the Web will drag you down. (I bypassed this problem by starting with the Stack Exchange data dumps and Wikipedia crawls, which are a lot more consistent than trying to deal with random websites.) CPU: Computers are really fast now; I’m using a 2-core computer from 2014 and it does what I need just fine. Disk: SATA is the new thing now, of course, but the difference these days is HDD vs SSD. SSD is faster: but you can design your architecture so that this mostly doesn’t matter, and even a “slow” HDD will be running at capacity. (The trick is to do linear streaming as much as possible, and avoid seeks at all costs.) Still, it’s probably a good idea to store your production index on an SSD, and it’s useful for intermediate data as well; by happenstance more than design I have a large HDD and a small SSD and they balance each other nicely. Storing files: 100% agree with this section, for the disk-seek reasons I mention above. Also, pages from the same website often compress very well…

If you dig the whole website (or geminisite, I dunno) you can find a post to which the very same criticism of the OP applies: https://memex.marginalia.nu/log/36-localized-programming-languages.gmi We all draw different lines for what feature is not common yet not rare enough to remove, so the context always matters. For this subject of character encodings, browsers have already reduced their repertoire of suppported encodings for a good chunk of the last decade: the Encoding Standard [1] is substantially smaller than what it used to be (I know because I did implement all of them back then). Some of them (e.g. UTF-7, not exactly "removed" from the Standard because it was never added AFAIK) caused security issues but others (e.g. HZ) didn't and only got removed by the lack of usage. This kind of decision, either for or against, can be only backed with telemetry-like quantification anyway. No criticism in the issue was based on the concrete evidence; Henri Sivonen is right to say that he "can't address remaining non-Latin problems without seeing concrete examples that need addressing". [1] https://encoding.spec.whatwg.org/

It's awesome! I like most suggestions, it feels a bit like https://wiby.me/surprise but generally less weird. I've often wanted to have a go at making my own search engine, and I think I would penalize any form of advertising (especially big ad networks, referral links) or tracking (Google Analytics, etc.) as these can create (or reveal) perverse incentives. This would likely get rid of most of the "SEO spam" that we see nowadays. Reading the about page[1], this seems like what you are doing here, but I can't really tell as it's light on details. Q: would this be able to handle foreign-language sites? I don't yet have a blog/personal website, but if I did, I guess it would be mixed-language. Should I submit some of my friends' blogs, even though they might not be entirely (or at all) written in English? A relatively new sort of search-engine junk, especially visible in non-English results from big search engines is also auto-generated (or probably machine-translated) websites, full of nonsensical content. They might be translated from genuine sites in other languages, I'm not sure. It would seem hard to fend these off, but luckily, fighting perverse incentives such as advertisement revenue probably gets rid of them too. I also wondered if this was curated list, and if the list was available somewhere, but it seems it's just a good old spider, and I guess that exposing too much info about the metrics might enable some to game the system? Not that marginalia is big enough…

Are there any technical infos about the search engine? Found some information here https://memex.marginalia.nu/projects/edge/about.gmi Author said they threw it together on consumer hardware. How big is the index? (TB used or entries) how is it realised? I'm pretty much interested in this since I myself am crawling some pages for my own "search index". Oh and thx for making and posting. Added it as a keyword to firefox Edit: Just realized that my question is a bit shallow. What I'm particular interested in is the storage before the indexing. I'm trying to store the raw html so that I can reindex everything with better algorithms, but I'm hitting many limits. It takes a few minutes getting the size of a site-directory (every site has it's own dir) and I'm at a point where I can't reasonably manage the scrape-versioning over git and I cycled through a few filesystems only to find that the metadata management kind of sucks for most of them. It's rather interesting how we store such files and I'm thinking about storing a few sites in a simple sqlite format for easy access and search. I'm thinking about a a few low overhead solutions like facebooks project haystack (implemented open source in seaweedfs) or something similar... Hopefully this gives some context to the question of storage and sites that are indexed