Seirdy

https://seirdy.one/ I'm trying to adopt as much IndieWeb as I can while still remaining a static JS-free site (except for the crappy search results page). Comments are Webmentions. I test compatibility with a lot more than just mainstream browsers: the Tor Browser's safest mode, various article extractors, NetSurf, Ladybird, w3m, and a dozen other user-agents work well. Accessibility-wise, I'm close to WCAG 2.2 AAA compliance, and have already passed AA; I consider WCAG a starting rather than a stopping point. More on its design is in the "Meta" section. It has long-form blog articles and short-form notes (microblogs). My best posts are on the homepage, followed by a bunch of webrings.

Mojeek is okay, but I think I'd struggle to use it as my primary search. I do include it in my personal metasearch results though, and find it worth paying for their API access (at my usage it's only about $20/year). If a page shows up in both Brave & Mojeek, it's probably a good one. I thought I'd tried Yacy, but apparently not - it's a distributed search. SearX is by asciimoo, the same person who has moved on to making Hister. asciimoo just posted about it here a few minutes ago: https://news.ycombinator.com/item?id=49159436 I feel like Seirdy has the definitive list of search engines, though the list is from 2021: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

I think a click stream isn't necessary, but Kagi is not a good basis for the argument in my opinion. Kagi is a primarily meta search engine. The click stream exists on their sources (Bing, Google, Yandex, Marginalia, not sure if they use Brave). They do have Teclis which is their own index that they use, and their systems for reordering the page of results such as downranking heavy ad pages, and based upon user preferences (which I love). https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ is a source I would recommend checking out if you are curious.

Here’s the best list of search indexes I’ve come across https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

A blog I read also uses Webmentions. This article is swimming in them: https://seirdy.one/posts/2020/11/23/website-best-practices/#interact

Actually seirdy.one added searchmysite.net to his excellent list[0] way back in March 2021[1]. [0] https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html [1] https://git.sr.ht/~seirdy/seirdy.one/commit/ab92d8ded69fd869baf7a0a415…

Yes, but I'm biased. Your perspective is as valid and I respect it. DDG does have some great features and we stand with them in fighting for user privacy. What benefits us all is a variety of different search engines and search services. It is the search engines that subtly determine our informational pathways, so variety is healthy. For one independent review of search engines and syndicates, including DDG see: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html

For searches where you want more that just the first result and want a depth of results to go through and maybe even check out more than the first page of results I like to use meta-search engines that grab results from multiple sources. Plus it helps route around censorship since you are getting results from a variety of sources. Searxng is the best known one. I was quite fond of ixquick but it shut down ages ago. These days I like etools.ch especially since it includes results from search engines like marginalia etc that I tend to forget to search directly but like having meshed into my general searches. Plus you can change which engines it uses in settings and it shows which search engine(s) each link came from which is handy. On a related note I like to check out Serdys list of search engines with their own indexes once if a while. It gets updated here and there and includes a fair amount of search engines I don't tend to see elsewhere. https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html

Very cool, I subscribed to the newsletter. I’ve experimented with retrieval and ranking across a sample of a million pages from the early days of the Common Crawl (around 2014) and I was surprised by how many of them seemed high quality. The CTO of CC tells me it’s because most of the early URLs were donated by Blekko, which was an old search engine that he used to work for. I don’t know what the quality of recent CC stuff is like, but I think it would be fun to supplement an index with this older data, especially because you’d get a lot of pages that are 404’s now (but you could deliver the extracted text to the user, or link to a temporally nearby snapshot from WayBack). Another fun thing to consider is making a meta search engine that functions like MetaCrawler used to, where it gets all (or a bunch of) the available results from all the source engines, and then actually fetches and extracts the text from the linked pages, and then matches the query and ranks the pages independent of what the source engines did. If you’d like to do that, I would recommend adapting the source code of 4get.ca (at least for the scrapers), because the guy who writes it is rather talented at coming up with and maintaining workarounds. If you monetize this, I’d be interested in working for you. I know Python, HTML, CSS, am familiar with JavaScript, and have a lot of experimental (and successful!) experience with ranking web results. Also, you might be interested in reading this article (from…

It's definitely a thing, but IME hosting a small instance, it's really not that bad. My advice for new admins would be to 1. find a small, decently up-to-date block list as a starting point (I can recommend FediNuke [0]), 2. check the #FediBlock hashtag (using any Mastodon UI or the RSS feed automatically generated) once a day or so for any warnings about bot attacks, or any other things you would like to be super proactive about, 3. promptly respond to the (likely very infrequent) reports you get from your users. I could see why this might not scale as well to instances with thousands of users, or if a user is a frequent target of harassment (where opt-in federation might be the only solution), but for our lab, the existence of all the nazi or whatever instances has been resolved via a simple "Oh, these again. click, click, click There, the whole instance is banned." a few times a year. I have a suspicion that Dunbar's number or lower might be the optimal size for any federated social media instance, since knowing every user on my instance makes it a lot easier to tailor the little effort I have to put in. [0] https://seirdy.one/posts/2023/05/02/fediverse-blocklists/

If you use a non-declarative adblocker, you're not just trusting the developer, but also all the third-party filter lists you've subscribed to. These filters have powerful capabilities and can even exfiltrate website data [1], and they are updated in real time, so if a bad actor pushed a malicious update (e.g. by gaining access to any EasyList contributor account), you would most likely be affected. However, it's true some websites (like YouTube) are especially problematic and a declarative adblocker is not enough. What you can do is combine both approaches: use a declarative adblocker (uBlock Origin Lite) as a baseline, and selectively enable non-declarative adblockers (uBlock Origin) for specific websites (see [2] for a detailed overview). I like this layered approach because it gets you the best of both worlds: the security and performance of a declarative adblocker, and the functionality of a non-declarative adblocker when you need it, without compromising your entire browsing session. [1] https://portswigger.net/research/ublock-i-exfiltrate-exploiting-ad-blo… [2] https://seirdy.one/posts/2022/06/04/layered-content-blocking/

> because there are no restrictions on things that just work with root, donation links for open source open geo data contribution platforms[1], roll your own payment scheme without giving anyone a cut, put ads in it if you like... Respectfully, that was not my point. The point was that having access to the source code fundamentally doesn't mean much. You can read more about why there since I don't want to open this debate again: https://seirdy.one/2022/02/02/floss-security.html > As does every web browser, but somehow banking on websites seems to very rarely be intercepted? Banking apps exist, and are required as a modern 2FA. Since 2021, strong 2FA is a requirement in the EU for banking operations. Mail clients also do this. DANE would be the ideal approach on web browsers. This might be up to a more general debate that doesn't belong here though. > Wouldn't it be nice if you could actually see what this app sends to Google about you? It's perfectly nice, and mitming is a great tool to ensure Google (and others) doesn't lie about what data they send. > Previously you'd add a cert to your OS and you'd be good to go. Now, this argument is rather moot though, since it's still doable. Not sure that the higher technical barrier would matter much, and most users will benefit from having certificate pinning anyway. Google is not doing this to prevent researchers to do their work. Nothing is permanently a blackbox anyhow. > Hah, the author just spent ~2800 words criticizing…

Try out some queries from your browser history and see if anything beats google. Other engines are more likely to beat google for queries that google has failed at. Majority of engines were found at below link. Surprisingly many engines use the same indices, for example duckduckgo is mostly just privacy-protected bing. Big thanks to Rohan Kumar. https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html A proxy is needed to bypass iframe restrictions for most websites. I used this x-frame-bypass client js code [https://github.com/niutech/x-frame-bypass] which (previously?) used proxy.megatunger.com . That free proxy could be recording your queries, who knows, so use caution. If (when actually) that proxy goes down I think I'll set up something like CORSflare to have a more reliable and trustworthy alternative. [https://github.com/Darkseal/CORSflare] I hacked this together over about one workday in total so it's likely to have significant issues. You can comment them here or (preferably) file an issue on the repo: [https://github.com/qpwo/searchall/] There's also a demo video on the repo if the site goes down but you wanted to see how it looks. Feel free to file feature requests on the repo as well. If you'd rather copy it and make your own version then that's fine too. I'll try to reply to all comments here.

a list of engines I ran across the other day: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ The three dominant English search engines with their own indexes are Google, Bing, and Yandex but this list has many spiders and engines that traverse the web.

The Google index is still accessible from many other "proxy search engines" that still work without JS, one example is Startpage. See the nice list from Seirdy for more details on search engines: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

Also check out this post (not mine): https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

>how many global Web indices do you think exist? Here's a list for the curious: >A look at search engines with their own indexes https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

The alternative search engines are powered by Bing, Yandex, or only index a small part of the web https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

Most search engine popping up are just a rebranding of bing and yandex. There is a good article about it: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ Regarding your question of cost, if you buy the cheapest hardware possible, and manage it yourself, because you don't want to pay AWS' premium, for storage (again, assuming to discard 90% of the pages, and only do english) you'll be at least at €80k (12 × €3000 disk servers with 300 × €150 HDDs) of upfront cost, then, if you colocate at Hetzner, just for storage, you'll bet at €500/month + ~€3k of electricity that you need to pay yourself. My gut feeling is that this would cost (only 10% of the English-speaking web) ~€150k of upfront cost and €5k/month to run. Assuming you buy the cheapest everything, and do your own admin sys. And this is not forecasting growth, serving ads, etc...

Pretty much proof that OSS != automatically more secure. And proof that OSS projects can get backdoored. See this for more ideas on this issue: https://seirdy.one/posts/2022/02/02/floss-security/

Kagi is shit. Use kagi if you use godaddy. Startpage uses google. https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/ Also https://crowdview.ai/ for 100% organic, forum search. I don't use kagi.com because I don't like idea of linking my credit card to search queries.

Good overview of search alternatives here https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

Not to disappoint, but currently Marginalia and now this. (It used to just be Marginalia, no idea why it was a folder) Though now that there's two I'm probably gonna pick up a few more from here that I like: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

It does give better results. The filters can also be quite powerful. The problem is paying a steep fee to proxy Google's index... It benefits you and Kagi in the short term, but strengthens the Google/Bing/Yandex index oligopoly [0] long term. Bad search results are the symptom - scarcity and opaqueness of viable indices is the root cause https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

Yes, it is. Bing has gotten a lot better recently. This is a good write up on indexes: https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/

They list out their reasons here https://seirdy.one/posts/2021/03/10/search-engines-with-own-indexes/#e… Sounds like two were excluded for having a far-right focus, one for crypto payments, and the rest for being small/simple proof-of-concepts That said, I would also be interested in what engines in particular were excluded, with a small blurb as to why it wasn't included.

> First, it is misleading to say our results just come from Bing. That's far from the case in actuality. That's just corpo-speak. For the most part, Duckduckgo is Bing with some additional features. That's true to the extent that when Bing decided to censor the Tank Man image, it was removed from your results too. [1] Not that you guys refrain from censorship yourselves. [2] The crawler (DuckDuckBot) doesn't have much of an impact on the search results, it's mainly used to provide instant answers. [3] [1]: https://www.theregister.com/AMP/2021/06/04/search_engine_tiananmen/ [2]: https://nitter.net/yegg/status/1501716484761997318 [3]: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html#fn:…

The search engine space is full of cool competitors today. Google, Bing, Yandex, DuckDuckGo, Kagi, Neeva, Brave, Startpage, Alexandria, Right DAO, Wiby, Marginalia, Teclis. Certainly, a few of those are Bing/Google frontends, but many more are independent or weighted mixes of commercial and noncommercial indexes. I really like Kagi, Wiby, Marginalia (Search), and Teclis to name a few. I found most of these through this post: https://seirdy.one/2021/03/10/search-engines-with-own-indexes.html