aliases:

• Community Search Engines tags:
• search
• localfirst

Community Search Engines are an approach to build highly relevant search indexes and collaborative content spaces for communities.

The interface is similar to agent chat interfaces -- chatting and refining to get what you're looking for -- as well as personal notes / bookmark collections¹ -- browsing and interacting with "your stuff" to find what you've stored before.

I'm using the label search engine as a term that I think will resonate with the mass market: you go there to find stuff.

What are we trying to fix?#

Google has broke the social contract of search. Even without [[AI slop]] being placed as the first "result", the results returned lead to top ranking pages that themselves are low quality SEO garbage.

I can't easily find results that are useful or relevant to me any more.

OK, I'll ask my friends for what they think! But, on corporate social platforms such as Twitter, Instagram, or Threads, the algorithm is centrally controlled and can't be scoped to your friends.

You're not sure that your request will be seen by your connections, and your connections won't necessarily see your post.

Discovery and Distribution#

This covers two aspects:

• discovery -- browsing and finding what you're looking for, ideally with a tune-able, transparent algorithm
• distribution -- being able to share "your stuff" in a way that it's possible for others to in turn see it and discover it

For both, we think that [[Open Social Protocols]] offer a way forward. We also think distribution is up stream of discovery aka [[DREAM]].

If your information isn't seen because it isn't indexed or shared, or is algorithmically black holed or suppressed, it can't be discovered.

Emerging Tools#

[[DXOS]] has their [[DXOS Composer]] suite. Local first content, LLM models, and composition of a number of different tools, from Discord API connections for indexing, to audio / video transcription.

The [[Patchwork]] system by [[Ink & Switch]] stores data local-first, cached in your browser. It relies on [[Automerge]] for sync. It doesn't have much in the way of a search index at all, and until [[Keyhive]] is integrated, doesn't have a permissions system. It can do multiplayer collaboration by sharing links to content.

[[Groundmist]] is a proof of concept that uses [[Ink & Switch]]'s tiny essay editor with local-first, private data (a precursor to Patchwork, it also uses [[Automerge]]) and combines it with [[ATProtocol]] for public publishing as well as structured data via the AT Protocol Lexicon system.

[[Tonk]] has just released the Tonk CLI and TonkbookLM. Uses [[Automerge]]. Specifically mentions [[Obsidian]] as a source of local data.

How do we build community search engines?#

I envision a [[Personal Notes & Publishing Stack]] that enables people and communities to collaborate, pool information, and add context and relevance.

Further Reading#

[[Dark Forest and Cozy Web]]: communities are moving to cozy web spaces, and/or have need for members only content. The open internet has degraded search incentivized by click throughs to show ads, and commercial social platforms

[[Brazilianization of the Internet]]: forming communities, which can be considered a kind of commons, trending towards [[Cozy Web]] spaces. Does this cause a kind of elitism, and/or further deteriorate public spaces?

[[Erin Kissane]]'s [[Against the Dark Forest]] specifically argues that we need to protect the publis social internet and not all retreat into [[Cozy Web]] corners. The entire article is dense and amazing and I could quote every paragraph.

[[How Algolia uses Electron to improve internal productivity]]: I’m still inspired by this description of Algolia implementing company wide search across internal tools, with desktop integration. This should be a tool for organizations of all kinds.

Previously#

I rewrote the page that was here previously to refactor it extensively.

I applied to [[SoP 2024 Application]] with some ideas around the original formulation of using events and digital communities as a way to bootstrap the search engine, without tackling the [[Personal Notes & Publishing Stack]].

Part of community is "flow" ([[Stock and Flow]]) and [[Proto Apps]] is how I think social graphs / communities / flows of information form.