nekomimi.pet
forked from
microcosm.blue/microcosm-rs
Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm
1microcosm: links
2================
3
4this repo contains libraries and apps for working with cross-record references in at-protocol.
5
6
7App: [Constellation](./constellation/)
8--------------------------------------------
9
10A global atproto backlink index ✨
11
12- Self hostable: handles the full write throughput of the global atproto firehose on a raspberry pi 4b + single SSD
13- Storage efficient: less than 2GB/day disk consumption indexing all references in all lexicons and all non-atproto URLs
14- Handles record deletion, account de/re-activation, and account deletion, ensuring accurate link counts and respecting users data choices
15- Simple JSON API
16
17All social interactions in atproto tend to be represented by links (or references) between PDS records. This index can answer questions like "how many likes does a bsky post have", "who follows an account", "what are all the comments on a [frontpage](https://frontpage.fyi/) post", and more.
18
19- **status**: works! api is unstable and likely to change, and no known instances have a full network backfill yet.
20- source: [./constellation/](./constellation/)
21- public instance: [constellation.microcosm.blue](https://constellation.microcosm.blue/)
22
23_note: the public instance currently runs on a little raspberry pi in my house, feel free to use it! it comes with only with best-effort uptime, no commitment to not breaking the api for now, and possible rate-limiting. if you want to be nice you can put your project name and bsky username (or email) in your user-agent header for api requests._
24
25
26App: Spacedust
27--------------
28
29A notification subscription service 💫
30
31using the same "link source" concept as [constellation](./constellation/), offer webhook notifications for new references created to records
32
33- **status**: in design
34
35
36Library: [links](./links/)
37------------------------------------
38
39A rust crate (not published on crates.io yet) for optimistically parsing links out of arbitrary atproto PDS records, and potentially canonicalizing them
40
41- **status**: unstable, might remain an internal lib for constellation (and spacedust, soon)
42
43
44
45---
46
47
48old notes follow, ignore
49------------------------
50
51
52as far as i can tell, atproto lexicons today don't follow much of a convention for referencing across documents: sometimes it's a StrongRef, sometimes it's a DID, sometimes it's a bare at-uri. lexicon authors choose any old link-sounding key name for the key in their document.
53
54it's pretty messy so embrace the mess: atproto wants to be part of the web, so this library will also extract URLs and other URIs if you want it to. all the links.
55
56
57why
58---
59
60the atproto firehose that bluesky sprays at you will contain raw _contents_ from peoples' pdses. these are isolated, decontextualized updates. it's very easy to build some kinds of interesting downstream apps off of this feed.
61
62- bluesky posts (firesky, deletions, )
63- blueksy post stats (emojis, )
64- trending keywords ()
65
66but bringing almost kind of _context_ into your project requires a big step up in complexity and potentially cost: you're entering "appview" territory. _how many likes does a post have? who follows this account?_
67
68you own your atproto data: it's kept in your personal data repository (PDS) and noone else can write to it. when someone likes your post, they create a "like" record in their _own_ pds, and that like belongs to _them_, not to you/your post.
69
70in the firehose you'll see a `app.bsky.feed.post` record created, with no details about who has liked it. then you'll see separate `app.bsky.feed.like` records show up for each like that comes in on that post, with no context about the post except a random-looking reference to it. storing these in order to do so is up to you!
71
72**so, why**
73
74everything is links, and they're a mess, but they all kinda work the same, so maybe some tooling can bring down that big step in complexity from firehose raw-content apps -> apps requiring any social context.
75
76everything is links:
77
78- likes
79- follows
80- blocks
81- reposts
82- quotes
83
84some low-level things you could make from links:
85
86- notification streams (part of ucosm)
87- a global reverse index (part of ucosm)
88
89i think that making these low-level services as easy to use as jetstream could open up pathways for building more atproto apps that operate at full scale with interesting features for reasonable effort at low cost to operate.
90
91
92extracting links
93---------------
94
95
96- low-level: pass a &str of a field value and get a parsed link back
97
98- med-level: pass a &str of record in json form and get a list of parsed links + json paths back. (todo: should also handle dag-cbor prob?)
99
100- high-ish level: pass the json record and maybe apply some pre-loaded rules based on known lexicons to get the best result.
101
102for now, a link is only considered if it matches for the entire value of the record's field -- links embedded in text content are not included. note that urls in bluesky posts _will_ still be extracted, since they are broken out into facets.
103
104
105resolving / canonicalizing links
106--------------------------------
107
108
109### at-uris
110
111every at-uri has at least two equivalent forms, one with a `DID`, and one with an account handle. the at-uri spec [illustrates this by example](https://atproto.com/specs/at-uri-scheme):
112
113- `at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3jwdwj2ctlk26`
114- `at://bnewbold.bsky.team/app.bsky.feed.post/3jwdwj2ctlk26`
115
116some applications, like a reverse link index, may wish to canonicalize at-uris to a single form. the `DID`-form is stable as an account changes its handle and probably the right choice to canonicalize to, but maybe some apps would actually perfer to canonicalise to handles?
117
118hopefully atrium will make it easy to resolve at-uris.
119
120
121### urls
122
123canonicalizing URLs is more annoying but also a bit more established. lots of details.
124
125- do we have to deal with punycode?
126- follow redirects (todo: only permanent ones, or all?)
127- check for rel=canonical http header and possibly follow it
128- check link rel=canonical meta tag and possibly follow it
129- do we need to check site maps??
130- do we have to care at all about AMP?
131- do we want anything to do with url shorteners??
132- how do multilingual sites affect this?
133- do we have to care about `script type="application/ld+json"` ???
134
135ugh. is there a crate for this.
136
137
138### relative uris?
139
140links might be relative, in which case they might need to be made absolute before being useful. is that a concern for this library, or up to the user? (seems like we might not have context here to determine its absolute)
141
142
143### canonicalizing
144
145there should be a few async functions available to canonicalize already-parsed links.
146
147- what happens if a link can't be resolved?
148
149
150---
151
152- using `tinyjson` because it's nice -- maybe should switch to serde_json to share deps with atrium?
153
154- would use atrium for parsing at-uris, but it's not in there. there's a did-only version in the non-lib commands.rs. its identifier parser is strict to did + handle, which makes sense, but for our purposes we might want to allow unknown methods too?
155
156 - rsky-syntax has an aturi
157 - adenosyne also
158 - might come back to these
159
160
161-------
162
163rocks
164
165```bash
166ROCKSDB_LIB_DIR=/nix/store/z2chn0hsik0clridr8mlprx1cngh1g3c-rocksdb-9.7.3/lib/ cargo build
167```