nekomimi.pet
forked from
microcosm.blue/microcosm-rs
Constellation, Spacedust, Slingshot, UFOs: atproto crates and services for microcosm
1---
2
3
4old notes follow, ignore
5------------------------
6
7
8as far as i can tell, atproto lexicons today don't follow much of a convention for referencing across documents: sometimes it's a StrongRef, sometimes it's a DID, sometimes it's a bare at-uri. lexicon authors choose any old link-sounding key name for the key in their document.
9
10it's pretty messy so embrace the mess: atproto wants to be part of the web, so this library will also extract URLs and other URIs if you want it to. all the links.
11
12
13why
14---
15
16the atproto firehose that bluesky sprays at you will contain raw _contents_ from peoples' pdses. these are isolated, decontextualized updates. it's very easy to build some kinds of interesting downstream apps off of this feed.
17
18- bluesky posts (firesky, deletions, )
19- blueksy post stats (emojis, )
20- trending keywords ()
21
22but bringing almost kind of _context_ into your project requires a big step up in complexity and potentially cost: you're entering "appview" territory. _how many likes does a post have? who follows this account?_
23
24you own your atproto data: it's kept in your personal data repository (PDS) and noone else can write to it. when someone likes your post, they create a "like" record in their _own_ pds, and that like belongs to _them_, not to you/your post.
25
26in the firehose you'll see a `app.bsky.feed.post` record created, with no details about who has liked it. then you'll see separate `app.bsky.feed.like` records show up for each like that comes in on that post, with no context about the post except a random-looking reference to it. storing these in order to do so is up to you!
27
28**so, why**
29
30everything is links, and they're a mess, but they all kinda work the same, so maybe some tooling can bring down that big step in complexity from firehose raw-content apps -> apps requiring any social context.
31
32everything is links:
33
34- likes
35- follows
36- blocks
37- reposts
38- quotes
39
40some low-level things you could make from links:
41
42- notification streams (part of ucosm)
43- a global reverse index (part of ucosm)
44
45i think that making these low-level services as easy to use as jetstream could open up pathways for building more atproto apps that operate at full scale with interesting features for reasonable effort at low cost to operate.
46
47
48extracting links
49---------------
50
51
52- low-level: pass a &str of a field value and get a parsed link back
53
54- med-level: pass a &str of record in json form and get a list of parsed links + json paths back. (todo: should also handle dag-cbor prob?)
55
56- high-ish level: pass the json record and maybe apply some pre-loaded rules based on known lexicons to get the best result.
57
58for now, a link is only considered if it matches for the entire value of the record's field -- links embedded in text content are not included. note that urls in bluesky posts _will_ still be extracted, since they are broken out into facets.
59
60
61resolving / canonicalizing links
62--------------------------------
63
64
65### at-uris
66
67every at-uri has at least two equivalent forms, one with a `DID`, and one with an account handle. the at-uri spec [illustrates this by example](https://atproto.com/specs/at-uri-scheme):
68
69- `at://did:plc:44ybard66vv44zksje25o7dz/app.bsky.feed.post/3jwdwj2ctlk26`
70- `at://bnewbold.bsky.team/app.bsky.feed.post/3jwdwj2ctlk26`
71
72some applications, like a reverse link index, may wish to canonicalize at-uris to a single form. the `DID`-form is stable as an account changes its handle and probably the right choice to canonicalize to, but maybe some apps would actually perfer to canonicalise to handles?
73
74hopefully atrium will make it easy to resolve at-uris.
75
76
77### urls
78
79canonicalizing URLs is more annoying but also a bit more established. lots of details.
80
81- do we have to deal with punycode?
82- follow redirects (todo: only permanent ones, or all?)
83- check for rel=canonical http header and possibly follow it
84- check link rel=canonical meta tag and possibly follow it
85- do we need to check site maps??
86- do we have to care at all about AMP?
87- do we want anything to do with url shorteners??
88- how do multilingual sites affect this?
89- do we have to care about `script type="application/ld+json"` ???
90
91ugh. is there a crate for this.
92
93
94### relative uris?
95
96links might be relative, in which case they might need to be made absolute before being useful. is that a concern for this library, or up to the user? (seems like we might not have context here to determine its absolute)
97
98
99### canonicalizing
100
101there should be a few async functions available to canonicalize already-parsed links.
102
103- what happens if a link can't be resolved?
104
105
106---
107
108- using `tinyjson` because it's nice -- maybe should switch to serde_json to share deps with atrium?
109
110- would use atrium for parsing at-uris, but it's not in there. there's a did-only version in the non-lib commands.rs. its identifier parser is strict to did + handle, which makes sense, but for our purposes we might want to allow unknown methods too?
111
112 - rsky-syntax has an aturi
113 - adenosyne also
114 - might come back to these
115
116
117-------
118
119rocks
120
121```bash
122ROCKSDB_LIB_DIR=/nix/store/z2chn0hsik0clridr8mlprx1cngh1g3c-rocksdb-9.7.3/lib/ cargo build
123```