commit 1d646ef2251dade1cc50379af6c035b6956405cd · nekomimi.pet/microcosm-rs

+1 -1

slingshot/Cargo.toml

···

       18
        
       metrics = "0.24.2"

     

       19
        
       metrics-exporter-prometheus = { version = "0.17.1", features = ["http-listener"] }

     

       20
        
       poem = { version = "3.1.12", features = ["acme"] }

     

       21
       -
       poem-openapi = { version = "5.1.16", features = ["scalar", "stoplight-elements"] }

     

       22
        
       reqwest = { version = "0.12.22", features = ["json"] }

     

       23
        
       rustls = "0.23.31"

     

       24
        
       serde = { version = "1.0.219", features = ["derive"] }

···

       18
        
       metrics = "0.24.2"

     

       19
        
       metrics-exporter-prometheus = { version = "0.17.1", features = ["http-listener"] }

     

       20
        
       poem = { version = "3.1.12", features = ["acme"] }

     

       21
       +
       poem-openapi = { version = "5.1.16", features = ["scalar"] }

     

       22
        
       reqwest = { version = "0.12.22", features = ["json"] }

     

       23
        
       rustls = "0.23.31"

     

       24
        
       serde = { version = "1.0.219", features = ["derive"] }

+73

slingshot/api-description.md

···

       1
       +
       _A [gravitational slingshot](https://en.wikipedia.org/wiki/Gravity_assist) makes use of the gravity and relative movements of celestial bodies to accelerate a spacecraft and change its trajectory._

     

       2
       +
       

     

       3
       +
       

     

       4
       +
       # Slingshot: edge record cache

     

       5
       +
       

     

       6
       +
       Applications in [ATProtocol](https://atproto.com/) store data in users' own [PDS](https://atproto.com/guides/self-hosting) (Personal Data Server), which are distributed across thousands of independently-run servers all over the world. Trying to access this data poses challenges for client applications:

     

       7
       +
       

     

       8
       +
       - A PDS might be far away with long network latency

     

       9
       +
       - or may be on an unreliable connection

     

       10
       +
       - or overloaded when you need it, or offline, or…

     

       11
       +
       

     

       12
       +
       Large projects like [Bluesky](https://bsky.app/) control their performance and reliability by syncing all app-relevant data from PDSs into first-party databases. But for new apps, building out this additional data infrastructure adds significant effort and complexity up front.

     

       13
       +
       

     

       14
       +
       **Slingshot is a fast, eager, production-grade cache of data in the [ATmosphere](https://atproto.com/)**, offering performance and reliability without custom infrastructure.

     

       15
       +
       

     

       16
       +
       

     

       17
       +
       ### Current status

     

       18
       +
       

     

       19
       +
       Slingshot is currently in a **v0, pre-release state**. There is one production instance and you can use it! Expect short downtimes for restarts as development progresses and lower cache hit-rates as the internal storage caches are adjusted and reset.

     

       20
       +
       

     

       21
       +
       The core APIs will not change, since they are standard third-party `com.atproto` query APIs from ATProtocol.

     

       22
       +
       

     

       23
       +
       

     

       24
       +
       ## Eager caching

     

       25
       +
       

     

       26
       +
       In many cases, Slingshot can cache the data you need *before* first request!

     

       27
       +
       

     

       28
       +
       Slingshot subscribes to the global [Firehose](https://atproto.com/specs/sync#firehose) of data updates. It keeps a short-term rolling indexed window of *all* data, and automatically promotes content likely to be requested to its longer-term main cache. _(automatic promotion is still a work in progress)_

     

       29
       +
       

     

       30
       +
       When there is a cache miss, Slingshot can often still accelerate record fetching, since it keeps a large cache of resolved identities: it can usually request from the correct PDS without extra lookups.

     

       31
       +
       

     

       32
       +
       

     

       33
       +
       ## Precise invalidation

     

       34
       +
       

     

       35
       +
       The fireshose includes **update** and **delete** events, which Slingshot uses to ensure stale and deleted data is removed within a very short window. Additonally, identity and account-level events can trigger rapid cleanup of data for deactivated and deleted accounts. _(some of this is still a work in progress)_

     

       36
       +
       

     

       37
       +
       

     

       38
       +
       ## Low-trust

     

       39
       +
       

     

       40
       +
       The "AT" in ATProtocol [stands for _Authenticated Transfer_](https://atproto.com/guides/glossary#at-protocol): all data is cryptographically signed, which makes it possible to broadcast data through third parties and trust that it's real _without_ having to directly contact the originating server.

     

       41
       +
       

     

       42
       +
       Two core standard query APIs are supported to balance convenience and trust. They both fetch [records](https://atproto.com/guides/glossary#record):

     

       43
       +
       

     

       44
       +
       ### [`com.atproto.repo.getRecord`](#tag/comatproto-queries/GET/xrpc/com.atproto.repo.getRecord)

     

       45
       +
       

     

       46
       +
       - convenient `JSON` response format

     

       47
       +
       - cannot be proven authentic

     

       48
       +
       

     

       49
       +
       ### [`com.atproto.sync.getRecord`](#tag/comatproto-queries/GET/xrpc/com.atproto.sync.getRecord)

     

       50
       +
       

     

       51
       +
       - [`DAG-CBOR`](https://atproto.com/specs/data-model)-encoded response requires extra libraries to decode, but

     

       52
       +
       - includes a cryptographic proof of authenticity!

     

       53
       +
       

     

       54
       +
       _(work on this endpoint is in progress)_

     

       55
       +
       

     

       56
       +
       

     

       57
       +
       ## Ergonomic APIs

     

       58
       +
       

     

       59
       +
       - Slingshot also offers variants of the `getRecord` endpoints that accept a full `at-uri` as a parameter, to save clients from needing to parse and validate all parts of a record location.

     

       60
       +
       

     

       61
       +
       - Bi-directionally verifying identity endpoints, so you can directly exchange atproto [`handle`](https://atproto.com/guides/glossary#handle)s for [`DID`](https://atproto.com/guides/glossary#did-decentralized-id)s without extra steps, plus a convenient [Mini-Doc](#tag/slingshot-specific-queries/GET/xrpc/com.bad-example.identity.resolveMiniDoc) verified identity summary.

     

       62
       +
       

     

       63
       +
       

     

       64
       +
       ## Part of microcosm

     

       65
       +
       

     

       66
       +
       [Microcosm](https://www.microcosm.blue/) is a collection of services and independent community-run infrastructure for ATProtocol.

     

       67
       +
       

     

       68
       +
       Slingshot excels when combined with _shallow indexing_ services, which offer fast queries of global data relationships but with only references to the data records. Microcosm has a few!

     

       69
       +
       

     

       70
       +
       - [🌌 Constellation](https://constellation.microcosm.blue/), a global backlink index (all social interactions in atproto are links!)

     

       71
       +
       - [🎇 Spacedust](https://spacedust.microcosm.blue/), a firehose of all social interactions

     

       72
       +
       

     

       73
       +
       All microcosm projects are [open source](https://tangled.sh/@bad-example.com/microcosm-links). **You can help sustain Slingshot** and all of microcosm by becoming a [Github sponsor](https://github.com/sponsors/uniphil/) or a [Ko-fi supporter](https://ko-fi.com/bad_example)!

+56 -7

slingshot/src/server.rs

···

       22
        
           middleware::{Cors, Tracing},

     

       23
        
       };

     

       24
        
       use poem_openapi::{

     

       25
       -
           ApiResponse, Object, OpenApi, OpenApiService, param::Query, payload::Json, types::Example,

     

       0
        
       
     

       26
        
       };

     

       27
        
       

     

       28
        
       fn example_handle() -> String {

     
···

       187
        
           repo: Arc<Repo>,

     

       188
        
       }

     

       189
        
       

     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       190
        
       #[OpenApi]

     

       191
        
       impl Xrpc {

     

       192
        
           /// com.atproto.repo.getRecord

     
···

       195
        
           ///

     

       196
        
           /// See also the [canonical `com.atproto` XRPC documentation](https://docs.bsky.app/docs/api/com-atproto-repo-get-record)

     

       197
        
           /// that this endpoint aims to be compatible with.

     

       198
       -
           #[oai(path = "/com.atproto.repo.getRecord", method = "get")]

     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       199
        
           async fn get_record(

     

       200
        
               &self,

     

       201
        
               /// The DID or handle of the repo

     
···

       222
        
           /// com.bad-example.repo.getUriRecord

     

       223
        
           ///

     

       224
        
           /// Ergonomic complement to [`com.atproto.repo.getRecord`](https://docs.bsky.app/docs/api/com-atproto-repo-get-record)

     

       225
       -
           /// which accepts an at-uri instead of individual repo/collection/rkey params

     

       226
       -
           #[oai(path = "/com.bad-example.repo.getUriRecord", method = "get")]

     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       227
        
           async fn get_uri_record(

     

       228
        
               &self,

     

       229
        
               /// The at-uri of the record

     
···

       281
        
           ///

     

       282
        
           /// Like [com.atproto.identity.resolveIdentity](https://docs.bsky.app/docs/api/com-atproto-identity-resolve-identity)

     

       283
        
           /// but instead of the full `didDoc` it returns an atproto-relevant subset.

     

       284
       -
           #[oai(path = "/com.bad-example.identity.resolveMiniDoc", method = "get")]

     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       285
        
           async fn resolve_mini_id(

     

       286
        
               &self,

     

       287
        
               /// Handle or DID to resolve

     
···

       562
        
           } else {

     

       563
        
               "http://localhost:3000".to_string()

     

       564
        
           })

     

       565
       -
           .url_prefix("/xrpc");

     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       0
        
       
     

       566
        
       

     

       567
        
           let mut app = Route::new()

     

       568
        
               .nest("/", api_service.scalar())

     

       569
       -
               .nest("/se", api_service.stoplight_elements())

     

       570
        
               .nest("/openapi.json", api_service.spec_endpoint())

     

       571
        
               .nest("/xrpc/", api_service);

     

       572

···

       22
        
           middleware::{Cors, Tracing},

     

       23
        
       };

     

       24
        
       use poem_openapi::{

     

       25
       +
           ApiResponse, ContactObject, ExternalDocumentObject, Object, OpenApi, OpenApiService, Tags,

     

       26
       +
           param::Query, payload::Json, types::Example,

     

       27
        
       };

     

       28
        
       

     

       29
        
       fn example_handle() -> String {

     
···

       188
        
           repo: Arc<Repo>,

     

       189
        
       }

     

       190
        
       

     

       191
       +
       #[derive(Tags)]

     

       192
       +
       enum ApiTags {

     

       193
       +
           /// Core ATProtocol-compatible APIs.

     

       194
       +
           ///

     

       195
       +
           /// Upstream documentation is available at

     

       196
       +
           /// https://docs.bsky.app/docs/category/http-reference

     

       197
       +
           ///

     

       198
       +
           /// These queries are usually executed directly against the PDS containing

     

       199
       +
           /// the data being requested. Slingshot offers a caching view of the same

     

       200
       +
           /// contents with better expected performance and reliability.

     

       201
       +
           #[oai(rename = "com.atproto.* queries")]

     

       202
       +
           ComAtproto,

     

       203
       +
           /// Additional and improved APIs.

     

       204
       +
           ///

     

       205
       +
           /// These APIs offer small tweaks to the core ATProtocol APIs, with more

     

       206
       +
           /// more convenient [request parameters](#tag/slingshot-specific-queries/GET/xrpc/com.bad-example.repo.getUriRecord)

     

       207
       +
           /// or [response formats](#tag/slingshot-specific-queries/GET/xrpc/com.bad-example.identity.resolveMiniDoc).

     

       208
       +
           ///

     

       209
       +
           /// At the moment, these are namespaced under the `com.bad-example.*` NSID

     

       210
       +
           /// prefix, but as they stabilize they will likely be moved to either

     

       211
       +
           /// `blue.microcosm.*` or a slingshot-instance-specific lexicon under its

     

       212
       +
           /// `did:web` (ie., `blue.microcosm.slingshot.*`). Maybe one day they can

     

       213
       +
           /// be promoted to the [Lexicon Community](https://discourse.lexicon.community/)

     

       214
       +
           /// namespace.

     

       215
       +
           #[oai(rename = "slingshot-specific queries")]

     

       216
       +
           Custom,

     

       217
       +
       }

     

       218
       +
       

     

       219
        
       #[OpenApi]

     

       220
        
       impl Xrpc {

     

       221
        
           /// com.atproto.repo.getRecord

     
···

       224
        
           ///

     

       225
        
           /// See also the [canonical `com.atproto` XRPC documentation](https://docs.bsky.app/docs/api/com-atproto-repo-get-record)

     

       226
        
           /// that this endpoint aims to be compatible with.

     

       227
       +
           #[oai(

     

       228
       +
               path = "/com.atproto.repo.getRecord",

     

       229
       +
               method = "get",

     

       230
       +
               tag = "ApiTags::ComAtproto"

     

       231
       +
           )]

     

       232
        
           async fn get_record(

     

       233
        
               &self,

     

       234
        
               /// The DID or handle of the repo

     
···

       255
        
           /// com.bad-example.repo.getUriRecord

     

       256
        
           ///

     

       257
        
           /// Ergonomic complement to [`com.atproto.repo.getRecord`](https://docs.bsky.app/docs/api/com-atproto-repo-get-record)

     

       258
       +
           /// which accepts an `at-uri` instead of individual repo/collection/rkey params

     

       259
       +
           #[oai(

     

       260
       +
               path = "/com.bad-example.repo.getUriRecord",

     

       261
       +
               method = "get",

     

       262
       +
               tag = "ApiTags::Custom"

     

       263
       +
           )]

     

       264
        
           async fn get_uri_record(

     

       265
        
               &self,

     

       266
        
               /// The at-uri of the record

     
···

       318
        
           ///

     

       319
        
           /// Like [com.atproto.identity.resolveIdentity](https://docs.bsky.app/docs/api/com-atproto-identity-resolve-identity)

     

       320
        
           /// but instead of the full `didDoc` it returns an atproto-relevant subset.

     

       321
       +
           #[oai(

     

       322
       +
               path = "/com.bad-example.identity.resolveMiniDoc",

     

       323
       +
               method = "get",

     

       324
       +
               tag = "ApiTags::Custom"

     

       325
       +
           )]

     

       326
        
           async fn resolve_mini_id(

     

       327
        
               &self,

     

       328
        
               /// Handle or DID to resolve

     
···

       603
        
           } else {

     

       604
        
               "http://localhost:3000".to_string()

     

       605
        
           })

     

       606
       +
           .url_prefix("/xrpc")

     

       607
       +
           .contact(

     

       608
       +
               ContactObject::new()

     

       609
       +
                   .name("@microcosm.blue")

     

       610
       +
                   .url("https://bsky.app/profile/microcosm.blue"),

     

       611
       +
           )

     

       612
       +
           .description(include_str!("../api-description.md"))

     

       613
       +
           .external_document(ExternalDocumentObject::new(

     

       614
       +
               "https://microcosm.blue/slingshot",

     

       615
       +
           ));

     

       616
        
       

     

       617
        
           let mut app = Route::new()

     

       618
        
               .nest("/", api_service.scalar())

     

       0
        
       
     

       619
        
               .nest("/openapi.json", api_service.spec_endpoint())

     

       620
        
               .nest("/xrpc/", api_service);

     

       621