Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a simple multi-user app that publishes your current "status" as an emoji.
4
5
6
7At various points we will cover how to:
8
9- Signin via OAuth
10- Fetch information about users (profiles)
11- Listen to the network firehose for new data
12- Publish data on the user's account using a custom schema
13
14We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
15
16## Where are we going?
17
18Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
19
20Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
21
22- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
23- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
24- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
25
26The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
27
28> `at://` is the URL scheme of the AT Protocol. Under the hood it uses common tech like HTTP and DNS, but it adds all of the features we'll be using in this tutorial.
29
30## Step 1. Starting with our ExpressJS app
31
32Start by cloning the repo and installing packages.
33
34```bash
35git clone TODO
36cd TODO
37npm i
38npm run dev # you can leave this running and it will auto-reload
39```
40
41Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](https://kysely.dev/).
42
43Our starting stack:
44
45- Typescript
46- NodeJS web server ([express](https://expressjs.com/))
47- SQLite database ([Kysley](https://kysely.dev/))
48- Server-side rendering ([uhtml](https://www.npmjs.com/package/uhtml))
49
50With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
51
52## Step 2. Signing in with OAuth
53
54When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
55
56We're going to accomplish this using OAuth ([spec](https://github.com/bluesky-social/proposals/tree/main/0004-oauth)). Most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](https://github.com/bluesky-social/atproto/tree/main/packages/oauth/oauth-client-node) library. This is the arrangement we're aiming toward:
57
58
59
60When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
61
62
63
64Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
65
66```html
67<!-- src/pages/login.ts -->
68<form action="/login" method="post" class="login-form">
69 <input
70 type="text"
71 name="handle"
72 placeholder="Enter your handle (eg alice.bsky.social)"
73 required
74 />
75 <button type="submit">Log in</button>
76</form>
77```
78
79When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
80
81```typescript
82/** src/routes.ts **/
83// Login handler
84router.post(
85 '/login',
86 handler(async (req, res) => {
87 // Initiate the OAuth flow
88 const url = await oauthClient.authorize(handle)
89 return res.redirect(url.toString())
90 })
91)
92```
93
94This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
95
96When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](https://atproto.com/specs/did) to their cookie-session.
97
98```typescript
99/** src/routes.ts **/
100// OAuth callback to complete session creation
101router.get(
102 '/oauth/callback',
103 handler(async (req, res) => {
104 // Store the credentials
105 const { agent } = await oauthClient.callback(params)
106
107 // Attach the account DID to our user via a cookie
108 const session = await getIronSession(req, res)
109 session.did = agent.accountDid
110 await session.save()
111
112 // Send them back to the app
113 return res.redirect('/')
114 })
115)
116```
117
118With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
119
120## Step 3. Fetching the user's profile
121
122Why don't we learn something about our user? In [Bluesky](https://bsky.app), users publish a "profile" record which looks like this:
123
124```typescript
125interface ProfileRecord {
126 displayName?: string // a human friendly name
127 description?: string // a short bio
128 avatar?: BlobRef // small profile picture
129 banner?: BlobRef // banner image to put on profiles
130 createdAt?: string // declared time this profile data was added
131 // ...
132}
133```
134
135You can examine this record directly using [atproto-browser.vercel.app](https://atproto-browser.vercel.app). For instance, [this is the profile record for @bsky.app](https://atproto-browser.vercel.app/at?u=at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.actor.profile/self).
136
137We're going to use the [Agent](https://github.com/bluesky-social/atproto/tree/main/packages/api) associated with the user's OAuth session to fetch this record.
138
139```typescript
140await agent.getRecord({
141 repo: agent.accountDid, // The user
142 collection: 'app.bsky.actor.profile', // The collection
143 rkey: 'self', // The record key
144})
145```
146
147When asking for a record, we provide three pieces of information.
148
149- **repo** The [DID](https://atproto.com/specs/did) which identifies the user,
150- **collection** The collection name, and
151- **rkey** The record key
152
153We'll explain the collection name shortly. Record keys are strings with [some restrictions](https://atproto.com/specs/record-key#record-key-syntax) and a couple of common patterns. The `"self"` pattern is used when a collection is expected to only contain one record which describes the user.
154
155Let's update our homepage to fetch this profile record:
156
157```typescript
158/** src/routes.ts **/
159// Homepage
160router.get(
161 '/',
162 handler(async (req, res) => {
163 // If the user is signed in, get an agent which communicates with their server
164 const agent = await getSessionAgent(req, res, ctx)
165
166 if (!agent) {
167 // Serve the logged-out view
168 return res.type('html').send(page(home()))
169 }
170
171 // Fetch additional information about the logged-in user
172 const { data: profileRecord } = await agent.getRecord({
173 repo: agent.accountDid, // our user's repo
174 collection: 'app.bsky.actor.profile', // the bluesky profile record type
175 rkey: 'self', // the record's key
176 })
177
178 // Serve the logged-in view
179 return res
180 .type('html')
181 .send(page(home({ profile: profileRecord.value || {} })))
182 })
183)
184```
185
186With that data, we can give a nice personalized welcome banner for our user:
187
188
189
190```html
191<!-- pages/home.ts -->
192<div class="card">
193 ${profile
194 ? html`<form action="/logout" method="post" class="session-form">
195 <div>
196 Hi, <strong>${profile.displayName || 'friend'}</strong>.
197 What's your status today?
198 </div>
199 <div>
200 <button type="submit">Log out</button>
201 </div>
202 </form>`
203 : html`<div class="session-form">
204 <div><a href="/login">Log in</a> to set your status!</div>
205 <div>
206 <a href="/login" class="button">Log in</a>
207 </div>
208 </div>`}
209</div>
210```
211
212## Step 4. Reading & writing records
213
214You can think of the user repositories as collections of JSON records:
215
216
217
218Let's look again at how we read the "profile" record:
219
220```typescript
221await agent.getRecord({
222 repo: agent.accountDid, // The user
223 collection: 'app.bsky.actor.profile', // The collection
224 rkey: 'self', // The record key
225})
226```
227
228We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
229
230```typescript
231// Generate a time-based key for our record
232const rkey = TID.nextStr()
233
234// Write the
235await agent.putRecord({
236 repo: agent.accountDid, // The user
237 collection: 'com.example.status', // The collection
238 rkey, // The record key
239 record: { // The record value
240 status: "👍",
241 createdAt: new Date().toISOString()
242 }
243})
244```
245
246Our `POST /status` route is going to use this API to publish the user's status to their repo.
247
248```typescript
249/** src/routes.ts **/
250// "Set status" handler
251router.post(
252 '/status',
253 handler(async (req, res) => {
254 // If the user is signed in, get an agent which communicates with their server
255 const agent = await getSessionAgent(req, res, ctx)
256 if (!agent) {
257 return res.status(401).type('html').send('<h1>Error: Session required</h1>')
258 }
259
260 // Construct their status record
261 const record = {
262 $type: 'com.example.status',
263 status: req.body?.status,
264 createdAt: new Date().toISOString(),
265 }
266
267 try {
268 // Write the status record to the user's repository
269 await agent.putRecord({
270 repo: agent.accountDid,
271 collection: 'com.example.status',
272 rkey: TID.nextStr(),
273 record,
274 })
275 } catch (err) {
276 logger.warn({ err }, 'failed to write record')
277 return res.status(500).type('html').send('<h1>Error: Failed to write record</h1>')
278 }
279
280 res.status(200).json({})
281 })
282)
283```
284
285Now in our homepage we can list out the status buttons:
286
287```html
288<!-- src/pages/home.ts -->
289<form action="/status" method="post" class="status-options">
290 ${STATUS_OPTIONS.map(status => html`
291 <button class="status-option" name="status" value="${status}">
292 ${status}
293 </button>
294 `)}
295</form>
296```
297
298And here we are!
299
300
301
302## Step 5. Creating a custom "status" schema
303
304Repo collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
305
306Anybody can create a new schema using the [Lexicon](https://atproto.com/specs/lexicon) language, which is very similar to [JSON-Schema](http://json-schema.org/). The schemas use [reverse-DNS IDs](https://atproto.com/specs/nsid) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
307
308> ### Why create a schema?
309>
310> Schemas help other applications understand the data your app is creating. By publishing your schemas, you make it easier for other application authors to publish data in a format your app will recognize and handle.
311
312Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](https://atproto.com/guides/lexicon).
313
314```json
315/** lexicons/status.json **/
316{
317 "lexicon": 1,
318 "id": "com.example.status",
319 "defs": {
320 "main": {
321 "type": "record",
322 "key": "tid",
323 "record": {
324 "type": "object",
325 "required": ["status", "createdAt"],
326 "properties": {
327 "status": {
328 "type": "string",
329 "minLength": 1,
330 "maxGraphemes": 1,
331 "maxLength": 32
332 },
333 "createdAt": {
334 "type": "string",
335 "format": "datetime"
336 }
337 }
338 }
339 }
340 }
341}
342```
343
344Now let's run some code-generation using our schema:
345
346```bash
347./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
348```
349
350This will produce Typescript interfaces as well as runtime validation functions that we can use in our app. Here's what that generated code looks like:
351
352```typescript
353/** src/lexicon/types/com/example/status.ts **/
354export interface Record {
355 status: string
356 createdAt: string
357 [k: string]: unknown
358}
359
360export function isRecord(v: unknown): v is Record {
361 return (
362 isObj(v) &&
363 hasProp(v, '$type') &&
364 (v.$type === 'com.example.status#main' || v.$type === 'com.example.status')
365 )
366}
367
368export function validateRecord(v: unknown): ValidationResult {
369 return lexicons.validate('com.example.status#main', v)
370}
371```
372
373Let's use that code to improve the `POST /status` route:
374
375```typescript
376/** src/routes.ts **/
377import * as Status from '#/lexicon/types/com/example/status'
378// ...
379// "Set status" handler
380router.post(
381 '/status',
382 handler(async (req, res) => {
383 // ...
384
385 // Construct & validate their status record
386 const record = {
387 $type: 'com.example.status',
388 status: req.body?.status,
389 createdAt: new Date().toISOString(),
390 }
391 if (!Status.validateRecord(record).success) {
392 return res.status(400).json({ error: 'Invalid status' })
393 }
394
395 // ...
396 })
397)
398```
399
400## Step 6. Listening to the firehose
401
402So far, we have:
403
404- Logged in via OAuth
405- Created a custom schema
406- Read & written records for the logged in user
407
408Now we want to fetch the status records from other users.
409
410Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
411
412
413
414Using a [Relay service](https://docs.bsky.app/docs/advanced-guides/federation-architecture#relay) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
415
416
417```typescript
418/** src/firehose.ts **/
419import * as Status from '#/lexicon/types/com/example/status'
420// ...
421const firehose = new Firehose({})
422
423for await (const evt of firehose.run()) {
424 // Watch for write events
425 if (evt.event === 'create' || evt.event === 'update') {
426 const record = evt.record
427
428 // If the write is a valid status update
429 if (
430 evt.collection === 'com.example.status' &&
431 Status.isRecord(record) &&
432 Status.validateRecord(record).success
433 ) {
434 // Store the status
435 // TODO
436 }
437 }
438}
439```
440
441Let's create a SQLite table to store these statuses:
442
443```typescript
444/** src/db.ts **/
445// Create our statuses table
446await db.schema
447 .createTable('status')
448 .addColumn('uri', 'varchar', (col) => col.primaryKey())
449 .addColumn('authorDid', 'varchar', (col) => col.notNull())
450 .addColumn('status', 'varchar', (col) => col.notNull())
451 .addColumn('createdAt', 'varchar', (col) => col.notNull())
452 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
453 .execute()
454```
455
456Now we can write these statuses into our database as they arrive from the firehose:
457
458```typescript
459/** src/firehose.ts **/
460// If the write is a valid status update
461if (
462 evt.collection === 'com.example.status' &&
463 Status.isRecord(record) &&
464 Status.validateRecord(record).success
465) {
466 // Store the status in our SQLite
467 await db
468 .insertInto('status')
469 .values({
470 uri: evt.uri.toString(),
471 authorDid: evt.author,
472 status: record.status,
473 createdAt: record.createdAt,
474 indexedAt: new Date().toISOString(),
475 })
476 .onConflict((oc) =>
477 oc.column('uri').doUpdateSet({
478 status: record.status,
479 indexedAt: new Date().toISOString(),
480 })
481 )
482 .execute()
483}
484```
485
486You can almost think of information flowing in a loop:
487
488
489
490Applications write to the repo. The write events are then emitted on the firehose where they're caught by the apps and ingested into their databases.
491
492Why sync from the event log like this? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in — including data published by other apps!
493
494## Step 7. Listing the latest statuses
495
496Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](https://atproto.com/specs/did)-to-handle resolver so we can show a nice username with the statuses:
497
498```typescript
499/** src/routes.ts **/
500// Homepage
501router.get(
502 '/',
503 handler(async (req, res) => {
504 // ...
505
506 // Fetch data stored in our SQLite
507 const statuses = await db
508 .selectFrom('status')
509 .selectAll()
510 .orderBy('indexedAt', 'desc')
511 .limit(10)
512 .execute()
513
514 // Map user DIDs to their domain-name handles
515 const didHandleMap = await resolver.resolveDidsToHandles(
516 statuses.map((s) => s.authorDid)
517 )
518
519 // ...
520 })
521)
522```
523
524Our HTML can now list these status records:
525
526```html
527<!-- src/pages/home.ts -->
528${statuses.map((status, i) => {
529 const handle = didHandleMap[status.authorDid] || status.authorDid
530 return html`
531 <div class="status-line">
532 <div>
533 <div class="status">${status.status}</div>
534 </div>
535 <div class="desc">
536 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
537 was feeling ${status.status} on ${status.indexedAt}.
538 </div>
539 </div>
540 `
541})}
542```
543
544
545
546## Step 8. Optimistic updates
547
548As a final optimization, let's introduce "optimistic updates."
549
550Remember the information flow loop with the repo write and the event log?
551
552
553
554Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
555
556
557
558This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
559
560To do this, we just update `POST /status` to include an additional write to our SQLite DB:
561
562```typescript
563/** src/routes.ts **/
564// "Set status" handler
565router.post(
566 '/status',
567 handler(async (req, res) => {
568 // ...
569
570 let uri
571 try {
572 // Write the status record to the user's repository
573 const res = await agent.putRecord({
574 repo: agent.accountDid,
575 collection: 'com.example.status',
576 rkey: TID.nextStr(),
577 record,
578 })
579 uri = res.uri
580 } catch (err) {
581 logger.warn({ err }, 'failed to write record')
582 return res.status(500).json({ error: 'Failed to write record' })
583 }
584
585 try {
586 // Optimistically update our SQLite <-- HERE!
587 await db
588 .insertInto('status')
589 .values({
590 uri,
591 authorDid: agent.accountDid,
592 status: record.status,
593 createdAt: record.createdAt,
594 indexedAt: new Date().toISOString(),
595 })
596 .execute()
597 } catch (err) {
598 logger.warn(
599 { err },
600 'failed to update computed view; ignoring as it should be caught by the firehose'
601 )
602 }
603
604 res.status(200).json({})
605 })
606)
607```
608
609You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
610
611## Thinking in AT Proto
612
613In this tutorial we've covered the key steps to building an atproto app. Data is published in its canonical form on users' `at://` repos and then aggregated into apps' databases to produce views of the network.
614
615When building your app, think in these four key steps:
616
617- Design the [Lexicon](#) schemas for the records you'll publish into the Atmosphere.
618- Create a database for aggregating the records into useful views.
619- Build your application to write the records on your users' repos.
620- Listen to the firehose to aggregate data across the network.
621
622Remember this flow of information throughout:
623
624
625
626This is how every app in the Atmosphere works, including the [Bluesky social app](https://bsky.app).
627
628## Next steps
629
630If you want to practice what you've learned, here are some additional challenges you could try:
631
632- Sync the profile records of all users so that you can show their display names instead of their handles.
633- Count the number of each status used and display the total counts.
634- Fetch the authed user's `app.bsky.graph.follow` follows and show statuses from them.
635- Create a different kind of schema, like a way to post links to websites and rate them 1 through 4 stars.
636
637You can find more information here:
638
639|Resources|-|
640|-|-|
641|[＠ ATProto docs](https://atproto.com)|Learn more about the AT Protocol.|
642|[🦋 Bluesky API docs](https://docs.bsky.app/)|See how Bluesky works as an ATProto app.|
643|[📦 ATProto monorepo](https://github.com/bluesky-social/atproto)|See the source code first-hand.|
644|[💬 ATProto discussions board](https://github.com/bluesky-social/atproto/discussions)|Ask any questions you have!|