Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a **simple multi-user app** that publishes your current "status" as an emoji.
4
5
6
7At various points we will cover how to:
8
9- Signin via OAuth
10- Fetch information about users (profiles)
11- Listen to the network firehose for new data
12- Publish data on the user's account using a custom schema
13
14We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
15
16## Where are we going?
17
18Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
19
20Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
21
22- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
23- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
24- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
25
26The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
27
28> `at://` is the URL scheme of the AT Protocol. Under the hood it uses common tech like HTTP and DNS, but it adds all of the features we'll be using in this tutorial.
29
30## Step 1. Starting with our ExpressJS app
31
32Start by cloning the repo and installing packages.
33
34```bash
35git clone TODO
36cd TODO
37npm i
38npm run dev # you can leave this running and it will auto-reload
39```
40
41Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](#todo).
42
43Our starting stack:
44
45- Typescript
46- NodeJS web server ([express](#todo))
47- SQLite database ([Kysley](#todo))
48- Server-side rendering ([uhtml](#todo))
49
50With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
51
52## Step 2. Signing in with OAuth
53
54When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
55
56We're going to accomplish this using OAuth ([spec](#todo)). You can find a [more extensive OAuth guide here](#todo), but for now just know that most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](#todo) library. This is the arrangement we're aiming toward:
57
58
59
60When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
61
62
63
64Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
65
66```html
67<!-- src/pages/login.ts -->
68<form action="/login" method="post" class="login-form">
69 <input
70 type="text"
71 name="handle"
72 placeholder="Enter your handle (eg alice.bsky.social)"
73 required
74 />
75 <button type="submit">Log in</button>
76</form>
77```
78
79When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
80
81```typescript
82/** src/routes.ts **/
83// Login handler
84router.post(
85 '/login',
86 handler(async (req, res) => {
87 // Initiate the OAuth flow
88 const url = await oauthClient.authorize(handle)
89 return res.redirect(url.toString())
90 })
91)
92```
93
94This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
95
96When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](#todo) to their cookie-session.
97
98```typescript
99/** src/routes.ts **/
100// OAuth callback to complete session creation
101router.get(
102 '/oauth/callback',
103 handler(async (req, res) => {
104 // Store the credentials
105 const { agent } = await oauthClient.callback(params)
106
107 // Attach the account DID to our user via a cookie
108 const session = await getIronSession(req, res)
109 session.did = agent.accountDid
110 await session.save()
111
112 // Send them back to the app
113 return res.redirect('/')
114 })
115)
116```
117
118With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
119
120## Step 3. Fetching the user's profile
121
122Why don't we learn something about our user? Let's start by getting the [Agent](#todo) object. The [Agent](#todo) is the client to the user's `at://` repo server.
123
124```typescript
125/** src/routes.ts **/
126async function getSessionAgent(
127 req: IncomingMessage,
128 res: ServerResponse<IncomingMessage>,
129 ctx: AppContext
130) {
131 // Fetch the session from their cookie
132 const session = await getIronSession(req, res)
133 if (!session.did) return null
134
135 // "Restore" the agent for the user
136 try {
137 return await ctx.oauthClient.restore(session.did)
138 } catch(err) {
139 ctx.logger.warn({ err }, 'oauth restore failed')
140 await session.destroy()
141 return null
142 }
143}
144```
145
146Users publish JSON records on their `at://` repos. In [Bluesky](https://bsky.app), they publish a "profile" record which looks like this:
147
148```typescript
149interface ProfileRecord {
150 displayName?: string // a human friendly name
151 description?: string // a short bio
152 avatar?: BlobRef // small profile picture
153 banner?: BlobRef // banner image to put on profiles
154 createdAt?: string // declared time this profile data was added
155 // ...
156}
157```
158
159We're going to use the [Agent](#todo) to fetch this record to include in our app.
160
161```typescript
162await agent.getRecord({
163 repo: agent.accountDid, // The user
164 collection: 'app.bsky.actor.profile', // The collection
165 rkey: 'self', // The record key
166})
167```
168
169When asking for a record, we provide three pieces of information.
170
171- The [DID](#todo) which identifies the user,
172- The collection name, and
173- The record key
174
175We'll explain the collection name shortly. Record keys are strings with [some limitations](https://atproto.com/specs/record-key#record-key-syntax) and a couple of common patterns. The `"self"` pattern is used when a collection is expected to only contain one record which describes the user.
176
177Let's update our homepage to fetch this profile record:
178
179```typescript
180/** src/routes.ts **/
181// Homepage
182router.get(
183 '/',
184 handler(async (req, res) => {
185 // If the user is signed in, get an agent which communicates with their server
186 const agent = await getSessionAgent(req, res, ctx)
187
188 if (!agent) {
189 // Serve the logged-out view
190 return res.type('html').send(page(home()))
191 }
192
193 // Fetch additional information about the logged-in user
194 const { data: profileRecord } = await agent.getRecord({
195 repo: agent.accountDid, // our user's repo
196 collection: 'app.bsky.actor.profile', // the bluesky profile record type
197 rkey: 'self', // the record's name
198 })
199
200 // Serve the logged-in view
201 return res
202 .type('html')
203 .send(page(home({ profile: profileRecord.value || {} })))
204 })
205)
206```
207
208With that data, we can give a nice personalized welcome banner for our user:
209
210```html
211<!-- pages/home.ts -->
212<div class="card">
213 ${profile
214 ? html`<form action="/logout" method="post" class="session-form">
215 <div>
216 Hi, <strong>${profile.displayName || 'friend'}</strong>.
217 What's your status today?
218 </div>
219 <div>
220 <button type="submit">Log out</button>
221 </div>
222 </form>`
223 : html`<div class="session-form">
224 <div><a href="/login">Log in</a> to set your status!</div>
225 <div>
226 <a href="/login" class="button">Log in</a>
227 </div>
228 </div>`}
229</div>
230```
231
232
233
234You can examine this record directly using [atproto-browser.vercel.app](https://atproto-browser.vercel.app). For instance, [this is the profile record for @bsky.app](https://atproto-browser.vercel.app/at?u=at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.actor.profile/self).
235
236## Step 4. Reading & writing records
237
238You can think of the user repositories as collections of JSON records:
239
240
241
242Let's look again at how we read the "profile" record:
243
244```typescript
245await agent.getRecord({
246 repo: agent.accountDid, // The user
247 collection: 'app.bsky.actor.profile', // The collection
248 rkey: 'self', // The record key
249})
250```
251
252We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
253
254```typescript
255// Generate a time-based key for our record
256const rkey = TID.nextStr()
257
258// Write the
259await agent.putRecord({
260 repo: agent.accountDid, // The user
261 collection: 'com.example.status', // The collection
262 rkey, // The record key
263 record: { // The record value
264 status: "👍",
265 createdAt: new Date().toISOString()
266 }
267})
268```
269
270Our `POST /status` route is going to use this API to publish the user's status to their repo.
271
272```typescript
273/** src/routes.ts **/
274// "Set status" handler
275router.post(
276 '/status',
277 handler(async (req, res) => {
278 // If the user is signed in, get an agent which communicates with their server
279 const agent = await getSessionAgent(req, res, ctx)
280 if (!agent) {
281 return res.status(401).type('html').send('<h1>Error: Session required</h1>')
282 }
283
284 // Construct their status record
285 const record = {
286 $type: 'com.example.status',
287 status: req.body?.status,
288 createdAt: new Date().toISOString(),
289 }
290
291 try {
292 // Write the status record to the user's repository
293 await agent.putRecord({
294 repo: agent.accountDid,
295 collection: 'com.example.status',
296 rkey: TID.nextStr(),
297 record,
298 })
299 } catch (err) {
300 logger.warn({ err }, 'failed to write record')
301 return res.status(500).type('html').send('<h1>Error: Failed to write record</h1>')
302 }
303
304 res.status(200).json({})
305 })
306)
307```
308
309Now in our homepage we can list out the status buttons:
310
311```html
312<!-- src/pages/home.ts -->
313<form action="/status" method="post" class="status-options">
314 ${STATUS_OPTIONS.map(status => html`
315 <button class="status-option" name="status" value="${status}">
316 ${status}
317 </button>
318 `)}
319</form>
320```
321
322And here we are!
323
324
325
326## Step 5. Creating a custom "status" schema
327
328The collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
329
330Anybody can create a new schema using the [Lexicon](#todo) language, which is very similar to [JSON-Schema](#todo). The schemas use [reverse-DNS IDs](#todo) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
331
332> ### Why create a schema?
333>
334> Schemas help other applications understand the data your app is creating. By publishing your schemas, you make it easier for other application authors to publish data in a format your app will recognize and handle.
335
336Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](#todo).
337
338```json
339/** lexicons/status.json **/
340{
341 "lexicon": 1,
342 "id": "com.example.status",
343 "defs": {
344 "main": {
345 "type": "record",
346 "key": "tid",
347 "record": {
348 "type": "object",
349 "required": ["status", "createdAt"],
350 "properties": {
351 "status": {
352 "type": "string",
353 "minLength": 1,
354 "maxGraphemes": 1,
355 "maxLength": 32
356 },
357 "createdAt": {
358 "type": "string",
359 "format": "datetime"
360 }
361 }
362 }
363 }
364 }
365}
366```
367
368Now let's run some code-generation using our schema:
369
370```bash
371./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
372```
373
374This will produce Typescript interfaces as well as runtime validation functions that we can use in our app. Here's what that generated code looks like:
375
376```typescript
377/** src/lexicon/types/com/example/status.ts **/
378export interface Record {
379 status: string
380 createdAt: string
381 [k: string]: unknown
382}
383
384export function isRecord(v: unknown): v is Record {
385 return (
386 isObj(v) &&
387 hasProp(v, '$type') &&
388 (v.$type === 'com.example.status#main' || v.$type === 'com.example.status')
389 )
390}
391
392export function validateRecord(v: unknown): ValidationResult {
393 return lexicons.validate('com.example.status#main', v)
394}
395```
396
397Let's use that code to improve the `POST /status` route:
398
399```typescript
400/** src/routes.ts **/
401import * as Status from '#/lexicon/types/com/example/status'
402// ...
403// "Set status" handler
404router.post(
405 '/status',
406 handler(async (req, res) => {
407 // ...
408
409 // Construct & validate their status record
410 const record = {
411 $type: 'com.example.status',
412 status: req.body?.status,
413 createdAt: new Date().toISOString(),
414 }
415 if (!Status.validateRecord(record).success) {
416 return res.status(400).json({ error: 'Invalid status' })
417 }
418
419 // ...
420 })
421)
422```
423
424## Step 6. Listening to the firehose
425
426So far, we have:
427
428- Logged in via OAuth
429- Created a custom schema
430- Read & written records for the logged in user
431
432Now we want to fetch the status records from other users.
433
434Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
435
436
437
438Using a [Relay service](#todo) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
439
440
441```typescript
442/** src/firehose.ts **/
443import * as Status from '#/lexicon/types/com/example/status'
444// ...
445const firehose = new Firehose({})
446
447for await (const evt of firehose.run()) {
448 // Watch for write events
449 if (evt.event === 'create' || evt.event === 'update') {
450 const record = evt.record
451
452 // If the write is a valid status update
453 if (
454 evt.collection === 'com.example.status' &&
455 Status.isRecord(record) &&
456 Status.validateRecord(record).success
457 ) {
458 // Store the status
459 // TODO
460 }
461 }
462}
463```
464
465Let's create a SQLite table to store these statuses:
466
467```typescript
468/** src/db.ts **/
469// Create our statuses table
470await db.schema
471 .createTable('status')
472 .addColumn('uri', 'varchar', (col) => col.primaryKey())
473 .addColumn('authorDid', 'varchar', (col) => col.notNull())
474 .addColumn('status', 'varchar', (col) => col.notNull())
475 .addColumn('createdAt', 'varchar', (col) => col.notNull())
476 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
477 .execute()
478```
479
480Now we can write these statuses into our database as they arrive from the firehose:
481
482```typescript
483/** src/firehose.ts **/
484// If the write is a valid status update
485if (
486 evt.collection === 'com.example.status' &&
487 Status.isRecord(record) &&
488 Status.validateRecord(record).success
489) {
490 // Store the status in our SQLite
491 await db
492 .insertInto('status')
493 .values({
494 uri: evt.uri.toString(),
495 authorDid: evt.author,
496 status: record.status,
497 createdAt: record.createdAt,
498 indexedAt: new Date().toISOString(),
499 })
500 .onConflict((oc) =>
501 oc.column('uri').doUpdateSet({
502 status: record.status,
503 indexedAt: new Date().toISOString(),
504 })
505 )
506 .execute()
507}
508```
509
510You can almost think of information flowing in a loop:
511
512
513
514Why read from the event log? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in -- including data published by other apps.
515
516## Step 7. Listing the latest statuses
517
518Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](#todo)-to-handle resolver so we can show a nice username with the statuses:
519
520```typescript
521/** src/routes.ts **/
522// Homepage
523router.get(
524 '/',
525 handler(async (req, res) => {
526 // ...
527
528 // Fetch data stored in our SQLite
529 const statuses = await db
530 .selectFrom('status')
531 .selectAll()
532 .orderBy('indexedAt', 'desc')
533 .limit(10)
534 .execute()
535
536 // Map user DIDs to their domain-name handles
537 const didHandleMap = await resolver.resolveDidsToHandles(
538 statuses.map((s) => s.authorDid)
539 )
540
541 // ...
542 })
543)
544```
545
546Our HTML can now list these status records:
547
548```html
549<!-- src/pages/home.ts -->
550${statuses.map((status, i) => {
551 const handle = didHandleMap[status.authorDid] || status.authorDid
552 return html`
553 <div class="status-line">
554 <div>
555 <div class="status">${status.status}</div>
556 </div>
557 <div class="desc">
558 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
559 was feeling ${status.status} on ${status.indexedAt}.
560 </div>
561 </div>
562 `
563})}
564```
565
566
567
568## Step 8. Optimistic updates
569
570As a final optimization, let's introduce "optimistic updates." Remember the information flow loop with the repo write and the event log? Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
571
572
573
574This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
575
576To do this, we just update `POST /status` to include an additional write to our SQLite DB:
577
578```typescript
579/** src/routes.ts **/
580// "Set status" handler
581router.post(
582 '/status',
583 handler(async (req, res) => {
584 // ...
585
586 let uri
587 try {
588 // Write the status record to the user's repository
589 const res = await agent.putRecord({
590 repo: agent.accountDid,
591 collection: 'com.example.status',
592 rkey: TID.nextStr(),
593 record,
594 })
595 uri = res.uri
596 } catch (err) {
597 logger.warn({ err }, 'failed to write record')
598 return res.status(500).json({ error: 'Failed to write record' })
599 }
600
601 try {
602 // Optimistically update our SQLite <-- HERE!
603 await db
604 .insertInto('status')
605 .values({
606 uri,
607 authorDid: agent.accountDid,
608 status: record.status,
609 createdAt: record.createdAt,
610 indexedAt: new Date().toISOString(),
611 })
612 .execute()
613 } catch (err) {
614 logger.warn(
615 { err },
616 'failed to update computed view; ignoring as it should be caught by the firehose'
617 )
618 }
619
620 res.status(200).json({})
621 })
622)
623```
624
625You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
626
627## Thinking in AT Proto
628
629In this tutorial we've covered the key steps to building an atproto app. Data is published in its canonical form on users' `at://` repos and then aggregated into apps' databases to produce views of the network.
630
631When building your app, think in these four key steps:
632
633- Design the [Lexicon](#) schemas for the records you'll publish into the Atmosphere.
634- Create a database for aggregating the records into useful views.
635- Build your application to write the records on your users' repos.
636- Listen to the firehose to hydrate your aggregated database.
637
638Remember this flow of information throughout:
639
640
641
642This is how every app in the Atmosphere works, including the [Bluesky social app](https://bsky.app).
643
644## Next steps
645
646TODO
647
648