Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a **simple multi-user app** that publishes your current "status" as an emoji.
4
5At various points we will cover how to:
6
7- Signin via OAuth
8- Fetch information about users (profiles)
9- Listen to the network firehose for new data
10- Publish data on the user's account using a custom schema
11
12We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
13
14## Where are we going?
15
16Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
17
18Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
19
20- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
21- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
22- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
23
24The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
25
26> `at://` is the URL scheme of the AT Protocol.
27
28## Step 1. Starting with our ExpressJS app
29
30Start by cloning the repo and installing packages.
31
32```bash
33git clone TODO
34cd TODO
35npm i
36npm run dev # you can leave this running and it will auto-reload
37```
38
39Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](#todo).
40
41Our starting stack:
42
43- Typescript
44- NodeJS web server ([express](#todo))
45- SQLite database ([Kysley](#todo))
46- Server-side rendering ([uhtml](#todo))
47
48With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
49
50## Step 2. Signing in with OAuth
51
52When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
53
54We're going to accomplish this using OAuth ([spec](#todo)). You can find a [more extensive OAuth guide here](#todo), but for now just know that most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](#todo) library. This is the arrangement we're aiming toward:
55
56```
57 ┌─App Server───────────────────┐
58 │ ┌─► Session store ◄┐ │
59 │ │ │ │ ┌───────────────┐
60 │ App code ──────►OAuth client─┼───►│ User's server │
61 └────▲─────────────────────────┘ └───────────────┘
62 ┌────┴──────────┐
63 │ Web browser │
64 └───────────────┘
65```
66
67When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
68
69Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
70
71```html
72<!-- src/pages/login.ts -->
73<form action="/login" method="post" class="login-form">
74 <input
75 type="text"
76 name="handle"
77 placeholder="Enter your handle (eg alice.bsky.social)"
78 required
79 />
80 <button type="submit">Log in</button>
81</form>
82```
83
84When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
85
86```typescript
87/** src/routes.ts **/
88// Login handler
89router.post(
90 '/login',
91 handler(async (req, res) => {
92 // Initiate the OAuth flow
93 const url = await oauthClient.authorize(handle)
94 return res.redirect(url.toString())
95 })
96)
97```
98
99This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
100
101When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](#todo) to their cookie-session.
102
103```typescript
104/** src/routes.ts **/
105// OAuth callback to complete session creation
106router.get(
107 '/oauth/callback',
108 handler(async (req, res) => {
109 // Store the credentials
110 const { agent } = await oauthClient.callback(params)
111
112 // Attach the account DID to our user via a cookie
113 const session = await getIronSession(req, res)
114 session.did = agent.accountDid
115 await session.save()
116
117 // Send them back to the app
118 return res.redirect('/')
119 })
120)
121```
122
123With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
124
125## Step 3. Fetching the user's profile
126
127Why don't we learn something about our user? Let's start by getting the [Agent](#todo) object. The [Agent](#todo) is the client to the user's `at://` repo server.
128
129```typescript
130/** src/routes.ts **/
131async function getSessionAgent(
132 req: IncomingMessage,
133 res: ServerResponse<IncomingMessage>,
134 ctx: AppContext
135) {
136 // Fetch the session from their cookie
137 const session = await getIronSession(req, res)
138 if (!session.did) return null
139
140 // "Restore" the agent for the user
141 return await ctx.oauthClient.restore(session.did).catch(async (err) => {
142 ctx.logger.warn({ err }, 'oauth restore failed')
143 await session.destroy()
144 return null
145 })
146}
147```
148
149Users publish JSON records on their `at://` repos. In [Bluesky](https://bsky.app), they publish a "profile" record which looks like this:
150
151```typescript
152interface ProfileRecord {
153 displayName?: string // a human friendly name
154 description?: string // a short bio
155 avatar?: BlobRef // small profile picture
156 banner?: BlobRef // banner image to put on profiles
157 createdAt?: string // declared time this profile data was added
158 // ...
159}
160```
161
162We're going to use the [Agent](#todo) to fetch this record to include in our app.
163
164```typescript
165await agent.getRecord({
166 repo: agent.accountDid, // The user
167 collection: 'app.bsky.actor.profile', // The collection
168 rkey: 'self', // The record key
169})
170```
171
172When asking for a record, we provide three pieces of information.
173
174- The [DID](#todo) which identifies the user,
175- The collection name, and
176- The record key
177
178We'll explain the collection name shortly. Record keys are strings with [some limitations](https://atproto.com/specs/record-key#record-key-syntax) and a couple of common patterns. The `"self"` pattern is used when a collection is expected to only contain one record which describes the user.
179
180Let's update our homepage to fetch this profile record:
181
182```typescript
183/** src/routes.ts **/
184// Homepage
185router.get(
186 '/',
187 handler(async (req, res) => {
188 // If the user is signed in, get an agent which communicates with their server
189 const agent = await getSessionAgent(req, res, ctx)
190
191 if (!agent) {
192 // Serve the logged-out view
193 return res.type('html').send(page(home()))
194 }
195
196 // Fetch additional information about the logged-in user
197 const { data: profileRecord } = await agent.getRecord({
198 repo: agent.accountDid, // our user's repo
199 collection: 'app.bsky.actor.profile', // the bluesky profile record type
200 rkey: 'self', // the record's name
201 })
202
203 // Serve the logged-in view
204 return res
205 .type('html')
206 .send(page(home({ profile: profileRecord.value || {} })))
207 })
208)
209```
210
211With that data, we can give a nice personalized welcome banner for our user:
212
213```html
214<!-- pages/home.ts -->
215<div class="card">
216 ${profile
217 ? html`<form action="/logout" method="post" class="session-form">
218 <div>
219 Hi, <strong>${profile.displayName || 'friend'}</strong>.
220 What's your status today?
221 </div>
222 <div>
223 <button type="submit">Log out</button>
224 </div>
225 </form>`
226 : html`<div class="session-form">
227 <div><a href="/login">Log in</a> to set your status!</div>
228 <div>
229 <a href="/login" class="button">Log in</a>
230 </div>
231 </div>`}
232</div>
233```
234
235You can examine this record directly using [atproto-browser.vercel.app](https://atproto-browser.vercel.app). For instance, [this is the profile record for @bsky.app](https://atproto-browser.vercel.app/at?u=at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.actor.profile/self).
236
237## Step 4. Reading & writing records
238
239You can think of the user repositories as collections of JSON records:
240
241```
242 ┌────────┐
243 ┌───| record │
244 ┌────────────┐ │ └────────┘
245 ┌───| collection |◄─┤ ┌────────┐
246┌──────┐ │ └────────────┘ └───| record │
247│ repo |◄──┤ └────────┘
248└──────┘ │ ┌────────────┐ ┌────────┐
249 └───┤ collection |◄─────| record │
250 └────────────┘ └────────┘
251```
252
253Let's look again at how we read the "profile" record:
254
255```typescript
256await agent.getRecord({
257 repo: agent.accountDid, // The user
258 collection: 'app.bsky.actor.profile', // The collection
259 rkey: 'self', // The record key
260})
261```
262
263We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
264
265```typescript
266// Generate a time-based key for our record
267const rkey = TID.nextStr()
268
269// Write the
270await agent.putRecord({
271 repo: agent.accountDid, // The user
272 collection: 'com.example.status', // The collection
273 rkey, // The record key
274 record: { // The record value
275 status: "👍",
276 createdAt: new Date().toISOString()
277 }
278})
279```
280
281Our `POST /status` route is going to use this API to publish the user's status to their repo.
282
283```typescript
284/** src/routes.ts **/
285// "Set status" handler
286router.post(
287 '/status',
288 handler(async (req, res) => {
289 // If the user is signed in, get an agent which communicates with their server
290 const agent = await getSessionAgent(req, res, ctx)
291 if (!agent) {
292 return res.status(401).json({ error: 'Session required' })
293 }
294
295 // Construct their status record
296 const record = {
297 $type: 'com.example.status',
298 status: req.body?.status,
299 createdAt: new Date().toISOString(),
300 }
301
302 try {
303 // Write the status record to the user's repository
304 await agent.putRecord({
305 repo: agent.accountDid,
306 collection: 'com.example.status',
307 rkey: TID.nextStr(),
308 record,
309 })
310 } catch (err) {
311 ctx.logger.warn({ err }, 'failed to write record')
312 return res.status(500).json({ error: 'Failed to write record' })
313 }
314
315 res.status(200).json({})
316 })
317)
318```
319
320Now in our homepage we can list out the status buttons:
321
322```html
323<!-- src/pages/home.ts -->
324<div class="status-options">
325 ${['👍', '🦋', '🥳', /*...*/].map(status => html`
326 <div class="status-option" data-value="${status}">
327 ${status}
328 </div>`
329 )}
330</div>
331```
332
333And write some client-side javascript to submit the status on click:
334
335```javascript
336/* src/pages/public/home.js */
337Array.from(document.querySelectorAll('.status-option'), (el) => {
338 el.addEventListener('click', async (ev) => {
339 const res = await fetch('/status', {
340 method: 'POST',
341 headers: { 'content-type': 'application/json' },
342 body: JSON.stringify({ status: el.dataset.value }),
343 })
344 const body = await res.json()
345 if (!body?.error) {
346 location.reload()
347 }
348 })
349})
350```
351
352## Step 5. Creating a custom "status" schema
353
354The collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
355
356Anybody can create a new schema using the [Lexicon](#todo) language, which is very similar to [JSON-Schema](#todo). The schemas use [reverse-DNS IDs](#todo) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
357
358> ### Why create a schema?
359>
360> Schemas help other applications understand the data your app is creating. By publishing your schemas, you enable compatibility with other apps and reduce the chances of bad data affecting your app.
361
362Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](#todo).
363
364```json
365/* lexicons/status.json */
366{
367 "lexicon": 1,
368 "id": "com.example.status",
369 "defs": {
370 "main": {
371 "type": "record",
372 "key": "tid",
373 "record": {
374 "type": "object",
375 "required": ["status", "createdAt"],
376 "properties": {
377 "status": {
378 "type": "string",
379 "minLength": 1,
380 "maxGraphemes": 1,
381 "maxLength": 32
382 },
383 "createdAt": {
384 "type": "string",
385 "format": "datetime"
386 }
387 }
388 }
389 }
390 }
391}
392```
393
394Now let's run some code-generation using our schema:
395
396```bash
397./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
398```
399
400This will produce Typescript interfaces as well as runtime validation functions that we can use in our `POST /status` route:
401
402```typescript
403/** src/routes.ts **/
404import * as Status from '#/lexicon/types/com/example/status'
405// ...
406// "Set status" handler
407router.post(
408 '/status',
409 handler(async (req, res) => {
410 // ...
411
412 // Construct & validate their status record
413 const record = {
414 $type: 'com.example.status',
415 status: req.body?.status,
416 createdAt: new Date().toISOString(),
417 }
418 if (!Status.validateRecord(record).success) {
419 return res.status(400).json({ error: 'Invalid status' })
420 }
421
422 // ...
423 })
424)
425```
426
427## Step 6. Listening to the firehose
428
429So far, we have:
430
431- Logged in via OAuth
432- Created a custom schema
433- Read & written records for the logged in user
434
435Now we want to fetch the status records from other users.
436
437Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
438
439```
440┌──────┐
441│ REPO │ Event stream
442├──────┘
443│ ┌───────────────────────────────────────────┐
444├───┼ 1 PUT /app.bsky.feed.post/3l244rmrxjx2v │
445│ └───────────────────────────────────────────┘
446│ ┌───────────────────────────────────────────┐
447├───┼ 2 DEL /app.bsky.feed.post/3l244rmrxjx2v │
448│ └───────────────────────────────────────────┘
449│ ┌───────────────────────────────────────────┐
450├───┼ 3 PUT /app.bsky.actor.profile/self │
451▼ └───────────────────────────────────────────┘
452```
453
454Using a [Relay service](#todo) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
455
456
457```typescript
458/** src/firehose.ts **/
459import * as Status from '#/lexicon/types/com/example/status'
460// ...
461const firehose = new Firehose({})
462
463for await (const evt of firehose.run()) {
464 // Watch for write events
465 if (evt.event === 'create' || evt.event === 'update') {
466 const record = evt.record
467
468 // If the write is a valid status update
469 if (
470 evt.collection === 'com.example.status' &&
471 Status.isRecord(record) &&
472 Status.validateRecord(record).success
473 ) {
474 // Store the status
475 // TODO
476 }
477 }
478}
479```
480
481Let's create a SQLite table to store these statuses:
482
483```typescript
484/** src/db.ts **/
485// Create our statuses table
486await db.schema
487 .createTable('status')
488 .addColumn('uri', 'varchar', (col) => col.primaryKey())
489 .addColumn('authorDid', 'varchar', (col) => col.notNull())
490 .addColumn('status', 'varchar', (col) => col.notNull())
491 .addColumn('createdAt', 'varchar', (col) => col.notNull())
492 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
493 .execute()
494```
495
496Now we can write these statuses into our database as they arrive from the firehose:
497
498```typescript
499/** src/firehose.ts **/
500// If the write is a valid status update
501if (
502 evt.collection === 'com.example.status' &&
503 Status.isRecord(record) &&
504 Status.validateRecord(record).success
505) {
506 // Store the status in our SQLite
507 await db
508 .insertInto('status')
509 .values({
510 uri: evt.uri.toString(),
511 authorDid: evt.author,
512 status: record.status,
513 createdAt: record.createdAt,
514 indexedAt: new Date().toISOString(),
515 })
516 .onConflict((oc) =>
517 oc.column('uri').doUpdateSet({
518 status: record.status,
519 indexedAt: new Date().toISOString(),
520 })
521 )
522 .execute()
523}
524```
525
526You can almost think of information flowing in a loop:
527
528```
529 ┌─────Repo put─────┐
530 │ ▼
531┌──────┴─────┐ ┌───────────┐
532│ App server │ │ User repo │
533└────────────┘ └─────┬─────┘
534 ▲ │
535 └────Event log─────┘
536```
537
538Why read from the event log? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in -- including data published by other apps.
539
540## Step 7. Listing the latest statuses
541
542Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](#todo)-to-handle resolver so we can show a nice username with the statuses:
543
544```typescript
545/** src/routes.ts **/
546// Homepage
547router.get(
548 '/',
549 handler(async (req, res) => {
550 // ...
551
552 // Fetch data stored in our SQLite
553 const statuses = await db
554 .selectFrom('status')
555 .selectAll()
556 .orderBy('indexedAt', 'desc')
557 .limit(10)
558 .execute()
559
560 // Map user DIDs to their domain-name handles
561 const didHandleMap = await resolver.resolveDidsToHandles(
562 statuses.map((s) => s.authorDid)
563 )
564
565 // ...
566 })
567)
568```
569
570Our HTML can now list these status records:
571
572```html
573<!-- src/pages/home.ts -->
574${statuses.map((status, i) => {
575 const handle = didHandleMap[status.authorDid] || status.authorDid
576 return html`
577 <div class="status-line">
578 <div>
579 <div class="status">${status.status}</div>
580 </div>
581 <div class="desc">
582 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
583 was feeling ${status.status} on ${status.indexedAt}.
584 </div>
585 </div>
586 `
587})}
588```
589
590## Step 8. Optimistic updates
591
592As a final optimization, let's introduce "optimistic updates." Remember the information flow loop with the repo write and the event log? Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
593
594```
595 ┌───Repo put──┬──────┐
596 │ │ ▼
597┌──────┴─────┐ │ ┌───────────┐
598│ App server │◄──────┘ │ User repo │
599└────────────┘ └───┬───────┘
600 ▲ │
601 └────Event log───────┘
602```
603
604This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
605
606To do this, we just update `POST /status` to include an additional write to our SQLite DB:
607
608```typescript
609/** src/routes.ts **/
610// "Set status" handler
611router.post(
612 '/status',
613 handler(async (req, res) => {
614 // ...
615
616 let uri
617 try {
618 // Write the status record to the user's repository
619 const res = await agent.putRecord({
620 repo: agent.accountDid,
621 collection: 'com.example.status',
622 rkey: TID.nextStr(),
623 record,
624 })
625 uri = res.uri
626 } catch (err) {
627 ctx.logger.warn({ err }, 'failed to write record')
628 return res.status(500).json({ error: 'Failed to write record' })
629 }
630
631 try {
632 // Optimistically update our SQLite <-- HERE!
633 await ctx.db
634 .insertInto('status')
635 .values({
636 uri,
637 authorDid: agent.accountDid,
638 status: record.status,
639 createdAt: record.createdAt,
640 indexedAt: new Date().toISOString(),
641 })
642 .execute()
643 } catch (err) {
644 ctx.logger.warn(
645 { err },
646 'failed to update computed view; ignoring as it should be caught by the firehose'
647 )
648 }
649
650 res.status(200).json({})
651 })
652)
653```
654
655You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
656
657## Next steps
658
659TODO
660
661