Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a **simple multi-user app** that publishes your current "status" as an emoji.
4
5At various points we will cover how to:
6
7- Signin via OAuth
8- Fetch information about users (profiles)
9- Listen to the network firehose for new data
10- Publish data on the user's account using a custom schema
11
12We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
13
14## Where are we going?
15
16Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
17
18Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
19
20- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
21- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
22- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
23
24The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
25
26> `at://` is the URL scheme of the AT Protocol.
27
28## Step 1. Starting with our ExpressJS app
29
30Start by cloning the repo and installing packages.
31
32```bash
33git clone TODO
34cd TODO
35npm i
36npm run dev # you can leave this running and it will auto-reload
37```
38
39Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](#todo).
40
41Our starting stack:
42
43- Typescript
44- NodeJS web server ([express](#todo))
45- SQLite database ([Kysley](#todo))
46- Server-side rendering ([uhtml](#todo))
47
48With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
49
50## Step 2. Signing in with OAuth
51
52When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
53
54We're going to accomplish this using OAuth ([spec](#todo)). You can find a [more extensive OAuth guide here](#todo), but for now just know that most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](#todo) library. This is the arrangement we're aiming toward:
55
56```
57 ┌─App Server───────────────────┐
58 │ ┌─► Session store ◄┐ │
59 │ │ │ │ ┌───────────────┐
60 │ App code ──────►OAuth client─┼───►│ User's server │
61 └────▲─────────────────────────┘ └───────────────┘
62 ┌────┴──────────┐
63 │ Web browser │
64 └───────────────┘
65```
66
67When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
68
69Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
70
71```html
72<!-- src/pages/login.ts -->
73<form action="/login" method="post" class="login-form">
74 <input
75 type="text"
76 name="handle"
77 placeholder="Enter your handle (eg alice.bsky.social)"
78 required
79 />
80 <button type="submit">Log in</button>
81</form>
82```
83
84When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
85
86```typescript
87/** src/routes.ts **/
88// Login handler
89router.post(
90 '/login',
91 handler(async (req, res) => {
92 // Initiate the OAuth flow
93 const url = await oauthClient.authorize(handle)
94 return res.redirect(url.toString())
95 })
96)
97```
98
99This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
100
101When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](#todo) to their cookie-session.
102
103```typescript
104/** src/routes.ts **/
105// OAuth callback to complete session creation
106router.get(
107 '/oauth/callback',
108 handler(async (req, res) => {
109 // Store the credentials
110 const { agent } = await oauthClient.callback(params)
111
112 // Attach the account DID to our user via a cookie
113 const session = await getIronSession(req, res)
114 session.did = agent.accountDid
115 await session.save()
116
117 // Send them back to the app
118 return res.redirect('/')
119 })
120)
121```
122
123With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
124
125## Step 3. Fetching the user's profile
126
127Why don't we learn something about our user? Let's start by getting the [Agent](#todo) object. The [Agent](#todo) is the client to the user's `at://` repo server.
128
129```typescript
130/** src/routes.ts **/
131async function getSessionAgent(
132 req: IncomingMessage,
133 res: ServerResponse<IncomingMessage>,
134 ctx: AppContext
135) {
136 // Fetch the session from their cookie
137 const session = await getIronSession(req, res)
138 if (!session.did) return null
139
140 // "Restore" the agent for the user
141 return await ctx.oauthClient.restore(session.did).catch(async (err) => {
142 ctx.logger.warn({ err }, 'oauth restore failed')
143 await session.destroy()
144 return null
145 })
146}
147```
148
149Users publish JSON records on their `at://` repos. In [Bluesky](https://bsky.app), they publish a "profile" record which looks like this:
150
151```typescript
152interface ProfileRecord {
153 displayName?: string // a human friendly name
154 description?: string // a short bio
155 avatar?: BlobRef // small profile picture
156 banner?: BlobRef // banner image to put on profiles
157 createdAt?: string // declared time this profile data was added
158 // ...
159}
160```
161
162We're going to use the [Agent](#todo) to fetch this record to include in our app.
163
164```typescript
165/** src/routes.ts **/
166// Homepage
167router.get(
168 '/',
169 handler(async (req, res) => {
170 // If the user is signed in, get an agent which communicates with their server
171 const agent = await getSessionAgent(req, res, ctx)
172
173 if (!agent) {
174 // Serve the logged-out view
175 return res.type('html').send(page(home()))
176 }
177
178 // Fetch additional information about the logged-in user
179 const { data: profileRecord } = await agent.getRecord({
180 repo: agent.accountDid, // our user's repo
181 collection: 'app.bsky.actor.profile', // the bluesky profile record type
182 rkey: 'self', // the record's name
183 })
184
185 // Serve the logged-in view
186 return res
187 .type('html')
188 .send(page(home({ profile: profileRecord.value || {} })))
189 })
190)
191```
192
193With that data, we can give a nice personalized welcome banner for our user:
194
195```html
196<!-- pages/home.ts -->
197<div class="card">
198 ${profile
199 ? html`<form action="/logout" method="post" class="session-form">
200 <div>
201 Hi, <strong>${profile.displayName || 'friend'}</strong>.
202 What's your status today?
203 </div>
204 <div>
205 <button type="submit">Log out</button>
206 </div>
207 </form>`
208 : html`<div class="session-form">
209 <div><a href="/login">Log in</a> to set your status!</div>
210 <div>
211 <a href="/login" class="button">Log in</a>
212 </div>
213 </div>`}
214</div>
215```
216
217## Step 4. Reading & writing records
218
219You can think of the user repositories as collections of JSON records:
220
221```
222 ┌────────┐
223 ┌───| record │
224 ┌────────────┐ │ └────────┘
225 ┌───| collection |◄─┤ ┌────────┐
226┌──────┐ │ └────────────┘ └───| record │
227│ repo |◄──┤ └────────┘
228└──────┘ │ ┌────────────┐ ┌────────┐
229 └───┤ collection |◄─────| record │
230 └────────────┘ └────────┘
231```
232
233Let's look again at how we read the "profile" record:
234
235```typescript
236await agent.getRecord({
237 repo: agent.accountDid, // The user
238 collection: 'app.bsky.actor.profile', // The collection
239 rkey: 'self', // The record key
240})
241```
242
243We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
244
245```typescript
246// Generate a time-based key for our record
247const rkey = TID.nextStr()
248
249// Write the
250await agent.putRecord({
251 repo: agent.accountDid, // The user
252 collection: 'com.example.status', // The collection
253 rkey, // The record key
254 record: { // The record value
255 status: "👍",
256 createdAt: new Date().toISOString()
257 }
258})
259```
260
261Our `POST /status` route is going to use this API to publish the user's status to their repo.
262
263```typescript
264/** src/routes.ts **/
265// "Set status" handler
266router.post(
267 '/status',
268 handler(async (req, res) => {
269 // If the user is signed in, get an agent which communicates with their server
270 const agent = await getSessionAgent(req, res, ctx)
271 if (!agent) {
272 return res.status(401).json({ error: 'Session required' })
273 }
274
275 // Construct their status record
276 const record = {
277 $type: 'com.example.status',
278 status: req.body?.status,
279 createdAt: new Date().toISOString(),
280 }
281
282 try {
283 // Write the status record to the user's repository
284 await agent.putRecord({
285 repo: agent.accountDid,
286 collection: 'com.example.status',
287 rkey: TID.nextStr(),
288 record,
289 })
290 } catch (err) {
291 ctx.logger.warn({ err }, 'failed to write record')
292 return res.status(500).json({ error: 'Failed to write record' })
293 }
294
295 res.status(200).json({})
296 })
297)
298```
299
300Now in our homepage we can list out the status buttons:
301
302```html
303<!-- src/pages/home.ts -->
304<div class="status-options">
305 ${['👍', '🦋', '🥳', /*...*/].map(status => html`
306 <div class="status-option" data-value="${status}">
307 ${status}
308 </div>`
309 )}
310</div>
311```
312
313And write some client-side javascript to submit the status on click:
314
315```javascript
316/* src/pages/public/home.js */
317Array.from(document.querySelectorAll('.status-option'), (el) => {
318 el.addEventListener('click', async (ev) => {
319 const res = await fetch('/status', {
320 method: 'POST',
321 headers: { 'content-type': 'application/json' },
322 body: JSON.stringify({ status: el.dataset.value }),
323 })
324 const body = await res.json()
325 if (!body?.error) {
326 location.reload()
327 }
328 })
329})
330```
331
332## Step 5. Creating a custom "status" schema
333
334The collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
335
336Anybody can create a new schema using the [Lexicon](#todo) language, which is very similar to [JSON-Schema](#todo). The schemas use [reverse-DNS IDs](#todo) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
337
338> ### Why create a schema?
339>
340> Schemas help other applications understand the data your app is creating. By publishing your schemas, you enable compatibility with other apps and reduce the chances of bad data affecting your app.
341
342Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](#todo).
343
344```json
345/* lexicons/status.json */
346{
347 "lexicon": 1,
348 "id": "com.example.status",
349 "defs": {
350 "main": {
351 "type": "record",
352 "key": "tid",
353 "record": {
354 "type": "object",
355 "required": ["status", "createdAt"],
356 "properties": {
357 "status": {
358 "type": "string",
359 "minLength": 1,
360 "maxGraphemes": 1,
361 "maxLength": 32
362 },
363 "createdAt": {
364 "type": "string",
365 "format": "datetime"
366 }
367 }
368 }
369 }
370 }
371}
372```
373
374Now let's run some code-generation using our schema:
375
376```bash
377./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
378```
379
380This will produce Typescript interfaces as well as runtime validation functions that we can use in our `POST /status` route:
381
382```typescript
383/** src/routes.ts **/
384import * as Status from '#/lexicon/types/com/example/status'
385// ...
386// "Set status" handler
387router.post(
388 '/status',
389 handler(async (req, res) => {
390 // ...
391
392 // Construct & validate their status record
393 const record = {
394 $type: 'com.example.status',
395 status: req.body?.status,
396 createdAt: new Date().toISOString(),
397 }
398 if (!Status.validateRecord(record).success) {
399 return res.status(400).json({ error: 'Invalid status' })
400 }
401
402 // ...
403 })
404)
405```
406
407## Step 6. Listening to the firehose
408
409So far, we have:
410
411- Logged in via OAuth
412- Created a custom schema
413- Read & written records for the logged in user
414
415Now we want to fetch the status records from other users.
416
417Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
418
419```
420┌──────┐
421│ REPO │ Event stream
422├──────┘
423│ ┌───────────────────────────────────────────┐
424├───┼ 1 PUT /app.bsky.feed.post/3l244rmrxjx2v │
425│ └───────────────────────────────────────────┘
426│ ┌───────────────────────────────────────────┐
427├───┼ 2 DEL /app.bsky.feed.post/3l244rmrxjx2v │
428│ └───────────────────────────────────────────┘
429│ ┌───────────────────────────────────────────┐
430├───┼ 3 PUT /app.bsky.actor.profile/self │
431▼ └───────────────────────────────────────────┘
432```
433
434Using a [Relay service](#todo) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
435
436
437```typescript
438/** src/firehose.ts **/
439import * as Status from '#/lexicon/types/com/example/status'
440// ...
441const firehose = new Firehose({})
442
443for await (const evt of firehose.run()) {
444 // Watch for write events
445 if (evt.event === 'create' || evt.event === 'update') {
446 const record = evt.record
447
448 // If the write is a valid status update
449 if (
450 evt.collection === 'com.example.status' &&
451 Status.isRecord(record) &&
452 Status.validateRecord(record).success
453 ) {
454 // Store the status
455 // TODO
456 }
457 }
458}
459```
460
461Let's create a SQLite table to store these statuses:
462
463```typescript
464/** src/db.ts **/
465// Create our statuses table
466await db.schema
467 .createTable('status')
468 .addColumn('uri', 'varchar', (col) => col.primaryKey())
469 .addColumn('authorDid', 'varchar', (col) => col.notNull())
470 .addColumn('status', 'varchar', (col) => col.notNull())
471 .addColumn('createdAt', 'varchar', (col) => col.notNull())
472 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
473 .execute()
474```
475
476Now we can write these statuses into our database as they arrive from the firehose:
477
478```typescript
479/** src/firehose.ts **/
480// If the write is a valid status update
481if (
482 evt.collection === 'com.example.status' &&
483 Status.isRecord(record) &&
484 Status.validateRecord(record).success
485) {
486 // Store the status in our SQLite
487 await db
488 .insertInto('status')
489 .values({
490 uri: evt.uri.toString(),
491 authorDid: evt.author,
492 status: record.status,
493 createdAt: record.createdAt,
494 indexedAt: new Date().toISOString(),
495 })
496 .onConflict((oc) =>
497 oc.column('uri').doUpdateSet({
498 status: record.status,
499 indexedAt: new Date().toISOString(),
500 })
501 )
502 .execute()
503}
504```
505
506You can almost think of information flowing in a loop:
507
508```
509 ┌─────Repo put─────┐
510 │ ▼
511┌──────┴─────┐ ┌───────────┐
512│ App server │ │ User repo │
513└────────────┘ └─────┬─────┘
514 ▲ │
515 └────Event log─────┘
516```
517
518Why read from the event log? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in -- including data published by other apps.
519
520## Step 7. Listing the latest statuses
521
522Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](#todo)-to-handle resolver so we can show a nice username with the statuses:
523
524```typescript
525/** src/routes.ts **/
526// Homepage
527router.get(
528 '/',
529 handler(async (req, res) => {
530 // ...
531
532 // Fetch data stored in our SQLite
533 const statuses = await db
534 .selectFrom('status')
535 .selectAll()
536 .orderBy('indexedAt', 'desc')
537 .limit(10)
538 .execute()
539
540 // Map user DIDs to their domain-name handles
541 const didHandleMap = await resolver.resolveDidsToHandles(
542 statuses.map((s) => s.authorDid)
543 )
544
545 // ...
546 })
547)
548```
549
550Our HTML can now list these status records:
551
552```html
553<!-- src/pages/home.ts -->
554${statuses.map((status, i) => {
555 const handle = didHandleMap[status.authorDid] || status.authorDid
556 return html`
557 <div class="status-line">
558 <div>
559 <div class="status">${status.status}</div>
560 </div>
561 <div class="desc">
562 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
563 was feeling ${status.status} on ${status.indexedAt}.
564 </div>
565 </div>
566 `
567})}
568```
569
570## Step 8. Optimistic updates
571
572As a final optimization, let's introduce "optimistic updates." Remember the information flow loop with the repo write and the event log? Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
573
574```
575 ┌───Repo put──┬──────┐
576 │ │ ▼
577┌──────┴─────┐ │ ┌───────────┐
578│ App server │◄──────┘ │ User repo │
579└────────────┘ └───┬───────┘
580 ▲ │
581 └────Event log───────┘
582```
583
584This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
585
586To do this, we just update `POST /status` to include an additional write to our SQLite DB:
587
588```typescript
589/** src/routes.ts **/
590// "Set status" handler
591router.post(
592 '/status',
593 handler(async (req, res) => {
594 // ...
595
596 let uri
597 try {
598 // Write the status record to the user's repository
599 const res = await agent.putRecord({
600 repo: agent.accountDid,
601 collection: 'com.example.status',
602 rkey: TID.nextStr(),
603 record,
604 })
605 uri = res.uri
606 } catch (err) {
607 ctx.logger.warn({ err }, 'failed to write record')
608 return res.status(500).json({ error: 'Failed to write record' })
609 }
610
611 try {
612 // Optimistically update our SQLite <-- HERE!
613 await ctx.db
614 .insertInto('status')
615 .values({
616 uri,
617 authorDid: agent.accountDid,
618 status: record.status,
619 createdAt: record.createdAt,
620 indexedAt: new Date().toISOString(),
621 })
622 .execute()
623 } catch (err) {
624 ctx.logger.warn(
625 { err },
626 'failed to update computed view; ignoring as it should be caught by the firehose'
627 )
628 }
629
630 res.status(200).json({})
631 })
632)
633```
634
635You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
636
637## Next steps
638
639TODO
640
641