Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a **simple multi-user app** that publishes your current "status" as an emoji.
4
5At various points we will cover how to:
6
7- Signin via OAuth
8- Fetch information about users (profiles)
9- Listen to the network firehose for new data
10- Publish data on the user's account using a custom schema
11
12We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
13
14## Where are we going?
15
16Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
17
18Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
19
20- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
21- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
22- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
23
24The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the `/status.json` records and aggregate them into our SQLite database.
25
26## Step 1. Starting with our ExpressJS app
27
28Start by cloning the repo and installing packages.
29
30```bash
31git clone TODO
32cd TODO
33npm i
34npm run dev # you can leave this running and it will auto-reload
35```
36
37Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](#todo).
38
39Our starting stack:
40
41- Typescript
42- NodeJS web server ([express](#todo))
43- SQLite database ([Kysley](#todo))
44- Server-side rendering ([uhtml](#todo))
45
46With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
47
48## Step 2. Signing in with OAuth
49
50When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
51
52We're going to accomplish this using OAuth ([spec](#todo)). You can find a [more extensive OAuth guide here](#todo), but for now just know that most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](#todo) library. This is the arrangement we're aiming toward:
53
54```
55 ┌─App Server───────────────────┐
56 │ ┌─► Session store ◄┐ │
57 │ │ │ │ ┌───────────────┐
58 │ App code ──────►OAuth client─┼───►│ User's server │
59 └────▲─────────────────────────┘ └───────────────┘
60 ┌────┴──────────┐
61 │ Web browser │
62 └───────────────┘
63```
64
65When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
66
67Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
68
69```html
70<!-- src/pages/login.ts -->
71<form action="/login" method="post" class="login-form">
72 <input
73 type="text"
74 name="handle"
75 placeholder="Enter your handle (eg alice.bsky.social)"
76 required
77 />
78 <button type="submit">Log in</button>
79</form>
80```
81
82When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
83
84```typescript
85/** src/routes.ts **/
86// Login handler
87router.post(
88 '/login',
89 handler(async (req, res) => {
90 // Initiate the OAuth flow
91 const url = await oauthClient.authorize(handle)
92 return res.redirect(url.toString())
93 })
94)
95```
96
97This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
98
99When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](#todo) to their cookie-session.
100
101```typescript
102/** src/routes.ts **/
103// OAuth callback to complete session creation
104router.get(
105 '/oauth/callback',
106 handler(async (req, res) => {
107 // Store the credentials
108 const { agent } = await oauthClient.callback(params)
109
110 // Attach the account DID to our user via a cookie
111 const session = await getIronSession(req, res)
112 session.did = agent.accountDid
113 await session.save()
114
115 // Send them back to the app
116 return res.redirect('/')
117 })
118)
119```
120
121With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
122
123## Step 3. Fetching the user's profile
124
125Why don't we learn something about our user? Let's start by getting the [Agent](#todo) object. The [Agent](#todo) is the client to the user's `at://` repo server.
126
127```typescript
128/** src/routes.ts **/
129async function getSessionAgent(
130 req: IncomingMessage,
131 res: ServerResponse<IncomingMessage>,
132 ctx: AppContext
133) {
134 // Fetch the session from their cookie
135 const session = await getIronSession(req, res)
136 if (!session.did) return null
137
138 // "Restore" the agent for the user
139 return await ctx.oauthClient.restore(session.did).catch(async (err) => {
140 ctx.logger.warn({ err }, 'oauth restore failed')
141 await session.destroy()
142 return null
143 })
144}
145```
146
147Users publish JSON records on their `at://` repos. In [Bluesky](https://bsky.app), they publish a "profile" record which looks like this:
148
149```typescript
150interface ProfileRecord {
151 displayName?: string // a human friendly name
152 description?: string // a short bio
153 avatar?: BlobRef // small profile picture
154 banner?: BlobRef // banner image to put on profiles
155 createdAt?: string // declared time this profile data was added
156 // ...
157}
158```
159
160We're going to use the [Agent](#todo) to fetch this record to include in our app.
161
162```typescript
163/** src/routes.ts **/
164// Homepage
165router.get(
166 '/',
167 handler(async (req, res) => {
168 // If the user is signed in, get an agent which communicates with their server
169 const agent = await getSessionAgent(req, res, ctx)
170
171 if (!agent) {
172 // Serve the logged-out view
173 return res.type('html').send(page(home()))
174 }
175
176 // Fetch additional information about the logged-in user
177 const { data: profileRecord } = await agent.getRecord({
178 repo: agent.accountDid, // our user's repo
179 collection: 'app.bsky.actor.profile', // the bluesky profile record type
180 rkey: 'self', // the record's name
181 })
182
183 // Serve the logged-in view
184 return res
185 .type('html')
186 .send(page(home({ profile: profileRecord.value || {} })))
187 })
188)
189```
190
191With that data, we can give a nice personalized welcome banner for our user:
192
193```html
194<!-- pages/home.ts -->
195<div class="card">
196 ${profile
197 ? html`<form action="/logout" method="post" class="session-form">
198 <div>
199 Hi, <strong>${profile.displayName || 'friend'}</strong>.
200 What's your status today?
201 </div>
202 <div>
203 <button type="submit">Log out</button>
204 </div>
205 </form>`
206 : html`<div class="session-form">
207 <div><a href="/login">Log in</a> to set your status!</div>
208 <div>
209 <a href="/login" class="button">Log in</a>
210 </div>
211 </div>`}
212</div>
213```
214
215## Step 4. Reading & writing records
216
217You can think of the user repositories as collections of JSON records:
218
219```
220 ┌────────┐
221 ┌───| record │
222 ┌────────────┐ │ └────────┘
223 ┌───| collection |◄─┤ ┌────────┐
224┌──────┐ │ └────────────┘ └───| record │
225│ repo |◄──┤ └────────┘
226└──────┘ │ ┌────────────┐ ┌────────┐
227 └───┤ collection |◄─────| record │
228 └────────────┘ └────────┘
229```
230
231Let's look again at how we read the "profile" record:
232
233```typescript
234await agent.getRecord({
235 repo: agent.accountDid, // The user
236 collection: 'app.bsky.actor.profile', // The collection
237 rkey: 'self', // The record name
238})
239```
240
241We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
242
243```typescript
244await agent.putRecord({
245 repo: agent.accountDid, // The user
246 collection: 'com.example.status', // The collection
247 rkey: 'self', // The record name
248 record: { // The record value
249 status: "👍",
250 updatedAt: new Date().toISOString()
251 }
252})
253```
254
255Our `POST /status` route is going to use this API to publish the user's status to their repo.
256
257```typescript
258/** src/routes.ts **/
259// "Set status" handler
260router.post(
261 '/status',
262 handler(async (req, res) => {
263 // If the user is signed in, get an agent which communicates with their server
264 const agent = await getSessionAgent(req, res, ctx)
265 if (!agent) {
266 return res.status(401).json({ error: 'Session required' })
267 }
268
269 // Construct their status record
270 const record = {
271 $type: 'com.example.status',
272 status: req.body?.status,
273 updatedAt: new Date().toISOString(),
274 }
275
276 try {
277 // Write the status record to the user's repository
278 await agent.putRecord({
279 repo: agent.accountDid,
280 collection: 'com.example.status',
281 rkey: 'self',
282 record,
283 })
284 } catch (err) {
285 ctx.logger.warn({ err }, 'failed to write record')
286 return res.status(500).json({ error: 'Failed to write record' })
287 }
288
289 res.status(200).json({})
290 })
291)
292```
293
294Now in our homepage we can list out the status buttons:
295
296```html
297<!-- src/pages/home.ts -->
298<div class="status-options">
299 ${['👍', '🦋', '🥳', /*...*/].map(status => html`
300 <div class="status-option" data-value="${status}">
301 ${status}
302 </div>`
303 )}
304</div>
305```
306
307And write some client-side javascript to submit the status on click:
308
309```javascript
310/* src/pages/public/home.js */
311Array.from(document.querySelectorAll('.status-option'), (el) => {
312 el.addEventListener('click', async (ev) => {
313 const res = await fetch('/status', {
314 method: 'POST',
315 headers: { 'content-type': 'application/json' },
316 body: JSON.stringify({ status: el.dataset.value }),
317 })
318 const body = await res.json()
319 if (!body?.error) {
320 location.reload()
321 }
322 })
323})
324```
325
326## Step 5. Creating a custom "status" schema
327
328The collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
329
330Anybody can create a new schema using the [Lexicon](#todo) language, which is very similar to [JSON-Schema](#todo). The schemas use [reverse-DNS IDs](#todo) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
331
332> ### Why create a schema?
333>
334> Schemas help other applications understand the data your app is creating. By publishing your schemas, you enable compatibility and reduce the chances of bad data affecting your app.
335
336Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](#todo).
337
338```json
339/* lexicons/status.json */
340{
341 "lexicon": 1,
342 "id": "com.example.status",
343 "defs": {
344 "main": {
345 "type": "record",
346 "key": "literal:self",
347 "record": {
348 "type": "object",
349 "required": ["status", "updatedAt"],
350 "properties": {
351 "status": {
352 "type": "string",
353 "minLength": 1,
354 "maxGraphemes": 1,
355 "maxLength": 32
356 },
357 "updatedAt": {
358 "type": "string",
359 "format": "datetime"
360 }
361 }
362 }
363 }
364 }
365}
366```
367
368Now let's run some code-generation using our schema:
369
370```bash
371./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
372```
373
374This will produce Typescript interfaces as well as runtime validation functions that we can use in our `POST /status` route:
375
376```typescript
377/** src/routes.ts **/
378import * as Status from '#/lexicon/types/com/example/status'
379// ...
380// "Set status" handler
381router.post(
382 '/status',
383 handler(async (req, res) => {
384 // ...
385
386 // Construct & validate their status record
387 const record = {
388 $type: 'com.example.status',
389 status: req.body?.status,
390 updatedAt: new Date().toISOString(),
391 }
392 if (!Status.validateRecord(record).success) {
393 return res.status(400).json({ error: 'Invalid status' })
394 }
395
396 // ...
397 })
398)
399```
400
401## Step 6. Listening to the firehose
402
403So far, we have:
404
405- Logged in via OAuth
406- Created a custom schema
407- Read & written records for the logged in user
408
409Now we want to fetch the status records from other users.
410
411Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
412
413```
414┌──────┐
415│ REPO │ Event stream
416├──────┘
417│ ┌───────────────────────────────────────────┐
418├───┼ 1 PUT /com.example.status/self │
419│ └───────────────────────────────────────────┘
420│ ┌───────────────────────────────────────────┐
421├───┼ 2 DEL /app.bsky.feed.post/3l244rmrxjx2v │
422│ └───────────────────────────────────────────┘
423│ ┌───────────────────────────────────────────┐
424├───┼ 3 PUT /app.bsky.actor/self │
425▼ └───────────────────────────────────────────┘
426```
427
428Using a [Relay service](#todo) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
429
430
431```typescript
432/** src/firehose.ts **/
433import * as Status from '#/lexicon/types/com/example/status'
434// ...
435const firehose = new Firehose({})
436
437for await (const evt of firehose.run()) {
438 // Watch for write events
439 if (evt.event === 'create' || evt.event === 'update') {
440 const record = evt.record
441
442 // If the write is a valid status update
443 if (
444 evt.collection === 'com.example.status' &&
445 Status.isRecord(record) &&
446 Status.validateRecord(record).success
447 ) {
448 // Store the status
449 // TODO
450 }
451 }
452}
453```
454
455Let's create a SQLite table to store these statuses:
456
457```typescript
458/** src/db.ts **/
459// Create our statuses table
460await db.schema
461 .createTable('status')
462 .addColumn('authorDid', 'varchar', (col) => col.primaryKey())
463 .addColumn('status', 'varchar', (col) => col.notNull())
464 .addColumn('updatedAt', 'varchar', (col) => col.notNull())
465 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
466 .execute()
467```
468
469Now we can write these statuses into our database as they arrive from the firehose:
470
471```typescript
472/** src/firehose.ts **/
473// If the write is a valid status update
474if (
475 evt.collection === 'com.example.status' &&
476 Status.isRecord(record) &&
477 Status.validateRecord(record).success
478) {
479 // Store the status in our SQLite
480 await db
481 .insertInto('status')
482 .values({
483 authorDid: evt.author,
484 status: record.status,
485 updatedAt: record.updatedAt,
486 indexedAt: new Date().toISOString(),
487 })
488 .onConflict((oc) =>
489 oc.column('authorDid').doUpdateSet({
490 status: record.status,
491 updatedAt: record.updatedAt,
492 indexedAt: new Date().toISOString(),
493 })
494 )
495 .execute()
496}
497```
498
499You can almost think of information flowing in a loop:
500
501```
502 ┌─────Repo put─────┐
503 │ ▼
504┌──────┴─────┐ ┌───────────┐
505│ App server │ │ User repo │
506└────────────┘ └─────┬─────┘
507 ▲ │
508 └────Event log─────┘
509```
510
511Why read from the event log? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in -- including data published by other apps.
512
513## Step 7. Listing the latest statuses
514
515Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](#todo)-to-handle resolver so we can show a nice username with the statuses:
516
517```typescript
518/** src/routes.ts **/
519// Homepage
520router.get(
521 '/',
522 handler(async (req, res) => {
523 // ...
524
525 // Fetch data stored in our SQLite
526 const statuses = await db
527 .selectFrom('status')
528 .selectAll()
529 .orderBy('indexedAt', 'desc')
530 .limit(10)
531 .execute()
532
533 // Map user DIDs to their domain-name handles
534 const didHandleMap = await resolver.resolveDidsToHandles(
535 statuses.map((s) => s.authorDid)
536 )
537
538 // ...
539 })
540)
541```
542
543Our HTML can now list these status records:
544
545```html
546<!-- src/pages/home.ts -->
547${statuses.map((status, i) => {
548 const handle = didHandleMap[status.authorDid] || status.authorDid
549 const date = ts(status)
550 return html`
551 <div class="status-line">
552 <div>
553 <div class="status">${status.status}</div>
554 </div>
555 <div class="desc">
556 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
557 was feeling ${status.status} on ${status.indexedAt}.
558 </div>
559 </div>
560 `
561})}
562```
563
564## Step 8. Optimistic updates
565
566As a final optimization, let's introduce "optimistic updates." Remember the information flow loop with the repo write and the event log? Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
567
568```
569 ┌───Repo put──┬──────┐
570 │ │ ▼
571┌──────┴─────┐ │ ┌───────────┐
572│ App server │◄──────┘ │ User repo │
573└────────────┘ └───┬───────┘
574 ▲ │
575 └────Event log───────┘
576```
577
578This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
579
580To do this, we just update `POST /status` to include an additional write to our SQLite DB:
581
582```typescript
583/** src/routes.ts **/
584// "Set status" handler
585router.post(
586 '/status',
587 handler(async (req, res) => {
588 // ...
589
590 try {
591 // Write the status record to the user's repository
592 await agent.putRecord({
593 repo: agent.accountDid,
594 collection: 'com.example.status',
595 rkey: 'self',
596 record,
597 })
598 } catch (err) {
599 ctx.logger.warn({ err }, 'failed to write record')
600 return res.status(500).json({ error: 'Failed to write record' })
601 }
602
603 try {
604 // Optimistically update our SQLite <-- HERE!
605 await ctx.db
606 .insertInto('status')
607 .values({
608 authorDid: agent.accountDid,
609 status: record.status,
610 updatedAt: record.updatedAt,
611 indexedAt: new Date().toISOString(),
612 })
613 .onConflict((oc) =>
614 oc.column('authorDid').doUpdateSet({
615 status: record.status,
616 updatedAt: record.updatedAt,
617 indexedAt: new Date().toISOString(),
618 })
619 )
620 .execute()
621 } catch (err) {
622 ctx.logger.warn(
623 { err },
624 'failed to update computed view; ignoring as it should be caught by the firehose'
625 )
626 }
627
628 res.status(200).json({})
629 })
630)
631```
632
633You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
634
635## Next steps
636
637TODO
638
639