Scratch space for learning atproto app development
cduck.me
1# Tutorial
2
3In this guide, we're going to build a **simple multi-user app** that publishes your current "status" as an emoji.
4
5
6
7At various points we will cover how to:
8
9- Signin via OAuth
10- Fetch information about users (profiles)
11- Listen to the network firehose for new data
12- Publish data on the user's account using a custom schema
13
14We're going to keep this light so you can quickly wrap your head around ATProto. There will be links with more information about each step.
15
16## Where are we going?
17
18Data in the Atmosphere is stored on users' personal repos. It's almost like each user has their own website. Our goal is to aggregate data from the users into our SQLite DB.
19
20Think of our app like a Google. If Google's job was to say which emoji each website had under `/status.json`, then it would show something like:
21
22- `nytimes.com` is feeling 📰 according to `https://nytimes.com/status.json`
23- `bsky.app` is feeling 🦋 according to `https://bsky.app/status.json`
24- `reddit.com` is feeling 🤓 according to `https://reddit.com/status.json`
25
26The Atmosphere works the same way, except we're going to check `at://` instead of `https://`. Each user has a data repo under an `at://` URL. We'll crawl all the `at://`s in the Atmosphere for all the "status.json" records and aggregate them into our SQLite database.
27
28> `at://` is the URL scheme of the AT Protocol.
29
30## Step 1. Starting with our ExpressJS app
31
32Start by cloning the repo and installing packages.
33
34```bash
35git clone TODO
36cd TODO
37npm i
38npm run dev # you can leave this running and it will auto-reload
39```
40
41Our repo is a regular Web app. We're rendering our HTML server-side like it's 1999. We also have a SQLite database that we're managing with [Kysley](#todo).
42
43Our starting stack:
44
45- Typescript
46- NodeJS web server ([express](#todo))
47- SQLite database ([Kysley](#todo))
48- Server-side rendering ([uhtml](#todo))
49
50With each step we'll explain how our Web app taps into the Atmosphere. Refer to the codebase for more detailed code — again, this tutorial is going to keep it light and quick to digest.
51
52## Step 2. Signing in with OAuth
53
54When somebody logs into our app, they'll give us read & write access to their personal `at://` repo. We'll use that to write the `status.json` record.
55
56We're going to accomplish this using OAuth ([spec](#todo)). You can find a [more extensive OAuth guide here](#todo), but for now just know that most of the OAuth flows are going to be handled for us using the [@atproto/oauth-client-node](#todo) library. This is the arrangement we're aiming toward:
57
58
59
60When the user logs in, the OAuth client will create a new session with their repo server and give us read/write access along with basic user info.
61
62
63
64Our login page just asks the user for their "handle," which is the domain name associated with their account. For [Bluesky](https://bsky.app) users, these tend to look like `alice.bsky.social`, but they can be any kind of domain (eg `alice.com`).
65
66```html
67<!-- src/pages/login.ts -->
68<form action="/login" method="post" class="login-form">
69 <input
70 type="text"
71 name="handle"
72 placeholder="Enter your handle (eg alice.bsky.social)"
73 required
74 />
75 <button type="submit">Log in</button>
76</form>
77```
78
79When they submit the form, we tell our OAuth client to initiate the authorization flow and then redirect the user to their server to complete the process.
80
81```typescript
82/** src/routes.ts **/
83// Login handler
84router.post(
85 '/login',
86 handler(async (req, res) => {
87 // Initiate the OAuth flow
88 const url = await oauthClient.authorize(handle)
89 return res.redirect(url.toString())
90 })
91)
92```
93
94This is the same kind of SSO flow that Google or GitHub uses. The user will be asked for their password, then asked to confirm the session with your application.
95
96When that finishes, they'll be sent back to `/oauth/callback` on our Web app. The OAuth client stores the access tokens for the server, and then we attach their account's [DID](#todo) to their cookie-session.
97
98```typescript
99/** src/routes.ts **/
100// OAuth callback to complete session creation
101router.get(
102 '/oauth/callback',
103 handler(async (req, res) => {
104 // Store the credentials
105 const { agent } = await oauthClient.callback(params)
106
107 // Attach the account DID to our user via a cookie
108 const session = await getIronSession(req, res)
109 session.did = agent.accountDid
110 await session.save()
111
112 // Send them back to the app
113 return res.redirect('/')
114 })
115)
116```
117
118With that, we're in business! We now have a session with the user's `at://` repo server and can use that to access their data.
119
120## Step 3. Fetching the user's profile
121
122Why don't we learn something about our user? Let's start by getting the [Agent](#todo) object. The [Agent](#todo) is the client to the user's `at://` repo server.
123
124```typescript
125/** src/routes.ts **/
126async function getSessionAgent(
127 req: IncomingMessage,
128 res: ServerResponse<IncomingMessage>,
129 ctx: AppContext
130) {
131 // Fetch the session from their cookie
132 const session = await getIronSession(req, res)
133 if (!session.did) return null
134
135 // "Restore" the agent for the user
136 return await ctx.oauthClient.restore(session.did).catch(async (err) => {
137 ctx.logger.warn({ err }, 'oauth restore failed')
138 await session.destroy()
139 return null
140 })
141}
142```
143
144Users publish JSON records on their `at://` repos. In [Bluesky](https://bsky.app), they publish a "profile" record which looks like this:
145
146```typescript
147interface ProfileRecord {
148 displayName?: string // a human friendly name
149 description?: string // a short bio
150 avatar?: BlobRef // small profile picture
151 banner?: BlobRef // banner image to put on profiles
152 createdAt?: string // declared time this profile data was added
153 // ...
154}
155```
156
157We're going to use the [Agent](#todo) to fetch this record to include in our app.
158
159```typescript
160await agent.getRecord({
161 repo: agent.accountDid, // The user
162 collection: 'app.bsky.actor.profile', // The collection
163 rkey: 'self', // The record key
164})
165```
166
167When asking for a record, we provide three pieces of information.
168
169- The [DID](#todo) which identifies the user,
170- The collection name, and
171- The record key
172
173We'll explain the collection name shortly. Record keys are strings with [some limitations](https://atproto.com/specs/record-key#record-key-syntax) and a couple of common patterns. The `"self"` pattern is used when a collection is expected to only contain one record which describes the user.
174
175Let's update our homepage to fetch this profile record:
176
177```typescript
178/** src/routes.ts **/
179// Homepage
180router.get(
181 '/',
182 handler(async (req, res) => {
183 // If the user is signed in, get an agent which communicates with their server
184 const agent = await getSessionAgent(req, res, ctx)
185
186 if (!agent) {
187 // Serve the logged-out view
188 return res.type('html').send(page(home()))
189 }
190
191 // Fetch additional information about the logged-in user
192 const { data: profileRecord } = await agent.getRecord({
193 repo: agent.accountDid, // our user's repo
194 collection: 'app.bsky.actor.profile', // the bluesky profile record type
195 rkey: 'self', // the record's name
196 })
197
198 // Serve the logged-in view
199 return res
200 .type('html')
201 .send(page(home({ profile: profileRecord.value || {} })))
202 })
203)
204```
205
206With that data, we can give a nice personalized welcome banner for our user:
207
208```html
209<!-- pages/home.ts -->
210<div class="card">
211 ${profile
212 ? html`<form action="/logout" method="post" class="session-form">
213 <div>
214 Hi, <strong>${profile.displayName || 'friend'}</strong>.
215 What's your status today?
216 </div>
217 <div>
218 <button type="submit">Log out</button>
219 </div>
220 </form>`
221 : html`<div class="session-form">
222 <div><a href="/login">Log in</a> to set your status!</div>
223 <div>
224 <a href="/login" class="button">Log in</a>
225 </div>
226 </div>`}
227</div>
228```
229
230
231
232You can examine this record directly using [atproto-browser.vercel.app](https://atproto-browser.vercel.app). For instance, [this is the profile record for @bsky.app](https://atproto-browser.vercel.app/at?u=at://did:plc:z72i7hdynmk6r22z27h6tvur/app.bsky.actor.profile/self).
233
234## Step 4. Reading & writing records
235
236You can think of the user repositories as collections of JSON records:
237
238
239
240Let's look again at how we read the "profile" record:
241
242```typescript
243await agent.getRecord({
244 repo: agent.accountDid, // The user
245 collection: 'app.bsky.actor.profile', // The collection
246 rkey: 'self', // The record key
247})
248```
249
250We write records using a similar API. Since our goal is to write "status" records, let's look at how that will happen:
251
252```typescript
253// Generate a time-based key for our record
254const rkey = TID.nextStr()
255
256// Write the
257await agent.putRecord({
258 repo: agent.accountDid, // The user
259 collection: 'com.example.status', // The collection
260 rkey, // The record key
261 record: { // The record value
262 status: "👍",
263 createdAt: new Date().toISOString()
264 }
265})
266```
267
268Our `POST /status` route is going to use this API to publish the user's status to their repo.
269
270```typescript
271/** src/routes.ts **/
272// "Set status" handler
273router.post(
274 '/status',
275 handler(async (req, res) => {
276 // If the user is signed in, get an agent which communicates with their server
277 const agent = await getSessionAgent(req, res, ctx)
278 if (!agent) {
279 return res.status(401).json({ error: 'Session required' })
280 }
281
282 // Construct their status record
283 const record = {
284 $type: 'com.example.status',
285 status: req.body?.status,
286 createdAt: new Date().toISOString(),
287 }
288
289 try {
290 // Write the status record to the user's repository
291 await agent.putRecord({
292 repo: agent.accountDid,
293 collection: 'com.example.status',
294 rkey: TID.nextStr(),
295 record,
296 })
297 } catch (err) {
298 logger.warn({ err }, 'failed to write record')
299 return res.status(500).json({ error: 'Failed to write record' })
300 }
301
302 res.status(200).json({})
303 })
304)
305```
306
307Now in our homepage we can list out the status buttons:
308
309```html
310<!-- src/pages/home.ts -->
311<div class="status-options">
312 ${['👍', '🦋', '🥳', /*...*/].map(status => html`
313 <div class="status-option" data-value="${status}">
314 ${status}
315 </div>`
316 )}
317</div>
318```
319
320And write some client-side javascript to submit the status on click:
321
322```javascript
323/* src/pages/public/home.js */
324Array.from(document.querySelectorAll('.status-option'), (el) => {
325 el.addEventListener('click', async (ev) => {
326 const res = await fetch('/status', {
327 method: 'POST',
328 headers: { 'content-type': 'application/json' },
329 body: JSON.stringify({ status: el.dataset.value }),
330 })
331 const body = await res.json()
332 if (!body?.error) {
333 location.reload()
334 }
335 })
336})
337```
338
339And here we are!
340
341
342
343## Step 5. Creating a custom "status" schema
344
345The collections are typed, meaning that they have a defined schema. The `app.bsky.actor.profile` type definition [can be found here](https://github.com/bluesky-social/atproto/blob/main/lexicons/app/bsky/actor/profile.json).
346
347Anybody can create a new schema using the [Lexicon](#todo) language, which is very similar to [JSON-Schema](#todo). The schemas use [reverse-DNS IDs](#todo) which indicate ownership, but for this demo app we're going to use `com.example` which is safe for non-production software.
348
349> ### Why create a schema?
350>
351> Schemas help other applications understand the data your app is creating. By publishing your schemas, you enable compatibility with other apps and reduce the chances of bad data affecting your app.
352
353Let's create our schema in the `/lexicons` folder of our codebase. You can [read more about how to define schemas here](#todo).
354
355```json
356/* lexicons/status.json */
357{
358 "lexicon": 1,
359 "id": "com.example.status",
360 "defs": {
361 "main": {
362 "type": "record",
363 "key": "tid",
364 "record": {
365 "type": "object",
366 "required": ["status", "createdAt"],
367 "properties": {
368 "status": {
369 "type": "string",
370 "minLength": 1,
371 "maxGraphemes": 1,
372 "maxLength": 32
373 },
374 "createdAt": {
375 "type": "string",
376 "format": "datetime"
377 }
378 }
379 }
380 }
381 }
382}
383```
384
385Now let's run some code-generation using our schema:
386
387```bash
388./node_modules/.bin/lex gen-server ./src/lexicon ./lexicons/*
389```
390
391This will produce Typescript interfaces as well as runtime validation functions that we can use in our `POST /status` route:
392
393```typescript
394/** src/routes.ts **/
395import * as Status from '#/lexicon/types/com/example/status'
396// ...
397// "Set status" handler
398router.post(
399 '/status',
400 handler(async (req, res) => {
401 // ...
402
403 // Construct & validate their status record
404 const record = {
405 $type: 'com.example.status',
406 status: req.body?.status,
407 createdAt: new Date().toISOString(),
408 }
409 if (!Status.validateRecord(record).success) {
410 return res.status(400).json({ error: 'Invalid status' })
411 }
412
413 // ...
414 })
415)
416```
417
418## Step 6. Listening to the firehose
419
420So far, we have:
421
422- Logged in via OAuth
423- Created a custom schema
424- Read & written records for the logged in user
425
426Now we want to fetch the status records from other users.
427
428Remember how we referred to our app as being like a Google, crawling around the repos to get their records? One advantage we have in the AT Protocol is that each repo publishes an event log of their updates.
429
430
431
432Using a [Relay service](#todo) we can listen to an aggregated firehose of these events across all users in the network. In our case what we're looking for are valid `com.example.status` records.
433
434
435```typescript
436/** src/firehose.ts **/
437import * as Status from '#/lexicon/types/com/example/status'
438// ...
439const firehose = new Firehose({})
440
441for await (const evt of firehose.run()) {
442 // Watch for write events
443 if (evt.event === 'create' || evt.event === 'update') {
444 const record = evt.record
445
446 // If the write is a valid status update
447 if (
448 evt.collection === 'com.example.status' &&
449 Status.isRecord(record) &&
450 Status.validateRecord(record).success
451 ) {
452 // Store the status
453 // TODO
454 }
455 }
456}
457```
458
459Let's create a SQLite table to store these statuses:
460
461```typescript
462/** src/db.ts **/
463// Create our statuses table
464await db.schema
465 .createTable('status')
466 .addColumn('uri', 'varchar', (col) => col.primaryKey())
467 .addColumn('authorDid', 'varchar', (col) => col.notNull())
468 .addColumn('status', 'varchar', (col) => col.notNull())
469 .addColumn('createdAt', 'varchar', (col) => col.notNull())
470 .addColumn('indexedAt', 'varchar', (col) => col.notNull())
471 .execute()
472```
473
474Now we can write these statuses into our database as they arrive from the firehose:
475
476```typescript
477/** src/firehose.ts **/
478// If the write is a valid status update
479if (
480 evt.collection === 'com.example.status' &&
481 Status.isRecord(record) &&
482 Status.validateRecord(record).success
483) {
484 // Store the status in our SQLite
485 await db
486 .insertInto('status')
487 .values({
488 uri: evt.uri.toString(),
489 authorDid: evt.author,
490 status: record.status,
491 createdAt: record.createdAt,
492 indexedAt: new Date().toISOString(),
493 })
494 .onConflict((oc) =>
495 oc.column('uri').doUpdateSet({
496 status: record.status,
497 indexedAt: new Date().toISOString(),
498 })
499 )
500 .execute()
501}
502```
503
504You can almost think of information flowing in a loop:
505
506
507
508Why read from the event log? Because there are other apps in the network that will write the records we're interested in. By subscribing to the event log, we ensure that we catch all the data we're interested in -- including data published by other apps.
509
510## Step 7. Listing the latest statuses
511
512Now that we have statuses populating our SQLite, we can produce a timeline of status updates by users. We also use a [DID](#todo)-to-handle resolver so we can show a nice username with the statuses:
513
514```typescript
515/** src/routes.ts **/
516// Homepage
517router.get(
518 '/',
519 handler(async (req, res) => {
520 // ...
521
522 // Fetch data stored in our SQLite
523 const statuses = await db
524 .selectFrom('status')
525 .selectAll()
526 .orderBy('indexedAt', 'desc')
527 .limit(10)
528 .execute()
529
530 // Map user DIDs to their domain-name handles
531 const didHandleMap = await resolver.resolveDidsToHandles(
532 statuses.map((s) => s.authorDid)
533 )
534
535 // ...
536 })
537)
538```
539
540Our HTML can now list these status records:
541
542```html
543<!-- src/pages/home.ts -->
544${statuses.map((status, i) => {
545 const handle = didHandleMap[status.authorDid] || status.authorDid
546 return html`
547 <div class="status-line">
548 <div>
549 <div class="status">${status.status}</div>
550 </div>
551 <div class="desc">
552 <a class="author" href="https://bsky.app/profile/${handle}">@${handle}</a>
553 was feeling ${status.status} on ${status.indexedAt}.
554 </div>
555 </div>
556 `
557})}
558```
559
560
561
562## Step 8. Optimistic updates
563
564As a final optimization, let's introduce "optimistic updates." Remember the information flow loop with the repo write and the event log? Since we're updating our users' repos locally, we can short-circuit that flow to our own database:
565
566
567
568This is an important optimization to make, because it ensures that the user sees their own changes while using your app. When the event eventually arrives from the firehose, we just discard it since we already have it saved locally.
569
570To do this, we just update `POST /status` to include an additional write to our SQLite DB:
571
572```typescript
573/** src/routes.ts **/
574// "Set status" handler
575router.post(
576 '/status',
577 handler(async (req, res) => {
578 // ...
579
580 let uri
581 try {
582 // Write the status record to the user's repository
583 const res = await agent.putRecord({
584 repo: agent.accountDid,
585 collection: 'com.example.status',
586 rkey: TID.nextStr(),
587 record,
588 })
589 uri = res.uri
590 } catch (err) {
591 logger.warn({ err }, 'failed to write record')
592 return res.status(500).json({ error: 'Failed to write record' })
593 }
594
595 try {
596 // Optimistically update our SQLite <-- HERE!
597 await db
598 .insertInto('status')
599 .values({
600 uri,
601 authorDid: agent.accountDid,
602 status: record.status,
603 createdAt: record.createdAt,
604 indexedAt: new Date().toISOString(),
605 })
606 .execute()
607 } catch (err) {
608 logger.warn(
609 { err },
610 'failed to update computed view; ignoring as it should be caught by the firehose'
611 )
612 }
613
614 res.status(200).json({})
615 })
616)
617```
618
619You'll notice this code looks almost exactly like what we're doing in `firehose.ts`.
620
621## Thinking in AT Proto
622
623In this tutorial we've covered the key steps to building an atproto app. Data is published in its canonical form on users' `at://` repos and then aggregated into apps' databases to produce views of the network.
624
625When building your app, think in these four key steps:
626
627- Design the [Lexicon](#) schemas for the records you'll publish into the Atmosphere.
628- Create a database for aggregating the records into useful views.
629- Build your application to write the records on your users' repos.
630- Listen to the firehose to hydrate your aggregated database.
631
632Remember this flow of information throughout:
633
634
635
636This is how every app in the Atmosphere works, including the [Bluesky social app](https://bsky.app).
637
638## Next steps
639
640TODO
641
642