A community based topic aggregation platform built on atproto
bretton.dev
1# Aggregator Setup Guide
2
3This guide explains how to set up and register an aggregator with Coves instances.
4
5## Table of Contents
6
7- [Overview](#overview)
8- [Architecture](#architecture)
9- [Prerequisites](#prerequisites)
10- [Quick Start](#quick-start)
11- [Detailed Setup Steps](#detailed-setup-steps)
12- [Authorization Process](#authorization-process)
13- [Posting to Communities](#posting-to-communities)
14- [Rate Limits](#rate-limits)
15- [Security Best Practices](#security-best-practices)
16- [Troubleshooting](#troubleshooting)
17- [API Reference](#api-reference)
18
19## Overview
20
21**Aggregators** are automated services that post content to Coves communities. They are similar to Bluesky's feed generators and labelers - self-managed external services that integrate with the platform.
22
23**Key characteristics**:
24- Self-owned: You create and manage your own PDS account
25- Domain-verified: Prove ownership via `.well-known/atproto-did`
26- Community-authorized: Moderators grant posting permission per-community
27- Rate-limited: 10 posts per hour per community
28
29**Example use cases**:
30- RSS feed aggregators (tech news, blog posts)
31- Social media cross-posters (Twitter → Coves)
32- Event notifications (GitHub releases, weather alerts)
33- Content curation bots (daily links, summaries)
34
35## Architecture
36
37### Data Flow
38
39```
40┌──────────────────────────────────────────────────────────┐
41│ 1. One-Time Setup │
42├──────────────────────────────────────────────────────────┤
43│ Aggregator creates PDS account │
44│ ↓ │
45│ Proves domain ownership (.well-known) │
46│ ↓ │
47│ Registers with Coves (enters users table) │
48│ ↓ │
49│ Writes service declaration │
50│ ↓ │
51│ Jetstream indexes into aggregators table │
52└──────────────────────────────────────────────────────────┘
53
54┌──────────────────────────────────────────────────────────┐
55│ 2. Per-Community Authorization │
56├──────────────────────────────────────────────────────────┤
57│ Moderator writes authorization record │
58│ ↓ │
59│ Jetstream indexes into aggregator_authorizations │
60└──────────────────────────────────────────────────────────┘
61
62┌──────────────────────────────────────────────────────────┐
63│ 3. Posting (Ongoing) │
64├──────────────────────────────────────────────────────────┤
65│ Aggregator calls post creation endpoint │
66│ ↓ │
67│ Handler validates: │
68│ - Author in users table ✓ │
69│ - Author in aggregators table ✓ │
70│ - Authorization exists ✓ │
71│ - Rate limit not exceeded ✓ │
72│ ↓ │
73│ Post written to community's PDS │
74│ ↓ │
75│ Jetstream indexes post │
76└──────────────────────────────────────────────────────────┘
77```
78
79### Database Tables
80
81**users** - All actors (users, communities, aggregators)
82```sql
83CREATE TABLE users (
84 did TEXT PRIMARY KEY,
85 handle TEXT NOT NULL,
86 pds_url TEXT,
87 indexed_at TIMESTAMPTZ
88);
89```
90
91**aggregators** - Aggregator-specific metadata
92```sql
93CREATE TABLE aggregators (
94 did TEXT PRIMARY KEY,
95 display_name TEXT NOT NULL,
96 description TEXT,
97 avatar_url TEXT,
98 config_schema JSONB,
99 source_url TEXT,
100 maintainer_did TEXT,
101 record_uri TEXT NOT NULL UNIQUE,
102 record_cid TEXT NOT NULL,
103 created_at TIMESTAMPTZ,
104 indexed_at TIMESTAMPTZ
105);
106```
107
108**aggregator_authorizations** - Community authorizations
109```sql
110CREATE TABLE aggregator_authorizations (
111 id BIGSERIAL PRIMARY KEY,
112 aggregator_did TEXT NOT NULL,
113 community_did TEXT NOT NULL,
114 enabled BOOLEAN NOT NULL DEFAULT true,
115 config JSONB,
116 created_by TEXT,
117 record_uri TEXT NOT NULL UNIQUE,
118 record_cid TEXT NOT NULL,
119 UNIQUE(aggregator_did, community_did)
120);
121```
122
123## Prerequisites
124
1251. **Domain ownership**: You must own a domain where you can host static files over HTTPS
1262. **Web server**: Ability to serve the `.well-known/atproto-did` file
1273. **Development tools**: `curl`, `jq`, basic shell scripting knowledge
1284. **Email address**: For creating the PDS account
129
130**Optional**:
131- Custom avatar image (PNG/JPEG/WebP, max 1MB)
132- GitHub repository for source code transparency
133
134## Quick Start
135
136We provide automated setup scripts:
137
138```bash
139cd scripts/aggregator-setup
140
141# Make scripts executable
142chmod +x *.sh
143
144# Run setup scripts in order
145./1-create-pds-account.sh
146./2-setup-wellknown.sh
147# (Upload .well-known to your web server)
148./3-register-with-coves.sh
149./4-create-service-declaration.sh
150```
151
152See [scripts/aggregator-setup/README.md](../../scripts/aggregator-setup/README.md) for detailed script documentation.
153
154## Detailed Setup Steps
155
156### Step 1: Create PDS Account
157
158Your aggregator needs its own atProto identity (DID). The easiest way is to create an account on an existing PDS.
159
160**Using an existing PDS (recommended)**:
161
162```bash
163curl -X POST https://bsky.social/xrpc/com.atproto.server.createAccount \
164 -H "Content-Type: application/json" \
165 -d '{
166 "handle": "mynewsbot.bsky.social",
167 "email": "bot@example.com",
168 "password": "secure-password-here"
169 }'
170```
171
172**Response**:
173```json
174{
175 "accessJwt": "eyJ...",
176 "refreshJwt": "eyJ...",
177 "handle": "mynewsbot.bsky.social",
178 "did": "did:plc:abc123...",
179 "didDoc": {...}
180}
181```
182
183**Save these credentials securely!** You'll need the DID and access token for all subsequent operations.
184
185**Alternative**: Run your own PDS or use `did:web` (advanced).
186
187### Step 2: Prove Domain Ownership
188
189To register with Coves, you must prove you own a domain by serving your DID at `https://yourdomain.com/.well-known/atproto-did`.
190
191**Create the file**:
192
193```bash
194mkdir -p .well-known
195echo "did:plc:abc123..." > .well-known/atproto-did
196```
197
198**Upload to your web server** so it's accessible at:
199```
200https://rss-bot.example.com/.well-known/atproto-did
201```
202
203**Verify it works**:
204```bash
205curl https://rss-bot.example.com/.well-known/atproto-did
206# Should return: did:plc:abc123...
207```
208
209**Nginx configuration example**:
210```nginx
211location /.well-known/atproto-did {
212 alias /var/www/.well-known/atproto-did;
213 default_type text/plain;
214 add_header Access-Control-Allow-Origin *;
215}
216```
217
218### Step 3: Register with Coves
219
220Call the registration endpoint to register your aggregator DID with the Coves instance.
221
222**Endpoint**: `POST /xrpc/social.coves.aggregator.register`
223
224**Request**:
225```bash
226curl -X POST https://api.coves.social/xrpc/social.coves.aggregator.register \
227 -H "Content-Type: application/json" \
228 -d '{
229 "did": "did:plc:abc123...",
230 "domain": "rss-bot.example.com"
231 }'
232```
233
234**Response** (Success):
235```json
236{
237 "did": "did:plc:abc123...",
238 "handle": "mynewsbot.bsky.social",
239 "message": "Aggregator registered successfully. Next step: create a service declaration record at at://did:plc:abc123.../social.coves.aggregator.service/self"
240}
241```
242
243**What happens**:
2441. Coves fetches `https://rss-bot.example.com/.well-known/atproto-did`
2452. Verifies it contains your DID
2463. Resolves your DID to get handle and PDS URL
2474. Inserts you into the `users` table
248
249**You're now registered!** But you need to create a service declaration next.
250
251### Step 4: Create Service Declaration
252
253Write a `social.coves.aggregator.service` record to your repository. This contains metadata about your aggregator and gets indexed by Coves' Jetstream consumer.
254
255**Endpoint**: `POST https://your-pds.com/xrpc/com.atproto.repo.createRecord`
256
257**Request**:
258```bash
259curl -X POST https://bsky.social/xrpc/com.atproto.repo.createRecord \
260 -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
261 -H "Content-Type: application/json" \
262 -d '{
263 "repo": "did:plc:abc123...",
264 "collection": "social.coves.aggregator.service",
265 "rkey": "self",
266 "record": {
267 "$type": "social.coves.aggregator.service",
268 "did": "did:plc:abc123...",
269 "displayName": "RSS News Aggregator",
270 "description": "Aggregates tech news from various RSS feeds",
271 "sourceUrl": "https://github.com/yourname/rss-aggregator",
272 "maintainer": "did:plc:your-personal-did",
273 "createdAt": "2024-01-15T12:00:00Z"
274 }
275 }'
276```
277
278**Response**:
279```json
280{
281 "uri": "at://did:plc:abc123.../social.coves.aggregator.service/self",
282 "cid": "bafyrei..."
283}
284```
285
286**Optional fields**:
287- `avatar`: Blob reference to avatar image
288- `configSchema`: JSON Schema for community-specific configuration
289
290**Wait 5-10 seconds** for Jetstream to index your service declaration into the `aggregators` table.
291
292## Authorization Process
293
294Before you can post to a community, a moderator must authorize your aggregator.
295
296### How Authorization Works
297
2981. **Moderator decision**: Community moderator evaluates your aggregator
2992. **Authorization record**: Moderator writes `social.coves.aggregator.authorization` to community's repo
3003. **Jetstream indexing**: Record gets indexed into `aggregator_authorizations` table
3014. **Posting enabled**: You can now post to that community
302
303### Authorization Record Structure
304
305**Location**: `at://{community_did}/social.coves.aggregator.authorization/{rkey}`
306
307**Example**:
308```json
309{
310 "$type": "social.coves.aggregator.authorization",
311 "aggregatorDid": "did:plc:abc123...",
312 "communityDid": "did:plc:community123...",
313 "enabled": true,
314 "createdBy": "did:plc:moderator...",
315 "createdAt": "2024-01-15T12:00:00Z",
316 "config": {
317 "maxPostsPerHour": 5,
318 "allowedCategories": ["tech", "news"]
319 }
320}
321```
322
323### Checking Your Authorizations
324
325**Endpoint**: `GET /xrpc/social.coves.aggregator.getAuthorizations`
326
327```bash
328curl "https://api.coves.social/xrpc/social.coves.aggregator.getAuthorizations?aggregatorDid=did:plc:abc123...&enabledOnly=true"
329```
330
331**Response**:
332```json
333{
334 "authorizations": [
335 {
336 "aggregatorDid": "did:plc:abc123...",
337 "communityDid": "did:plc:community123...",
338 "communityHandle": "~tech@coves.social",
339 "enabled": true,
340 "createdAt": "2024-01-15T12:00:00Z",
341 "config": {...}
342 }
343 ]
344}
345```
346
347## Posting to Communities
348
349Once authorized, you can post to communities using the standard post creation endpoint.
350
351### Create Post
352
353**Endpoint**: `POST /xrpc/social.coves.community.post.create`
354
355**Request**:
356```bash
357curl -X POST https://api.coves.social/xrpc/social.coves.community.post.create \
358 -H "Authorization: Bearer YOUR_ACCESS_TOKEN" \
359 -H "Content-Type: application/json" \
360 -d '{
361 "communityDid": "did:plc:community123...",
362 "post": {
363 "text": "New blog post: Understanding atProto Identity\nhttps://example.com/post",
364 "createdAt": "2024-01-15T12:00:00Z",
365 "facets": [
366 {
367 "index": { "byteStart": 50, "byteEnd": 75 },
368 "features": [
369 {
370 "$type": "social.coves.richtext.facet#link",
371 "uri": "https://example.com/post"
372 }
373 ]
374 }
375 ]
376 }
377 }'
378```
379
380**Response**:
381```json
382{
383 "uri": "at://did:plc:abc123.../social.coves.community.post/3k...",
384 "cid": "bafyrei..."
385}
386```
387
388### Post Validation
389
390The handler validates:
3911. **Authentication**: Valid JWT token
3922. **Author exists**: DID in `users` table
3933. **Is aggregator**: DID in `aggregators` table
3944. **Authorization**: Active authorization for (aggregator, community)
3955. **Rate limit**: Less than 10 posts/hour to this community
3966. **Content**: Valid post structure per lexicon
397
398### Rate Limits
399
400**Per-community rate limit**: 10 posts per hour
401
402This is tracked in the `aggregator_posts` table and enforced at the handler level.
403
404**Why?**: Prevents spam while allowing useful bot activity.
405
406**Best practices**:
407- Batch similar content
408- Post only high-quality content
409- Respect community guidelines
410- Monitor your posting rate
411
412## Security Best Practices
413
414### Credential Management
415
416✅ **DO**:
417- Store credentials in environment variables or secret management
418- Use HTTPS for all API calls
419- Rotate access tokens regularly (use refresh tokens)
420- Keep `aggregator-config.env` out of version control
421
422❌ **DON'T**:
423- Hardcode credentials in source code
424- Commit credentials to Git
425- Share access tokens publicly
426- Reuse personal credentials for bots
427
428### Domain Security
429
430✅ **DO**:
431- Use HTTPS for `.well-known` endpoint
432- Keep domain under your control
433- Monitor for unauthorized changes
434- Use DNSSEC if possible
435
436❌ **DON'T**:
437- Use HTTP (will fail verification)
438- Use shared/untrusted hosting
439- Allow others to modify `.well-known` files
440- Use expired SSL certificates
441
442### Content Security
443
444✅ **DO**:
445- Validate all external content before posting
446- Sanitize URLs and text
447- Rate-limit your own posting
448- Implement circuit breakers for failures
449
450❌ **DON'T**:
451- Post unvalidated user input
452- Include malicious links
453- Spam communities
454- Bypass rate limits
455
456## Troubleshooting
457
458### Registration Errors
459
460#### Error: "DomainVerificationFailed"
461
462**Cause**: `.well-known/atproto-did` not accessible or contains wrong DID
463
464**Solutions**:
4651. Verify file is accessible: `curl https://yourdomain.com/.well-known/atproto-did`
4662. Check content matches your DID exactly (no extra whitespace)
4673. Ensure HTTPS is working (not HTTP)
4684. Check web server logs for access errors
4695. Verify firewall rules allow HTTPS traffic
470
471#### Error: "AlreadyRegistered"
472
473**Cause**: This DID is already registered with this Coves instance
474
475**Solutions**:
476- This is safe to ignore if you're re-running setup
477- If you need to update info, just create a new service declaration
478- Contact instance admin if you need to remove registration
479
480#### Error: "DIDResolutionFailed"
481
482**Cause**: Could not resolve DID document from PLC directory
483
484**Solutions**:
4851. Verify DID exists: `curl https://plc.directory/{your-did}`
4862. Wait 30 seconds and retry (PLC propagation delay)
4873. Check PDS is accessible
4884. Verify DID format is correct (must start with `did:plc:` or `did:web:`)
489
490### Posting Errors
491
492#### Error: "NotAuthorized"
493
494**Cause**: No active authorization for this (aggregator, community) pair
495
496**Solutions**:
4971. Check authorizations: `GET /xrpc/social.coves.aggregator.getAuthorizations`
4982. Contact community moderator to request authorization
4993. Verify authorization wasn't disabled
5004. Wait for Jetstream to index authorization (5-10 seconds)
501
502#### Error: "RateLimitExceeded"
503
504**Cause**: Exceeded 10 posts/hour to this community
505
506**Solutions**:
5071. Wait for the rate limit window to reset
5082. Batch posts to stay under limit
5093. Distribute posts across multiple communities
5104. Implement posting queue in your aggregator
511
512### Service Declaration Not Appearing
513
514**Symptoms**: Service declaration created but not in `aggregators` table
515
516**Solutions**:
5171. Wait 5-10 seconds for Jetstream to index
5182. Check Jetstream consumer logs for errors
5193. Verify record was created: Check PDS at `at://your-did/social.coves.aggregator.service/self`
5204. Verify `$type` field is exactly `"social.coves.aggregator.service"`
5215. Check `displayName` is not empty (required field)
522
523## API Reference
524
525### Registration Endpoint
526
527**`POST /xrpc/social.coves.aggregator.register`**
528
529**Input**:
530```typescript
531{
532 did: string // DID of aggregator (did:plc or did:web)
533 domain: string // Domain serving .well-known/atproto-did
534}
535```
536
537**Output**:
538```typescript
539{
540 did: string // Registered DID
541 handle: string // Handle from DID document
542 message: string // Next steps message
543}
544```
545
546**Errors**:
547- `InvalidDID`: DID format invalid
548- `DomainVerificationFailed`: .well-known verification failed
549- `AlreadyRegistered`: DID already registered
550- `DIDResolutionFailed`: Could not resolve DID
551
552### Query Endpoints
553
554**`GET /xrpc/social.coves.aggregator.getServices`**
555
556Get aggregator service details.
557
558**Parameters**:
559- `dids`: Array of DIDs (comma-separated)
560
561**`GET /xrpc/social.coves.aggregator.getAuthorizations`**
562
563List communities that authorized an aggregator.
564
565**Parameters**:
566- `aggregatorDid`: Aggregator DID
567- `enabledOnly`: Filter to enabled only (default: false)
568
569**`GET /xrpc/social.coves.aggregator.listForCommunity`**
570
571List aggregators authorized by a community.
572
573**Parameters**:
574- `communityDid`: Community DID
575- `enabledOnly`: Filter to enabled only (default: false)
576
577## Further Reading
578
579- [Aggregator PRD](PRD_AGGREGATORS.md) - Architecture and design decisions
580- [atProto Guide](../../ATPROTO_GUIDE.md) - atProto fundamentals
581- [Communities PRD](../PRD_COMMUNITIES.md) - Community system overview
582- [Setup Scripts README](../../scripts/aggregator-setup/README.md) - Script documentation
583
584## Support
585
586For issues or questions:
587
5881. Check this guide's troubleshooting section
5892. Review the PRD and architecture docs
5903. Check Coves GitHub issues
5914. Ask in Coves developer community