A community based topic aggregation platform built on atproto
bretton.dev
1# Communities PRD: Federated Forum System
2
3**Status:** In Development
4**Owner:** Platform Team
5**Last Updated:** 2025-10-10
6
7## Overview
8
9Coves communities are federated, instance-scoped forums built on atProto. Each community is identified by a scoped handle (`!gaming@coves.social`) and owns its own atProto repository, enabling true portability and decentralized governance.
10
11## Architecture Evolution
12
13### ✅ V2 Architecture (Current - 2025-10-10)
14
15**Communities own their own repositories:**
16- Each community has its own DID (`did:plc:xxx`)
17- Each community owns its own atProto repository (`at://community_did/...`)
18- Each community has its own PDS account (managed by Coves backend)
19- Communities are truly portable - can migrate between instances by updating DID document
20
21**Repository Structure:**
22```
23Repository: at://did:plc:community789/social.coves.community.profile/self
24Owner: did:plc:community789 (community owns itself)
25Hosted By: did:web:coves.social (instance manages credentials)
26```
27
28**Key Benefits:**
29- ✅ True atProto compliance (matches feed generators, labelers)
30- ✅ Portable URIs (never change when migrating instances)
31- ✅ Self-owned identity model
32- ✅ Standard rkey="self" for singleton profiles
33
34---
35
36## ✅ Completed Features (2025-10-10)
37
38### Core Infrastructure
39- [x] **V2 Architecture:** Communities own their own repositories
40- [x] **PDS Account Provisioning:** Automatic account creation for each community
41- [x] **Credential Management:** Secure storage of community PDS credentials
42- [x] **Encryption at Rest:** PostgreSQL pgcrypto for sensitive credentials
43- [x] **Write-Forward Pattern:** Service → PDS → Firehose → AppView
44- [x] **Jetstream Consumer:** Real-time indexing from firehose
45- [x] **V2 Validation:** Strict rkey="self" enforcement (no V1 compatibility)
46
47### Security & Data Protection
48- [x] **Encrypted Credentials:** Access/refresh tokens encrypted in database
49- [x] **Credential Persistence:** PDS credentials survive server restarts
50- [x] **JSON Exclusion:** Credentials never exposed in API responses (`json:"-"` tags)
51- [x] **Password Hashing:** bcrypt for PDS account passwords
52- [x] **Timeout Handling:** 30s timeout for write operations, 10s for reads
53
54### Database Schema
55- [x] **Communities Table:** Full metadata with V2 credential columns
56- [x] **Subscriptions Table:** Lightweight feed following
57- [x] **Memberships Table:** Active participation tracking
58- [x] **Moderation Table:** Local moderation actions
59- [x] **Encryption Keys Table:** Secure key management for pgcrypto
60- [x] **Indexes:** Optimized for search, visibility filtering, and lookups
61
62### Service Layer
63- [x] **CreateCommunity:** Provisions PDS account, creates record, persists credentials
64- [x] **UpdateCommunity:** Uses community's own credentials (not instance credentials)
65- [x] **GetCommunity:** Fetches from AppView DB with decrypted credentials
66- [x] **ListCommunities:** Pagination, filtering, sorting
67- [x] **SearchCommunities:** Full-text search on name/description
68- [x] **Subscribe/Unsubscribe:** Create subscription records
69- [x] **Handle Validation:** Scoped handle format (`!name@instance`)
70- [x] **DID Generation:** Uses `did:plc` for portability
71
72### Jetstream Consumer
73- [x] **Profile Events:** Create, update, delete community profiles
74- [x] **Subscription Events:** Index user subscriptions to communities
75- [x] **V2 Enforcement:** Reject non-"self" rkeys (no V1 communities)
76- [x] **Self-Ownership Validation:** Verify owner_did == did
77- [x] **Error Handling:** Graceful handling of malformed events
78
79### Testing Coverage
80- [x] **Integration Tests:** Full CRUD operations
81- [x] **Credential Tests:** Persistence, encryption, decryption
82- [x] **V2 Validation Tests:** Rkey enforcement, self-ownership
83- [x] **Consumer Tests:** Firehose event processing
84- [x] **Repository Tests:** Database operations
85- [x] **Unit Tests:** Service layer logic, timeout handling
86
87---
88
89## 🚧 In Progress / Needs Testing
90
91### XRPC API Endpoints
92**Status:** All core endpoints E2E tested! ✅
93
94**✅ E2E Tested (via community_e2e_test.go):**
95- [x] `social.coves.community.create` - Full E2E test with real PDS
96- [x] `social.coves.community.get` - E2E test validates HTTP endpoint
97- [x] `social.coves.community.list` - E2E test with pagination/filtering
98- [x] `social.coves.community.update` - E2E test verifies write-forward + PDS update
99- [x] `social.coves.community.subscribe` - E2E test verifies subscription in user's repo
100- [x] `social.coves.community.unsubscribe` - E2E test verifies PDS deletion
101
102**📍 Post-Alpha:**
103- [ ] `social.coves.community.search` - Handler exists, defer E2E testing to post-alpha
104
105**✅ OAuth Authentication Complete (2025-10-16):**
106- User access tokens now flow through middleware → handlers → service
107- Subscribe/unsubscribe operations use correct user-scoped credentials
108- All E2E tests validate real PDS authentication with user tokens
109
110---
111
112## ⚠️ Alpha Blockers (Must Complete Before Alpha Launch)
113
114### Critical Missing Features
115- [ ] **Subscription Visibility Level (1-5 Scale):** Implement feed slider from DOMAIN_KNOWLEDGE.md
116 - Lexicon: ✅ Ready ([subscription.json:28-34](internal/atproto/lexicon/social/coves/actor/subscription.json))
117 - Service: ❌ Not using `contentVisibility` field
118 - Handler: ❌ Subscribe endpoint doesn't accept/store visibility level
119 - **Impact:** Users can't control how much content they see from each community
120
121- [ ] **Community Blocking:** Users can block communities from their feeds
122 - Lexicon: ❌ Need new record type (extend `social.coves.actor.block` or create new)
123 - Service: ❌ No implementation (`BlockCommunity()` / `UnblockCommunity()`)
124 - Handler: ❌ No endpoints
125 - Repository: ❌ No methods
126 - **Impact:** Users have no way to hide unwanted communities
127
128### Critical Infrastructure (BLOCKING)
129- [ ] **⚠️ Subscription Indexing - NO PRODUCTION CONSUMER**
130 - **Status:** Subscriptions write to PDS but are NEVER indexed in AppView
131 - **Root Cause:** `CommunityEventConsumer` only runs in tests, not in production
132 - **Impact:**
133 - ❌ Users CAN subscribe/unsubscribe (writes to their PDS repo) ✅
134 - ❌ AppView has NO KNOWLEDGE of subscriptions (not consuming from Jetstream)
135 - ❌ Cannot query user's subscriptions (data doesn't exist in AppView)
136 - ❌ Feed generation impossible (don't know who's subscribed to what)
137 - **Required Fixes:**
138 1. Start `CommunityEventConsumer` in production ([cmd/server/main.go](cmd/server/main.go))
139 2. Subscribe to local Jetstream: `ws://localhost:6008/subscribe?wantedCollections=social.coves.community.subscribe`
140 3. Fix unsubscribe handler - should handle `delete` operation on `social.coves.community.subscribe`, NOT a separate collection
141 4. Remove incorrect `social.coves.community.unsubscribe` case ([community_consumer.go:40](internal/atproto/jetstream/community_consumer.go#L40))
142 - **Files:**
143 - Consumer: [internal/atproto/jetstream/community_consumer.go](internal/atproto/jetstream/community_consumer.go) (exists, needs fixes)
144 - Server: [cmd/server/main.go](cmd/server/main.go) (needs to instantiate consumer)
145 - **See:** Issue discovered 2025-10-16 during OAuth user token implementation
146
147### Critical Security (High Priority)
148- [x] **OAuth Authentication:** ✅ COMPLETE - User access tokens flow end-to-end
149 - ✅ Middleware stores user access token in context
150 - ✅ Handlers extract and pass token to service
151 - ✅ Service uses user token for user repo operations (subscribe/unsubscribe)
152 - ✅ All E2E tests pass with real PDS authentication
153 - **Completed:** 2025-10-16
154
155- [ ] **Token Refresh Logic:** Auto-refresh expired PDS access tokens
156 - **Impact:** Communities break after ~2 hours when tokens expire
157 - **See:** [PRD_BACKLOG.md P1 Priority](docs/PRD_BACKLOG.md#L31-L38)
158
159---
160
161## 📍 Beta Features (High Priority - Post Alpha)
162
163### Posts in Communities
164**Status:** Lexicon designed, implementation TODO
165**Priority:** HIGHEST for Beta 1
166
167- [ ] `social.coves.post` already has `community` field ✅
168- [ ] Create post endpoint (decide: membership validation?)
169- [ ] Feed generation for community posts
170- [ ] Post consumer (index community posts from firehose)
171- [ ] Community post count tracking
172- [ ] Decide membership requirements for posting
173
174**Without posts, communities exist but can't be used!**
175
176---
177
178## 📍 Beta Features (Lower Priority)
179
180### Membership System
181**Status:** Lexicon exists, design decisions needed
182**Deferred:** Answer design questions before implementing
183
184- [ ] Decide: Auto-join on first post vs explicit join?
185- [ ] Decide: Reputation tracking in lexicon vs AppView only?
186- [ ] Implement membership record creation (if explicit join)
187- [ ] Member lists endpoint
188- [ ] Reputation tracking (if in lexicon)
189
190### Community Management
191- [ ] **Community Deletion:** Soft delete / permanent delete
192- [ ] **Wiki System:** Lexicon exists, no implementation
193- [ ] **Advanced Rules:** Separate rules records, moderation config
194- [ ] **Moderator Management:** Assign/remove moderators (governance work)
195- [ ] **Categories:** REMOVE from lexicon and code (not needed)
196
197### User Features
198- [ ] **Saved Items:** Save posts/comments for later
199- [ ] **User Flairs:** Per-community user flair (design TBD)
200
201### Instance Moderation
202- [ ] **Delist Community:** Remove from search/directory
203- [ ] **Quarantine Community:** Show warning label
204- [ ] **Remove Community:** Hide from instance AppView
205- [ ] **Moderation Audit Log:** Track all moderation actions
206
207---
208
209## ⏳ TODO Before V1 Production Launch
210
211### Community Discovery & Visibility
212- [ ] **Visibility Enforcement:** Respect public/unlisted/private settings in listings
213- [ ] **Federation Config:** Honor `allowExternalDiscovery` flag
214- [ ] **Search Relevance:** Implement ranking algorithm (members, activity, etc.)
215- [ ] **Directory Endpoint:** Public community directory with filters
216- [ ] **Rate Limiting:** Prevent community creation spam (e.g., 5 per user per hour)
217- [ ] **Handle Collision Detection:** Prevent duplicate community handles
218- [ ] **DID Validation:** Verify DIDs before accepting create requests
219
220### Token Refresh & Resilience
221- [ ] **Retry Mechanism:** Retry failed PDS calls with backoff
222- [ ] **Credential Rotation:** Periodic password rotation for security
223- [ ] **Error Recovery:** Graceful degradation if PDS is unavailable
224
225### Performance & Scaling
226- [ ] **Database Indexes:** Verify all common queries are indexed
227- [ ] **Query Optimization:** Review N+1 query patterns
228- [ ] **Caching Strategy:** Cache frequently accessed communities
229- [ ] **Pagination Limits:** Enforce max results per request
230- [ ] **Connection Pooling:** Optimize PDS HTTP client reuse
231
232### Documentation & Deployment
233- [ ] **API Documentation:** OpenAPI/Swagger specs for all endpoints
234- [ ] **Deployment Guide:** Production setup instructions
235- [ ] **Migration Guide:** How to upgrade from test to production
236- [ ] **Monitoring Guide:** Metrics and alerting setup
237- [ ] **Security Checklist:** Pre-launch security audit
238
239### Infrastructure & DNS
240- [ ] **DNS Wildcard Setup:** Configure `*.communities.coves.social` for community handle resolution
241- [ ] **Well-Known Endpoint:** Implement `.well-known/atproto-did` handler for `*.communities.coves.social` subdomains
242
243---
244
245## Out of Scope (Future Versions)
246
247### V3: Federation & Discovery
248- [ ] Cross-instance community search
249- [ ] Federated moderation signals
250- [ ] Trust networks between instances
251- [ ] Moderation signal subscription
252
253### V4: Community Governance
254- [ ] Community-owned governance (voting on moderators)
255- [ ] Migration voting (community votes to move instances)
256- [ ] Custom domain DIDs (`did:web:gaming.community`)
257- [ ] Governance thresholds and time locks
258
259---
260
261## Recent Critical Fixes (2025-10-10)
262
263### Security & Credential Management
264**Issue:** PDS credentials were created but never persisted
265**Fix:** Service layer now immediately persists credentials via `repo.Create()`
266**Impact:** Communities can now be updated after creation (credentials survive restarts)
267
268**Issue:** Credentials stored in plaintext in PostgreSQL
269**Fix:** Added pgcrypto encryption for access/refresh tokens
270**Impact:** Database compromise no longer exposes active tokens
271
272**Issue:** UpdateCommunity used instance credentials instead of community credentials
273**Fix:** Changed to use `existing.DID` and `existing.PDSAccessToken`
274**Impact:** Updates now correctly authenticate as the community itself
275
276### V2 Architecture Enforcement
277**Issue:** Consumer accepted V1 communities with TID-based rkeys
278**Fix:** Strict validation - only rkey="self" accepted
279**Impact:** No legacy V1 data in production
280
281**Issue:** PDS write operations timed out (10s too short)
282**Fix:** Dynamic timeout - writes get 30s, reads get 10s
283**Impact:** Community creation no longer fails on slow PDS operations
284
285---
286
287## Lexicon Summary
288
289### `social.coves.community.profile`
290**Status:** ✅ Implemented and tested
291
292**Required Fields:**
293- `handle` - atProto handle (DNS-resolvable, e.g., `gaming.communities.coves.social`)
294- `name` - Short community name for !mentions (e.g., `gaming`)
295- `createdBy` - DID of user who created community
296- `hostedBy` - DID of hosting instance
297- `visibility` - `"public"`, `"unlisted"`, or `"private"`
298- `federation.allowExternalDiscovery` - Boolean
299
300**Note:** The `!gaming@coves.social` format is derived client-side from `name` + instance for UI display. The `handle` field contains only the DNS-resolvable atProto handle.
301
302**Optional Fields:**
303- `displayName` - Display name for UI
304- `description` - Community description
305- `descriptionFacets` - Rich text annotations
306- `avatar` - Blob reference for avatar image
307- `banner` - Blob reference for banner image
308- `moderationType` - `"moderator"` or `"sortition"`
309- `contentWarnings` - Array of content warning types
310- `memberCount` - Cached count
311- `subscriberCount` - Cached count
312
313### `social.coves.community.subscription`
314**Status:** ✅ Schema exists, consumer TODO
315
316**Fields:**
317- `community` - DID of community being subscribed to
318- `subscribedAt` - Timestamp
319
320### `social.coves.post` (Community Extension)
321**Status:** ⏳ TODO
322
323**New Field:**
324- `community` - Optional DID of community this post belongs to
325
326---
327
328## Success Metrics
329
330### Pre-Launch Checklist
331- [ ] All XRPC endpoints have E2E tests
332- [ ] OAuth authentication working on all protected endpoints
333- [ ] Rate limiting prevents abuse
334- [ ] Communities can be created, updated, searched, and subscribed to
335- [ ] Jetstream consumer indexes events in < 1 second
336- [ ] Database handles 10,000+ communities without performance issues
337- [ ] Security audit completed
338
339### V1 Launch Goals
340- Communities can be created with scoped handles
341- Posts can be made to communities (when implemented)
342- Community discovery works on local instance
343- All three visibility levels function correctly
344- Basic moderation (delist/remove) works
345
346---
347
348## Technical Decisions Log
349
350### 2025-10-11: Single Handle Field (atProto-Compliant)
351**Decision:** Use single `handle` field containing DNS-resolvable atProto handle; remove `atprotoHandle` field
352
353**Rationale:**
354- Matches Bluesky pattern: `app.bsky.actor.profile` has one `handle` field
355- Reduces confusion about which handle is "real"
356- Simplifies lexicon (one field vs two)
357- `!gaming@coves.social` display format is client-side UX concern, not protocol concern
358- Follows separation of concerns: protocol layer uses DNS handles, UI layer formats for display
359
360**Implementation:**
361- Lexicon: `handle` = `gaming.communities.coves.social` (DNS-resolvable)
362- Client derives display: `!${name}@${instance}` from `name` + parsed instance
363- Rich text facets can encode community mentions with `!` prefix for UX
364
365**Trade-offs Accepted:**
366- Clients must parse/format for display (but already do this for `@user` mentions)
367- No explicit "display handle" in record (but `displayName` serves this purpose)
368
369---
370
371### 2025-10-10: V2 Architecture Completed
372- Migrated from instance-owned to community-owned repositories
373- Each community now has own PDS account
374- Credentials encrypted at rest using pgcrypto
375- Strict V2 enforcement (no V1 compatibility)
376
377### 2025-10-08: DID Architecture & atProto Compliance
378- Migrated from `did:coves` to `did:plc` (portable DIDs)
379- Added required `did` field to lexicon
380- Fixed critical `record_uri` bug
381- Matches Bluesky feed generator pattern
382
383---
384
385## References
386
387- atProto Lexicon Spec: https://atproto.com/specs/lexicon
388- DID Web Spec: https://w3c-ccg.github.io/did-method-web/
389- Bluesky Handle System: https://atproto.com/specs/handle
390- PLC Directory: https://plc.directory