A community based topic aggregation platform built on atproto
bretton.dev
1# Communities PRD: Federated Forum System
2
3**Status:** In Development
4**Owner:** Platform Team
5**Last Updated:** 2025-10-17
6
7## Overview
8
9Coves communities are federated, instance-scoped forums built on atProto. Each community is identified by a scoped handle (`!gaming@coves.social`) and owns its own atProto repository, enabling true portability and decentralized governance.
10
11## Architecture Evolution
12
13### ✅ V2 Architecture (Current - 2025-10-10)
14
15**Communities own their own repositories:**
16- Each community has its own DID (`did:plc:xxx`)
17- Each community owns its own atProto repository (`at://community_did/...`)
18- Each community has its own PDS account (managed by Coves backend)
19- Communities are truly portable - can migrate between instances by updating DID document
20
21**Repository Structure:**
22```
23Repository: at://did:plc:community789/social.coves.community.profile/self
24Owner: did:plc:community789 (community owns itself)
25Hosted By: did:web:coves.social (instance manages credentials)
26```
27
28**Key Benefits:**
29- ✅ True atProto compliance (matches feed generators, labelers)
30- ✅ Portable URIs (never change when migrating instances)
31- ✅ Self-owned identity model
32- ✅ Standard rkey="self" for singleton profiles
33
34---
35
36## ✅ Completed Features (Updated 2025-10-17)
37
38### Core Infrastructure
39- [x] **V2 Architecture:** Communities own their own repositories
40- [x] **PDS Account Provisioning:** Automatic account creation for each community
41- [x] **Credential Management:** Secure storage of community PDS credentials
42- [x] **Token Refresh:** Automatic refresh of expired access tokens (completed 2025-10-17)
43- [x] **Encryption at Rest:** PostgreSQL pgcrypto for sensitive credentials
44- [x] **Write-Forward Pattern:** Service → PDS → Firehose → AppView
45- [x] **Jetstream Consumer:** Real-time indexing from firehose
46- [x] **V2 Validation:** Strict rkey="self" enforcement (no V1 compatibility)
47
48### Security & Data Protection
49- [x] **Encrypted Credentials:** Access/refresh tokens encrypted in database
50- [x] **Credential Persistence:** PDS credentials survive server restarts
51- [x] **Automatic Token Refresh:** Tokens refresh 5 minutes before expiration (completed 2025-10-17)
52- [x] **Password Fallback:** Re-authentication when refresh tokens expire
53- [x] **Concurrency Safety:** Per-community mutex prevents refresh race conditions
54- [x] **JSON Exclusion:** Credentials never exposed in API responses (`json:"-"` tags)
55- [x] **Password Encryption:** Encrypted (not hashed) for session creation fallback
56- [x] **Timeout Handling:** 30s timeout for write operations, 10s for reads
57
58### Database Schema
59- [x] **Communities Table:** Full metadata with V2 credential columns
60- [x] **Subscriptions Table:** Lightweight feed following
61- [x] **Memberships Table:** Active participation tracking
62- [x] **Moderation Table:** Local moderation actions
63- [x] **Encryption Keys Table:** Secure key management for pgcrypto
64- [x] **Indexes:** Optimized for search, visibility filtering, and lookups
65
66### Service Layer
67- [x] **CreateCommunity:** Provisions PDS account, creates record, persists credentials
68- [x] **UpdateCommunity:** Uses community's own credentials (not instance credentials)
69- [x] **GetCommunity:** Fetches from AppView DB with decrypted credentials
70- [x] **ListCommunities:** Pagination, filtering, sorting
71- [x] **SearchCommunities:** Full-text search on name/description
72- [x] **Subscribe/Unsubscribe:** Create subscription records
73- [x] **Handle Validation:** Scoped handle format (`!name@instance`)
74- [x] **DID Generation:** Uses `did:plc` for portability
75
76### Jetstream Consumer
77- [x] **Profile Events:** Create, update, delete community profiles
78- [x] **Subscription Events:** Index user subscriptions to communities
79- [x] **V2 Enforcement:** Reject non-"self" rkeys (no V1 communities)
80- [x] **Self-Ownership Validation:** Verify owner_did == did
81- [x] **Error Handling:** Graceful handling of malformed events
82
83### Testing Coverage
84- [x] **Integration Tests:** Full CRUD operations
85- [x] **Credential Tests:** Persistence, encryption, decryption
86- [x] **Token Refresh Tests:** JWT parsing, credential updates, concurrency (completed 2025-10-17)
87- [x] **V2 Validation Tests:** Rkey enforcement, self-ownership
88- [x] **Consumer Tests:** Firehose event processing
89- [x] **Repository Tests:** Database operations
90- [x] **Unit Tests:** Service layer logic, timeout handling
91
92---
93
94## 🚧 In Progress / Needs Testing
95
96### XRPC API Endpoints
97**Status:** All core endpoints E2E tested! ✅
98
99**✅ E2E Tested (via community_e2e_test.go):**
100- [x] `social.coves.community.create` - Full E2E test with real PDS
101- [x] `social.coves.community.get` - E2E test validates HTTP endpoint
102- [x] `social.coves.community.list` - E2E test with pagination/filtering
103- [x] `social.coves.community.update` - E2E test verifies write-forward + PDS update
104- [x] `social.coves.community.subscribe` - E2E test verifies subscription in user's repo
105- [x] `social.coves.community.unsubscribe` - E2E test verifies PDS deletion
106
107**📍 Post-Alpha:**
108- [ ] `social.coves.community.search` - Handler exists, defer E2E testing to post-alpha
109
110**✅ OAuth Authentication Complete (2025-10-16):**
111- User access tokens now flow through middleware → handlers → service
112- Subscribe/unsubscribe operations use correct user-scoped credentials
113- All E2E tests validate real PDS authentication with user tokens
114
115---
116
117## ⚠️ Alpha Blockers (Must Complete Before Alpha Launch)
118
119### Critical Missing Features
120- [x] **Community Blocking:** ✅ COMPLETE - Users can block communities from their feeds
121 - ✅ Lexicon: `social.coves.community.block` record type implemented
122 - ✅ Service: `BlockCommunity()` / `UnblockCommunity()` / `GetBlockedCommunities()` / `IsBlocked()`
123 - ✅ Handlers: Block/unblock endpoints implemented
124 - ✅ Repository: Full blocking methods with indexing
125 - ✅ Jetstream Consumer: Real-time indexing of block events
126 - ✅ Integration tests: Comprehensive coverage
127 - **Completed:** 2025-10-16
128 - **Impact:** Users can now hide unwanted communities from their feeds
129
130### ✅ Critical Infrastructure - RESOLVED (2025-10-16)
131- [x] **✅ Subscription Indexing & ContentVisibility - COMPLETE**
132 - **Status:** Subscriptions now fully indexed in AppView with feed slider support
133 - **Completed:** 2025-10-16
134 - **What Was Fixed:**
135 1. ✅ Fixed critical collection name bug (`social.coves.actor.subscription` → `social.coves.community.subscription`)
136 2. ✅ Implemented ContentVisibility (1-5 slider) across all layers (handler, service, consumer, repository)
137 3. ✅ Production Jetstream consumer now running ([cmd/server/main.go:220-243](cmd/server/main.go#L220-L243))
138 4. ✅ Migration 008 adds `content_visibility` column with defaults and constraints
139 5. ✅ Atomic subscriber count updates (SubscribeWithCount/UnsubscribeWithCount)
140 6. ✅ DELETE operations properly handled (unsubscribe indexing)
141 7. ✅ Idempotent operations (safe for Jetstream event replays)
142 8. ✅ atProto naming compliance: singular namespace + `subject` field
143 - **Impact:**
144 - ✅ Users CAN subscribe/unsubscribe (writes to their PDS repo)
145 - ✅ AppView INDEXES subscriptions from Jetstream in real-time
146 - ✅ Can query user's subscriptions (data persisted with contentVisibility)
147 - ✅ Feed generation ENABLED (know who's subscribed with visibility preferences)
148 - ✅ Subscriber counts accurate (atomic updates)
149 - **Testing:**
150 - ✅ 13 comprehensive integration tests (subscription_indexing_test.go) - ALL PASSING
151 - ✅ Enhanced E2E tests verify complete flow (HTTP → PDS → Jetstream → AppView)
152 - ✅ ContentVisibility clamping tested (0→1, 10→5, defaults to 3)
153 - ✅ Idempotency verified (duplicate events handled gracefully)
154 - **Files:**
155 - Implementation Doc: [docs/IMPLEMENTATION_SUBSCRIPTION_INDEXING.md](docs/IMPLEMENTATION_SUBSCRIPTION_INDEXING.md)
156 - Lexicon: [internal/atproto/lexicon/social/coves/community/subscription.json](internal/atproto/lexicon/social/coves/community/subscription.json)
157 - Consumer: [internal/atproto/jetstream/community_consumer.go](internal/atproto/jetstream/community_consumer.go)
158 - Connector: [internal/atproto/jetstream/community_jetstream_connector.go](internal/atproto/jetstream/community_jetstream_connector.go)
159 - Migration: [internal/db/migrations/008_add_content_visibility_to_subscriptions.sql](internal/db/migrations/008_add_content_visibility_to_subscriptions.sql)
160 - Tests: [tests/integration/subscription_indexing_test.go](tests/integration/subscription_indexing_test.go)
161
162### Critical Security (High Priority)
163- [x] **OAuth Authentication:** ✅ COMPLETE - User access tokens flow end-to-end
164 - ✅ Middleware stores user access token in context
165 - ✅ Handlers extract and pass token to service
166 - ✅ Service uses user token for user repo operations (subscribe/unsubscribe)
167 - ✅ All E2E tests pass with real PDS authentication
168 - **Completed:** 2025-10-16
169
170- [x] **Token Refresh Logic:** ✅ COMPLETE - Auto-refresh expired PDS access tokens
171 - ✅ Automatic token refresh before PDS operations (5-minute buffer)
172 - ✅ Password fallback when refresh tokens expire (~2 months)
173 - ✅ Concurrency-safe with per-community mutex locking
174 - ✅ Atomic credential updates in database
175 - ✅ Integration tests and structured logging
176 - **Completed:** 2025-10-17
177 - **See:** [IMPLEMENTATION_TOKEN_REFRESH.md](docs/IMPLEMENTATION_TOKEN_REFRESH.md)
178
179---
180
181## 📍 Beta Features (High Priority - Post Alpha)
182
183### Content Rules System
184**Status:** Lexicon complete (2025-10-18), implementation TODO
185**Priority:** CRITICAL for Alpha - Enables community content policies
186
187Communities can define content posting restrictions via the `contentRules` object in their profile:
188
189**Key Features:**
190- ✅ Lexicon: `contentRules` object defined in `social.coves.community.profile`
191- [ ] Validation: Post creation validates against community rules
192- [ ] AppView indexing: Index post characteristics (embed_type, text_length)
193- [ ] Error handling: Clear `ContentRuleViolation` errors
194
195**Example Use Cases:**
196- **Text-only communities:** "AskCoves" requires text, blocks all embeds
197- **Image communities:** "CovesPics" requires at least 1 image
198- **No restrictions:** General communities allow all content types
199
200**See:** [PRD_GOVERNANCE.md - Content Rules System](PRD_GOVERNANCE.md#content-rules-system) for full details
201
202---
203
204### Blob Upload Proxy System
205**Status:** Design documented, implementation TODO
206**Priority:** CRITICAL for Beta - Required for image/video posts in communities
207
208**Problem:** Users on external PDSs cannot directly upload blobs to community-owned PDS repositories because they lack authentication credentials for the community's PDS.
209
210**Solution:** Coves AppView acts as an authenticated proxy for blob uploads:
211
212**Flow:**
2131. User uploads blob to Coves AppView via `social.coves.blob.uploadForCommunity`
2142. AppView validates user can post to community (not banned, community accessible)
2153. AppView uses community's PDS credentials to upload blob via `com.atproto.repo.uploadBlob`
2164. AppView returns CID to user
2175. User creates post record referencing the CID
2186. Post and blob both live in community's PDS
219
220**Implementation Checklist:**
221- [ ] Handler: `social.coves.blob.uploadForCommunity` endpoint
222- [ ] Validation: Check user authorization to post in community
223- [ ] Credential Management: Reuse community token refresh logic
224- [ ] Upload Proxy: Forward blob to community's PDS with community credentials
225- [ ] Security: Size limits, content-type validation, rate limiting
226- [ ] Testing: E2E test with federated user uploading to community
227
228**Why This Approach:**
229- ✅ Works with federated users (any PDS)
230- ✅ Reuses existing community credential infrastructure
231- ✅ Matches V2 architecture (AppView orchestrates, communities own data)
232- ✅ Blobs stored on correct PDS (community's repository)
233- ❌ AppView becomes upload intermediary (bandwidth cost)
234
235**Alternative Considered:** Direct user uploads to community PDS
236- Rejected: Would require creating temporary user accounts on every community PDS (complex, insecure)
237
238**See:** Design discussion in context of ATProto blob architecture
239
240---
241
242### Posts in Communities
243**Status:** Lexicon designed, implementation TODO
244**Priority:** HIGHEST for Beta 1
245
246- [x] `social.coves.post` already has `community` field ✅
247- [x] Removed `postType` enum in favor of content rules ✅ (2025-10-18)
248- [ ] Create post endpoint with content rules validation
249- [ ] Feed generation for community posts
250- [ ] Post consumer (index community posts from firehose)
251- [ ] Community post count tracking
252- [ ] Decide membership requirements for posting
253
254**Without posts, communities exist but can't be used!**
255
256**Depends on:** Blob Upload Proxy System (for image/video posts)
257
258---
259
260## 📍 Beta Features (Lower Priority)
261
262### Membership System
263**Status:** Lexicon exists, design decisions needed
264**Deferred:** Answer design questions before implementing
265
266- [ ] Decide: Auto-join on first post vs explicit join?
267- [ ] Decide: Reputation tracking in lexicon vs AppView only?
268- [ ] Implement membership record creation (if explicit join)
269- [ ] Member lists endpoint
270- [ ] Reputation tracking (if in lexicon)
271
272### Community Management
273- [ ] **Community Deletion:** Soft delete / permanent delete
274- [ ] **Wiki System:** Lexicon exists, no implementation
275- [ ] **Advanced Rules:** Separate rules records, moderation config
276- [ ] **Moderator Management:** Assign/remove moderators (governance work)
277- [ ] **Categories:** REMOVE from lexicon and code (not needed)
278
279### User Features
280- [ ] **Saved Items:** Save posts/comments for later
281- [ ] **User Flairs:** Per-community user flair (design TBD)
282
283### Instance Moderation
284- [ ] **Delist Community:** Remove from search/directory
285- [ ] **Quarantine Community:** Show warning label
286- [ ] **Remove Community:** Hide from instance AppView
287- [ ] **Moderation Audit Log:** Track all moderation actions
288
289---
290
291## ⏳ TODO Before V1 Production Launch
292
293### Community Discovery & Visibility
294- [ ] **Visibility Enforcement:** Respect public/unlisted/private settings in listings
295- [ ] **Federation Config:** Honor `allowExternalDiscovery` flag
296- [ ] **Search Relevance:** Implement ranking algorithm (members, activity, etc.)
297- [ ] **Directory Endpoint:** Public community directory with filters
298- [ ] **Rate Limiting:** Prevent community creation spam (e.g., 5 per user per hour)
299- [ ] **Handle Collision Detection:** Prevent duplicate community handles
300- [ ] **DID Validation:** Verify DIDs before accepting create requests
301
302### Token Refresh & Resilience
303- [ ] **Retry Mechanism:** Retry failed PDS calls with backoff
304- [ ] **Credential Rotation:** Periodic password rotation for security
305- [ ] **Error Recovery:** Graceful degradation if PDS is unavailable
306
307### Performance & Scaling
308- [ ] **Database Indexes:** Verify all common queries are indexed
309- [ ] **Query Optimization:** Review N+1 query patterns
310- [ ] **Caching Strategy:** Cache frequently accessed communities
311- [ ] **Pagination Limits:** Enforce max results per request
312- [ ] **Connection Pooling:** Optimize PDS HTTP client reuse
313
314### Documentation & Deployment
315- [ ] **API Documentation:** OpenAPI/Swagger specs for all endpoints
316- [ ] **Deployment Guide:** Production setup instructions
317- [ ] **Migration Guide:** How to upgrade from test to production
318- [ ] **Monitoring Guide:** Metrics and alerting setup
319- [ ] **Security Checklist:** Pre-launch security audit
320
321### Infrastructure & DNS
322- [ ] **DNS Wildcard Setup:** Configure `*.community.coves.social` for community handle resolution
323- [ ] **Well-Known Endpoint:** Implement `.well-known/atproto-did` handler for `*.community.coves.social` subdomains
324
325---
326
327## Out of Scope (Future Versions)
328
329### V3: Federation & Discovery
330- [ ] Cross-instance community search
331- [ ] Federated moderation signals
332- [ ] Trust networks between instances
333- [ ] Moderation signal subscription
334
335### V4: Community Governance
336- [ ] Community-owned governance (voting on moderators)
337- [ ] Migration voting (community votes to move instances)
338- [ ] Custom domain DIDs (`did:web:gaming.community`)
339- [ ] Governance thresholds and time locks
340
341---
342
343## Recent Critical Fixes (2025-10-10)
344
345### Security & Credential Management
346**Issue:** PDS credentials were created but never persisted
347**Fix:** Service layer now immediately persists credentials via `repo.Create()`
348**Impact:** Communities can now be updated after creation (credentials survive restarts)
349
350**Issue:** Credentials stored in plaintext in PostgreSQL
351**Fix:** Added pgcrypto encryption for access/refresh tokens
352**Impact:** Database compromise no longer exposes active tokens
353
354**Issue:** UpdateCommunity used instance credentials instead of community credentials
355**Fix:** Changed to use `existing.DID` and `existing.PDSAccessToken`
356**Impact:** Updates now correctly authenticate as the community itself
357
358### V2 Architecture Enforcement
359**Issue:** Consumer accepted V1 communities with TID-based rkeys
360**Fix:** Strict validation - only rkey="self" accepted
361**Impact:** No legacy V1 data in production
362
363**Issue:** PDS write operations timed out (10s too short)
364**Fix:** Dynamic timeout - writes get 30s, reads get 10s
365**Impact:** Community creation no longer fails on slow PDS operations
366
367---
368
369## Lexicon Summary
370
371### `social.coves.community.profile`
372**Status:** ✅ Implemented and tested
373
374**Required Fields:**
375- `handle` - atProto handle (DNS-resolvable, e.g., `gaming.community.coves.social`)
376- `name` - Short community name for !mentions (e.g., `gaming`)
377- `createdBy` - DID of user who created community
378- `hostedBy` - DID of hosting instance
379- `visibility` - `"public"`, `"unlisted"`, or `"private"`
380- `federation.allowExternalDiscovery` - Boolean
381
382**Note:** The `!gaming@coves.social` format is derived client-side from `name` + instance for UI display. The `handle` field contains only the DNS-resolvable atProto handle.
383
384**Optional Fields:**
385- `displayName` - Display name for UI
386- `description` - Community description
387- `descriptionFacets` - Rich text annotations
388- `avatar` - Blob reference for avatar image
389- `banner` - Blob reference for banner image
390- `moderationType` - `"moderator"` or `"sortition"`
391- `contentWarnings` - Array of content warning types
392- `contentRules` - Content posting restrictions (see [PRD_GOVERNANCE.md](PRD_GOVERNANCE.md#content-rules-system))
393 - `allowedEmbedTypes` - Array of allowed embed types (images, video, external, record)
394 - `requireText` - Whether text content is required
395 - `minTextLength` / `maxTextLength` - Text length constraints
396 - `requireTitle` - Whether title is required
397 - `minImages` / `maxImages` - Image count constraints
398 - `allowFederated` - Whether federated posts allowed
399- `memberCount` - Cached count
400- `subscriberCount` - Cached count
401
402### `social.coves.community.subscription`
403**Status:** ✅ Schema exists, consumer TODO
404
405**Fields:**
406- `community` - DID of community being subscribed to
407- `subscribedAt` - Timestamp
408
409### `social.coves.post` (Community Extension)
410**Status:** ⏳ TODO
411
412**New Field:**
413- `community` - Optional DID of community this post belongs to
414
415---
416
417## Success Metrics
418
419### Pre-Launch Checklist
420- [ ] All XRPC endpoints have E2E tests
421- [ ] OAuth authentication working on all protected endpoints
422- [ ] Rate limiting prevents abuse
423- [ ] Communities can be created, updated, searched, and subscribed to
424- [ ] Jetstream consumer indexes events in < 1 second
425- [ ] Database handles 10,000+ communities without performance issues
426- [ ] Security audit completed
427
428### V1 Launch Goals
429- Communities can be created with scoped handles
430- Posts can be made to communities (when implemented)
431- Community discovery works on local instance
432- All three visibility levels function correctly
433- Basic moderation (delist/remove) works
434
435---
436
437## Technical Decisions Log
438
439### 2025-10-18: Content Rules Over Post Type Enum
440**Decision:** Remove `postType` enum from post creation; use flexible `contentRules` in community profile instead
441
442**Rationale:**
443- `postType` enum forced users to explicitly select type (bad UX - app should infer from structure)
444- Enum was rigid - couldn't support nuanced rules like "text required, images optional"
445- Content rules are more extensible (add new constraints without changing post lexicon)
446- Follows atProto philosophy: describe data structure, not UI intent
447- Enables both community restrictions ("text only") AND user filtering ("show videos only")
448
449**Implementation:**
450- Community profile contains optional `contentRules` object
451- Post validation checks structure against rules at creation time
452- AppView indexes post characteristics (embed_type, text_length) for filtering
453- Errors use `ContentRuleViolation` instead of `InvalidPostType`
454
455**Examples:**
456- "AskCoves": `allowedEmbedTypes: []` + `requireText: true` + `minTextLength: 50`
457- "CovesPics": `allowedEmbedTypes: ["images"]` + `minImages: 1`
458- General communities: `contentRules: null` (no restrictions)
459
460**Trade-offs Accepted:**
461- Validation logic more complex than simple enum check (but more powerful)
462- Communities can't programmatically restrict to exact "article" vs "text" types (but structure-based rules are better)
463
464**See:** [PRD_GOVERNANCE.md - Content Rules System](PRD_GOVERNANCE.md#content-rules-system)
465
466---
467
468### 2025-10-11: Single Handle Field (atProto-Compliant)
469**Decision:** Use single `handle` field containing DNS-resolvable atProto handle; remove `atprotoHandle` field
470
471**Rationale:**
472- Matches Bluesky pattern: `app.bsky.actor.profile` has one `handle` field
473- Reduces confusion about which handle is "real"
474- Simplifies lexicon (one field vs two)
475- `!gaming@coves.social` display format is client-side UX concern, not protocol concern
476- Follows separation of concerns: protocol layer uses DNS handles, UI layer formats for display
477
478**Implementation:**
479- Lexicon: `handle` = `gaming.community.coves.social` (DNS-resolvable)
480- Client derives display: `!${name}@${instance}` from `name` + parsed instance
481- Rich text facets can encode community mentions with `!` prefix for UX
482
483**Trade-offs Accepted:**
484- Clients must parse/format for display (but already do this for `@user` mentions)
485- No explicit "display handle" in record (but `displayName` serves this purpose)
486
487---
488
489### 2025-10-10: V2 Architecture Completed
490- Migrated from instance-owned to community-owned repositories
491- Each community now has own PDS account
492- Credentials encrypted at rest using pgcrypto
493- Strict V2 enforcement (no V1 compatibility)
494
495### 2025-10-08: DID Architecture & atProto Compliance
496- Migrated from `did:coves` to `did:plc` (portable DIDs)
497- Added required `did` field to lexicon
498- Fixed critical `record_uri` bug
499- Matches Bluesky feed generator pattern
500
501---
502
503## References
504
505- atProto Lexicon Spec: https://atproto.com/specs/lexicon
506- DID Web Spec: https://w3c-ccg.github.io/did-method-web/
507- Bluesky Handle System: https://atproto.com/specs/handle
508- PLC Directory: https://plc.directory