···
# Aggregators PRD: Automated Content Posting System
3
-
**Status:** Planning / Design Phase
3
+
**Status:** In Development - Phase 1 (Core Infrastructure)
5
-
**Last Updated:** 2025-10-19
5
+
**Last Updated:** 2025-10-20
9
-
Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This system enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control over which aggregators can post and what content they can create.
11
+
Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control.
**Key Differentiator:** Unlike other platforms where users manually aggregate content, Coves communities can enable automated aggregators to handle routine posting tasks, creating a more dynamic and up-to-date community experience.
## Architecture Principles
### ✅ atProto-Compliant Design
17
-
Aggregators follow established atProto patterns for autonomous services:
19
-
**Pattern:** Feed Generators + Labelers Model
20
-
- Each aggregator has its own DID (like feed generators)
21
-
- Declaration record published in aggregator's repo (like `app.bsky.feed.generator`)
22
-
- DID document advertises service endpoint
23
-
- Service makes authenticated XRPC calls
24
-
- Communities explicitly authorize aggregators (like subscribing to labelers)
26
-
**Key Design Decisions:**
21
+
Aggregators follow established atProto patterns for autonomous services (Feed Generators + Labelers model):
1. **Aggregators are Actors, Not a Separate System**
29
-
- Aggregators authenticate as themselves (their DID)
24
+
- Each aggregator has its own DID
25
+
- Authenticate as themselves via JWT
- Use existing `social.coves.post.create` endpoint
- Post record's `author` field = aggregator DID (server-populated)
- No separate posting API needed
2. **Community Authorization Model**
35
-
- Communities create `social.coves.aggregator.authorization` records
36
-
- These records grant specific aggregators permission to post
37
-
- Authorizations include configuration (which RSS feeds, which users to mirror, etc.)
38
-
- Can be enabled/disabled at any time
31
+
- Communities create `social.coves.aggregator.authorization` records in their repo
32
+
- Records grant specific aggregators permission to post
33
+
- Include aggregator-specific configuration
34
+
- Can be enabled/disabled without deletion
41
-
- Coves can host official aggregators (RSS, sports, media)
42
-
- Third parties can build and host their own aggregators
43
-
- SDK provided for easy aggregator development
44
-
- All aggregators use same authorization system
48
-
## Architecture Overview
51
-
┌────────────────────────────────────────────────────────────┐
52
-
│ Aggregator Service (External) │
53
-
│ DID: did:web:rss-bot.coves.social │
55
-
│ - Watches external data sources (RSS, APIs, etc.) │
56
-
│ - Processes content (LLM deduplication, formatting) │
57
-
│ - Queries which communities have authorized it │
58
-
│ - Creates posts via social.coves.post.create │
59
-
│ - Responds to config queries via XRPC │
60
-
└────────────────────────────────────────────────────────────┘
62
-
│ 1. Authenticate as aggregator DID (JWT)
63
-
│ 2. Call social.coves.post.create
65
-
│ community: "did:plc:gaming123",
68
-
│ federatedFrom: { platform: "rss", ... }
71
-
┌────────────────────────────────────────────────────────────┐
72
-
│ Coves AppView (social.coves.post.create Handler) │
74
-
│ 1. Extract DID from JWT (aggregator's DID) │
75
-
│ 2. Check if DID is registered aggregator │
76
-
│ 3. Validate authorization record exists & enabled │
77
-
│ 4. Apply aggregator-specific rate limits │
78
-
│ 5. Validate content against community rules │
79
-
│ 6. Create post with author = aggregator DID │
80
-
└────────────────────────────────────────────────────────────┘
82
-
│ Post record created:
84
-
│ $type: "social.coves.post.record",
85
-
│ author: "did:web:rss-bot.coves.social",
86
-
│ community: "did:plc:gaming123",
87
-
│ title: "Tech News Roundup",
91
-
│ uri: "https://techcrunch.com/..."
95
-
┌────────────────────────────────────────────────────────────┐
96
-
│ Jetstream → AppView Indexing │
97
-
│ - Post indexed with aggregator attribution │
98
-
│ - UI shows: "🤖 Posted by RSS Aggregator" │
99
-
│ - Community feed includes automated posts │
100
-
└────────────────────────────────────────────────────────────┘
37
+
- Coves can host official aggregators
38
+
- Third parties can build and host their own
39
+
- All use same authorization system
107
-
### 1. RSS News Aggregator
108
-
**Problem:** Multiple users posting the same breaking news from different sources
109
-
**Solution:** RSS aggregator with LLM deduplication
110
-
- Watches configured RSS feeds
111
-
- Uses LLM to identify duplicate stories from different outlets
112
-
- Creates single "megathread" with all sources linked
113
-
- Posts unbiased summary of event
114
-
- Automatically tags with relevant topics
116
-
**Community Config:**
119
-
"aggregatorDid": "did:web:rss-bot.coves.social",
123
-
"https://techcrunch.com/feed",
124
-
"https://arstechnica.com/feed"
126
-
"topics": ["technology", "ai"],
127
-
"dedupeWindow": "6h",
133
-
### 2. Bluesky Post Mirror
134
-
**Problem:** Want to surface specific Bluesky discussions in community
135
-
**Solution:** Bluesky mirror aggregator
136
-
- Monitors specific users or hashtags on Bluesky
137
-
- Creates posts in community when criteria met
138
-
- Preserves `originalAuthor` metadata
139
-
- Links back to original Bluesky thread
141
-
**Community Config:**
144
-
"aggregatorDid": "did:web:bsky-mirror.coves.social",
148
-
"alice.bsky.social",
151
-
"hashtags": ["covesalpha"],
157
-
### 3. Sports Results Aggregator
158
-
**Problem:** Need post-game threads created immediately after games end
159
-
**Solution:** Sports aggregator watching game APIs
160
-
- Monitors sports APIs for game completions
161
-
- Creates post-game thread with final score, stats
162
-
- Tags with team names and league
163
-
- Posts within minutes of game ending
45
+
### 1. Service Declaration Record
46
+
**Lexicon:** `social.coves.aggregator.service`
47
+
**Location:** Aggregator's repository
48
+
**Key:** `literal:self`
165
-
**Community Config:**
168
-
"aggregatorDid": "did:web:sports-bot.coves.social",
172
-
"teams": ["Lakers", "Warriors"],
173
-
"includeStats": true,
50
+
Declares aggregator existence and provides metadata for discovery.
179
-
### 4. TV/Movie Discussion Aggregator
180
-
**Problem:** Want episode discussion threads created when shows air
181
-
**Solution:** Media aggregator tracking release schedules
182
-
- Uses TMDB/IMDB APIs for release dates
183
-
- Creates discussion threads when episodes/movies release
184
-
- Includes metadata (cast, synopsis, ratings)
185
-
- Automatically pins for premiere episodes
52
+
**Required Fields:**
53
+
- `did` - Aggregator's DID (must match repo)
54
+
- `displayName` - Human-readable name
55
+
- `createdAt` - Creation timestamp
187
-
**Community Config:**
190
-
"aggregatorDid": "did:web:media-bot.coves.social",
194
-
{"tmdbId": "1234", "name": "Breaking Bad"}
196
-
"createOn": "airDate",
197
-
"timezone": "America/New_York",
198
-
"spoilerProtection": true
57
+
**Optional Fields:**
58
+
- `description` - What this aggregator does
59
+
- `avatar` - Avatar image blob
60
+
- `configSchema` - JSON Schema for community config validation
61
+
- `sourceUrl` - Link to source code (transparency)
62
+
- `maintainer` - DID of maintainer
66
+
### 2. Authorization Record
67
+
**Lexicon:** `social.coves.aggregator.authorization`
68
+
**Location:** Community's repository
207
-
### 1. Aggregator Service Declaration
71
+
Grants an aggregator permission to post with specific configuration.
209
-
**Collection:** `social.coves.aggregator.service`
210
-
**Key:** `literal:self` (one per aggregator account)
211
-
**Location:** Aggregator's own repository
213
-
This record declares the existence of an aggregator service and provides metadata for discovery.
218
-
"id": "social.coves.aggregator.service",
222
-
"description": "Declaration of an aggregator service that can post to communities",
223
-
"key": "literal:self",
226
-
"required": ["did", "displayName", "createdAt", "aggregatorType"],
231
-
"description": "DID of the aggregator service (must match repo DID)"
235
-
"maxGraphemes": 64,
237
-
"description": "Human-readable name (e.g., 'RSS News Aggregator')"
241
-
"maxGraphemes": 300,
243
-
"description": "Description of what this aggregator does"
247
-
"accept": ["image/png", "image/jpeg"],
248
-
"maxSize": 1000000,
249
-
"description": "Avatar image for bot identity"
251
-
"aggregatorType": {
254
-
"social.coves.aggregator.types#rss",
255
-
"social.coves.aggregator.types#blueskyMirror",
256
-
"social.coves.aggregator.types#sports",
257
-
"social.coves.aggregator.types#media",
258
-
"social.coves.aggregator.types#custom"
260
-
"description": "Type of aggregator for categorization"
264
-
"description": "JSON Schema describing config options for this aggregator. Communities use this to know what configuration fields are available."
269
-
"description": "URL to aggregator's source code (for transparency)"
274
-
"description": "DID of person/organization maintaining this aggregator"
278
-
"format": "datetime"
73
+
**Required Fields:**
74
+
- `aggregatorDid` - DID of authorized aggregator
75
+
- `communityDid` - DID of community (must match repo)
76
+
- `enabled` - Active status (toggleable)
77
+
- `createdAt` - When authorized
287
-
**Example Record:**
290
-
"$type": "social.coves.aggregator.service",
291
-
"did": "did:web:rss-bot.coves.social",
292
-
"displayName": "RSS News Aggregator",
293
-
"description": "Automatically posts breaking news from configured RSS feeds with LLM-powered deduplication",
294
-
"aggregatorType": "social.coves.aggregator.types#rss",
300
-
"items": { "type": "string", "format": "uri" }
304
-
"items": { "type": "string" }
306
-
"dedupeWindow": { "type": "string" },
307
-
"minSources": { "type": "integer", "minimum": 1 }
310
-
"sourceUrl": "https://github.com/coves-social/rss-aggregator",
311
-
"maintainer": "did:plc:coves-platform",
312
-
"createdAt": "2025-10-19T12:00:00Z"
79
+
**Optional Fields:**
80
+
- `config` - Aggregator-specific config (validated against schema)
81
+
- `createdBy` - Moderator who authorized
82
+
- `disabledAt` / `disabledBy` - Audit trail
318
-
### 2. Community Authorization Record
320
-
**Collection:** `social.coves.aggregator.authorization`
321
-
**Key:** `any` (one per aggregator per community)
322
-
**Location:** Community's repository
324
-
This record grants an aggregator permission to post to a community and contains aggregator-specific configuration.
329
-
"id": "social.coves.aggregator.authorization",
333
-
"description": "Authorization for an aggregator to post to a community with specific configuration",
337
-
"required": ["aggregatorDid", "communityDid", "createdAt", "enabled"],
342
-
"description": "DID of the authorized aggregator"
347
-
"description": "DID of the community granting access (must match repo DID)"
351
-
"description": "Whether this aggregator is currently active. Can be toggled without deleting the record."
355
-
"description": "Aggregator-specific configuration. Must conform to the aggregator's configSchema."
359
-
"format": "datetime"
364
-
"description": "DID of moderator who authorized this aggregator"
368
-
"format": "datetime",
369
-
"description": "When this authorization was disabled (if enabled=false)"
374
-
"description": "DID of moderator who disabled this aggregator"
383
-
**Example Record:**
386
-
"$type": "social.coves.aggregator.authorization",
387
-
"aggregatorDid": "did:web:rss-bot.coves.social",
388
-
"communityDid": "did:plc:gaming123",
392
-
"https://techcrunch.com/feed",
393
-
"https://arstechnica.com/feed"
395
-
"topics": ["technology", "ai", "gaming"],
396
-
"dedupeWindow": "6h",
399
-
"createdAt": "2025-10-19T14:00:00Z",
400
-
"createdBy": "did:plc:alice123"
406
-
### 3. Aggregator Type Definitions
408
-
**Collection:** `social.coves.aggregator.types`
409
-
**Purpose:** Define known aggregator types for categorization
414
-
"id": "social.coves.aggregator.types",
418
-
"description": "Aggregator that monitors RSS/Atom feeds"
422
-
"description": "Aggregator that mirrors Bluesky posts"
426
-
"description": "Aggregator for sports scores and game threads"
430
-
"description": "Aggregator for TV/movie discussion threads"
434
-
"description": "Custom third-party aggregator"
89
+
Aggregator Service (External)
91
+
│ 1. Authenticates as aggregator DID (JWT)
92
+
│ 2. Calls social.coves.post.create
94
+
Coves AppView Handler
96
+
│ 1. Extract DID from JWT
97
+
│ 2. Check if DID is registered aggregator
98
+
│ 3. Validate authorization exists & enabled
99
+
│ 4. Apply aggregator rate limits
100
+
│ 5. Create post with author = aggregator DID
102
+
Jetstream → AppView Indexing
104
+
│ Post indexed with aggregator attribution
105
+
│ UI shows: "🤖 Posted by [Aggregator Name]"
···
### For Communities (Moderators)
446
-
#### `social.coves.aggregator.enable`
447
-
Enable an aggregator for a community
452
-
"aggregatorDid": "did:web:rss-bot.coves.social",
454
-
"feeds": ["https://techcrunch.com/feed"],
455
-
"topics": ["technology"]
463
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
464
-
"cid": "bafyreif5...",
466
-
"aggregatorDid": "did:web:rss-bot.coves.social",
467
-
"communityDid": "did:plc:gaming123",
470
-
"createdAt": "2025-10-19T14:00:00Z"
476
-
- Validates caller is community moderator
477
-
- Validates aggregator exists and has service declaration
478
-
- Validates config against aggregator's configSchema
479
-
- Creates authorization record in community's repo
480
-
- Indexes to AppView for authorization checks
483
-
- `NotAuthorized` - Caller is not a moderator
484
-
- `AggregatorNotFound` - Aggregator DID doesn't exist
485
-
- `InvalidConfig` - Config doesn't match configSchema
489
-
#### `social.coves.aggregator.disable`
490
-
Disable an aggregator for a community
495
-
"aggregatorDid": "did:web:rss-bot.coves.social"
502
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
504
-
"disabledAt": "2025-10-19T15:00:00Z"
509
-
- Validates caller is community moderator
510
-
- Updates authorization record (sets `enabled=false`, `disabledAt`, `disabledBy`)
511
-
- Aggregator can no longer post until re-enabled
515
-
#### `social.coves.aggregator.updateConfig`
516
-
Update configuration for an enabled aggregator
521
-
"aggregatorDid": "did:web:rss-bot.coves.social",
523
-
"feeds": ["https://techcrunch.com/feed", "https://arstechnica.com/feed"],
524
-
"topics": ["technology", "gaming"]
532
-
"uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",
533
-
"cid": "bafyreif6...",
540
-
#### `social.coves.aggregator.listForCommunity`
541
-
List all aggregators (enabled and disabled) for a community
546
-
"community": "did:plc:gaming123",
547
-
"enabledOnly": false,
558
-
"aggregatorDid": "did:web:rss-bot.coves.social",
559
-
"displayName": "RSS News Aggregator",
560
-
"description": "...",
561
-
"aggregatorType": "social.coves.aggregator.types#rss",
564
-
"createdAt": "2025-10-19T14:00:00Z"
116
+
- **`social.coves.aggregator.enable`** - Create authorization record
117
+
- **`social.coves.aggregator.disable`** - Set enabled=false
118
+
- **`social.coves.aggregator.updateConfig`** - Update config
119
+
- **`social.coves.aggregator.listForCommunity`** - List aggregators for community
575
-
#### Existing: `social.coves.post.create`
576
-
**Modified Behavior:** Now handles aggregator authentication
578
-
**Authorization Flow:**
579
-
1. Extract DID from JWT
580
-
2. Check if DID is registered aggregator (query `aggregators` table)
582
-
- Validate authorization record exists for this community
583
-
- Check `enabled=true`
584
-
- Apply aggregator rate limits (e.g., 10 posts/hour)
585
-
4. If regular user:
586
-
- Validate membership, bans, etc. (existing logic)
587
-
5. Create post with `author = actorDID`
590
-
- Regular users: 20 posts/hour per community
591
-
- Aggregators: 10 posts/hour per community (to prevent spam)
595
-
#### `social.coves.aggregator.getAuthorizations`
596
-
Get list of communities that have authorized this aggregator
601
-
"aggregatorDid": "did:web:rss-bot.coves.social",
602
-
"enabledOnly": true,
611
-
"authorizations": [
613
-
"communityDid": "did:plc:gaming123",
614
-
"communityName": "Gaming News",
617
-
"createdAt": "2025-10-19T14:00:00Z"
624
-
**Use Case:** Aggregator queries this to know which communities to post to
123
+
- **`social.coves.post.create`** - Modified to handle aggregator auth
124
+
- **`social.coves.aggregator.getAuthorizations`** - Query authorized communities
630
-
#### `social.coves.aggregator.list`
631
-
List all available aggregators
636
-
"type": "social.coves.aggregator.types#rss",
647
-
"did": "did:web:rss-bot.coves.social",
648
-
"displayName": "RSS News Aggregator",
649
-
"description": "...",
650
-
"aggregatorType": "social.coves.aggregator.types#rss",
652
-
"maintainer": "did:plc:coves-platform",
653
-
"sourceUrl": "https://github.com/coves-social/rss-aggregator"
662
-
#### `social.coves.aggregator.get`
663
-
Get detailed information about a specific aggregator
668
-
"aggregatorDid": "did:web:rss-bot.coves.social"
675
-
"did": "did:web:rss-bot.coves.social",
676
-
"displayName": "RSS News Aggregator",
677
-
"description": "...",
678
-
"aggregatorType": "social.coves.aggregator.types#rss",
679
-
"configSchema": {...},
680
-
"sourceUrl": "...",
681
-
"maintainer": "...",
683
-
"communitiesUsing": 42,
684
-
"postsCreated": 1337,
685
-
"createdAt": "2025-10-19T12:00:00Z"
128
+
- **`social.coves.aggregator.getServices`** - Fetch aggregator details by DID(s)
695
-
Indexed aggregator service declarations from Jetstream
698
-
CREATE TABLE aggregators (
699
-
did TEXT PRIMARY KEY,
700
-
display_name TEXT NOT NULL,
702
-
aggregator_type TEXT NOT NULL,
703
-
config_schema JSONB,
706
-
maintainer_did TEXT,
708
-
-- Indexing metadata
709
-
record_uri TEXT NOT NULL,
710
-
record_cid TEXT NOT NULL,
711
-
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
714
-
communities_using INTEGER NOT NULL DEFAULT 0,
715
-
posts_created BIGINT NOT NULL DEFAULT 0,
135
+
Indexes aggregator service declarations from Jetstream.
717
-
CONSTRAINT aggregators_type_check CHECK (
718
-
aggregator_type IN (
719
-
'social.coves.aggregator.types#rss',
720
-
'social.coves.aggregator.types#blueskyMirror',
721
-
'social.coves.aggregator.types#sports',
722
-
'social.coves.aggregator.types#media',
723
-
'social.coves.aggregator.types#custom'
728
-
CREATE INDEX idx_aggregators_type ON aggregators(aggregator_type);
729
-
CREATE INDEX idx_aggregators_indexed_at ON aggregators(indexed_at DESC);
138
+
- `did` (PK) - Aggregator DID
139
+
- `display_name`, `description` - Service metadata
140
+
- `config_schema` - JSON Schema for config validation
141
+
- `avatar_url`, `source_url`, `maintainer_did` - Metadata
142
+
- `record_uri`, `record_cid` - atProto record metadata
143
+
- `communities_using`, `posts_created` - Cached stats (updated by triggers)
### `aggregator_authorizations` Table
735
-
Indexed authorization records from communities
146
+
Indexes community authorization records from Jetstream.
738
-
CREATE TABLE aggregator_authorizations (
739
-
id BIGSERIAL PRIMARY KEY,
149
+
- `aggregator_did`, `community_did` - Authorization pair (unique together)
150
+
- `enabled` - Active status
151
+
- `config` - Community-specific JSON config
152
+
- `created_by`, `disabled_by` - Audit trail
153
+
- `record_uri`, `record_cid` - atProto record metadata
741
-
-- Authorization identity
742
-
aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,
743
-
community_did TEXT NOT NULL,
155
+
**Critical Indexes:**
156
+
- `idx_aggregator_auth_lookup` - Fast (aggregator_did, community_did, enabled) lookups for post creation
745
-
-- Authorization state
746
-
enabled BOOLEAN NOT NULL DEFAULT true,
158
+
### `aggregator_posts` Table
159
+
AppView-only tracking for rate limiting and stats (not from lexicon).
750
-
created_at TIMESTAMPTZ NOT NULL,
751
-
created_by TEXT NOT NULL, -- DID of moderator
752
-
disabled_at TIMESTAMPTZ,
753
-
disabled_by TEXT, -- DID of moderator
755
-
-- atProto record metadata
756
-
record_uri TEXT NOT NULL UNIQUE,
757
-
record_cid TEXT NOT NULL,
758
-
indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
760
-
UNIQUE(aggregator_did, community_did)
763
-
CREATE INDEX idx_aggregator_auth_agg_did ON aggregator_authorizations(aggregator_did) WHERE enabled = true;
764
-
CREATE INDEX idx_aggregator_auth_comm_did ON aggregator_authorizations(community_did) WHERE enabled = true;
765
-
CREATE INDEX idx_aggregator_auth_enabled ON aggregator_authorizations(enabled);
162
+
- `aggregator_did`, `community_did`, `post_uri`
163
+
- `created_at` - For rate limit calculations
770
-
### `aggregator_posts` Table
771
-
Track posts created by aggregators (for rate limiting and stats)
774
-
CREATE TABLE aggregator_posts (
775
-
id BIGSERIAL PRIMARY KEY,
777
-
aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,
778
-
community_did TEXT NOT NULL,
779
-
post_uri TEXT NOT NULL,
780
-
post_cid TEXT NOT NULL,
170
+
- DID-based authentication via JWT signatures
171
+
- No shared secrets or API keys
172
+
- Aggregators can only post to authorized communities
782
-
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
174
+
### Authorization Checks
175
+
- Server validates aggregator status (not client-provided)
176
+
- Checks `aggregator_authorizations` table on every post
177
+
- Config validated against aggregator's JSON schema
180
+
- Aggregators: 10 posts/hour per community
181
+
- Tracked via `aggregator_posts` table
787
-
CREATE INDEX idx_aggregator_posts_agg_did_created ON aggregator_posts(aggregator_did, created_at DESC);
788
-
CREATE INDEX idx_aggregator_posts_comm_did_created ON aggregator_posts(community_did, created_at DESC);
790
-
-- For rate limiting: count posts in last hour
791
-
CREATE INDEX idx_aggregator_posts_rate_limit ON aggregator_posts(aggregator_did, community_did, created_at DESC);
185
+
- `created_by` / `disabled_by` track moderator actions
186
+
- Full history preserved in authorization records
796
-
## Implementation Plan
190
+
## Implementation Phases
798
-
### Phase 1: Core Infrastructure (Coves AppView)
192
+
### ✅ Phase 1: Core Infrastructure (COMPLETE)
193
+
**Status:** ✅ COMPLETE - All components implemented and tested
**Goal:** Enable aggregator authentication and authorization
802
-
#### 1.1 Database Setup
803
-
- [ ] Create migration for `aggregators` table
804
-
- [ ] Create migration for `aggregator_authorizations` table
805
-
- [ ] Create migration for `aggregator_posts` table
807
-
#### 1.2 Lexicon Definitions
808
-
- [ ] Create `social.coves.aggregator.service.json`
809
-
- [ ] Create `social.coves.aggregator.authorization.json`
810
-
- [ ] Create `social.coves.aggregator.types.json`
811
-
- [ ] Generate Go types from lexicons
813
-
#### 1.3 Repository Layer
815
-
// internal/core/aggregators/repository.go
817
-
type Repository interface {
818
-
// Aggregator management
819
-
CreateAggregator(ctx context.Context, agg *Aggregator) error
820
-
GetAggregator(ctx context.Context, did string) (*Aggregator, error)
821
-
ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)
822
-
UpdateAggregatorStats(ctx context.Context, did string, stats Stats) error
824
-
// Authorization management
825
-
CreateAuthorization(ctx context.Context, auth *Authorization) error
826
-
GetAuthorization(ctx context.Context, aggDID, commDID string) (*Authorization, error)
827
-
ListAuthorizationsForAggregator(ctx context.Context, aggDID string, enabledOnly bool) ([]*Authorization, error)
828
-
ListAuthorizationsForCommunity(ctx context.Context, commDID string) ([]*Authorization, error)
829
-
UpdateAuthorization(ctx context.Context, auth *Authorization) error
830
-
IsAuthorized(ctx context.Context, aggDID, commDID string) (bool, error)
832
-
// Post tracking (for rate limiting)
833
-
RecordAggregatorPost(ctx context.Context, aggDID, commDID, postURI string) error
834
-
CountRecentPosts(ctx context.Context, aggDID, commDID string, since time.Time) (int, error)
838
-
#### 1.4 Service Layer
840
-
// internal/core/aggregators/service.go
842
-
type Service interface {
843
-
// For communities (moderators)
844
-
EnableAggregator(ctx context.Context, commDID, aggDID string, config map[string]interface{}) (*Authorization, error)
845
-
DisableAggregator(ctx context.Context, commDID, aggDID string) error
846
-
UpdateAggregatorConfig(ctx context.Context, commDID, aggDID string, config map[string]interface{}) error
847
-
ListCommunityAggregators(ctx context.Context, commDID string, enabledOnly bool) ([]*AggregatorInfo, error)
850
-
GetAuthorizedCommunities(ctx context.Context, aggDID string) ([]*CommunityAuth, error)
853
-
ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)
854
-
GetAggregator(ctx context.Context, did string) (*AggregatorDetail, error)
856
-
// Internal: called by post creation handler
857
-
ValidateAggregatorPost(ctx context.Context, aggDID, commDID string) error
197
+
- ✅ Lexicon schemas (9 files)
198
+
- ✅ Database migrations (2 migrations: 3 tables, 2 triggers, indexes)
199
+
- ✅ Repository layer (CRUD operations, bulk queries, optimized indexes)
200
+
- ✅ Service layer (business logic, validation, rate limiting)
201
+
- ✅ Modified post creation handler (aggregator authentication & authorization)
202
+
- ✅ XRPC query handlers (getServices, getAuthorizations, listForCommunity)
203
+
- ✅ Jetstream consumer (indexes service & authorization records from firehose)
204
+
- ✅ Integration tests (10+ test suites, E2E validation)
205
+
- ✅ E2E test validation (verified records exist in both PDS and AppView)
861
-
#### 1.5 Modify Post Creation Handler
863
-
// internal/api/handlers/post/create.go
207
+
**Milestone:** ✅ ACHIEVED - Aggregators can authenticate and post to authorized communities
865
-
func CreatePost(ctx context.Context, input *CreatePostInput) (*CreatePostOutput, error) {
866
-
actorDID := GetDIDFromAuth(ctx)
209
+
**Deferred to Phase 2:**
210
+
- Write-forward operations (enable, disable, updateConfig) - require PDS integration
211
+
- Moderator permission checks - require communities ownership validation
868
-
// Check if actor is an aggregator
869
-
if isAggregator, _ := aggregatorService.IsAggregator(ctx, actorDID); isAggregator {
870
-
// Validate aggregator authorization
871
-
if err := aggregatorService.ValidateAggregatorPost(ctx, actorDID, input.Community); err != nil {
875
-
// Apply aggregator rate limits
876
-
if err := rateLimitAggregator(ctx, actorDID, input.Community); err != nil {
877
-
return nil, ErrRateLimitExceeded
880
-
// Regular user validation (existing logic)
881
-
// ... membership checks, ban checks, etc.
215
+
### Phase 2: Aggregator SDK (Post-Alpha)
216
+
**Deferred** - Will build SDK after Phase 1 is validated in production.
884
-
// Create post (author will be actorDID - either user or aggregator)
885
-
post, err := postService.CreatePost(ctx, actorDID, input)
218
+
Core functionality works without SDK - aggregators just need to:
219
+
1. Create atProto account (get DID)
220
+
2. Publish service declaration record
221
+
3. Sign JWTs with their DID keys
222
+
4. Call existing XRPC endpoints
890
-
// If aggregator, track the post
892
-
_ = aggregatorService.RecordPost(ctx, actorDID, input.Community, post.URI)
226
+
### Phase 3: Reference Implementation (Future)
227
+
**Deferred** - First aggregator will likely be built inline to validate the system.
899
-
#### 1.6 XRPC Handlers
900
-
- [ ] `social.coves.aggregator.enable` handler
901
-
- [ ] `social.coves.aggregator.disable` handler
902
-
- [ ] `social.coves.aggregator.updateConfig` handler
903
-
- [ ] `social.coves.aggregator.listForCommunity` handler
904
-
- [ ] `social.coves.aggregator.getAuthorizations` handler
905
-
- [ ] `social.coves.aggregator.list` handler
906
-
- [ ] `social.coves.aggregator.get` handler
229
+
Potential first aggregator: RSS news bot for select communities.
908
-
#### 1.7 Jetstream Consumer
910
-
// internal/atproto/jetstream/aggregator_consumer.go
912
-
func (c *AggregatorConsumer) HandleEvent(ctx context.Context, evt *jetstream.Event) error {
913
-
switch evt.Collection {
914
-
case "social.coves.aggregator.service":
915
-
switch evt.Operation {
916
-
case "create", "update":
917
-
return c.indexAggregatorService(ctx, evt)
919
-
return c.deleteAggregator(ctx, evt.DID)
233
+
## Key Design Decisions
922
-
case "social.coves.aggregator.authorization":
923
-
switch evt.Operation {
924
-
case "create", "update":
925
-
return c.indexAuthorization(ctx, evt)
927
-
return c.deleteAuthorization(ctx, evt.URI)
235
+
### 2025-10-20: Remove `aggregatorType` Field
236
+
**Decision:** Removed `aggregatorType` enum from service declaration and database.
934
-
#### 1.8 Integration Tests
935
-
- [ ] Test aggregator service indexing from Jetstream
936
-
- [ ] Test authorization record indexing
937
-
- [ ] Test `social.coves.post.create` with aggregator auth
938
-
- [ ] Test authorization validation (enabled/disabled)
939
-
- [ ] Test rate limiting for aggregators
940
-
- [ ] Test config validation against schema
239
+
- Pre-production - can break things
240
+
- Over-engineering for alpha
241
+
- Description field is sufficient for discovery
242
+
- Avoids rigid categorization
243
+
- Can add tags later if needed
942
-
**Milestone:** Aggregators can authenticate and post to communities with authorization
246
+
- Simplified lexicons
247
+
- Removed database constraint
248
+
- More flexible for third-party developers
946
-
### Phase 2: Aggregator SDK (Go)
252
+
### 2025-10-19: Reuse `social.coves.post.create` Endpoint
253
+
**Decision:** Aggregators use existing post creation endpoint.
948
-
**Goal:** Provide SDK for building aggregators easily
952
-
// github.com/coves-social/aggregator-sdk-go
256
+
- Post record already server-populates `author` from JWT
257
+
- Simpler: one code path for all post creation
258
+
- Follows atProto principle: actors are actors
259
+
- `federatedFrom` field handles external content attribution
956
-
type Aggregator interface {
959
-
GetDisplayName() string
960
-
GetDescription() string
962
-
GetConfigSchema() map[string]interface{}
965
-
Start(ctx context.Context) error
968
-
// Posting (provided by SDK)
969
-
CreatePost(ctx context.Context, communityDID string, post Post) error
970
-
GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error)
261
+
**Implementation:**
262
+
- Add branching logic in post handler: if aggregator, check authorization; else check membership
263
+
- Apply different rate limits based on actor type
973
-
type BaseAggregator struct {
978
-
PrivateKey crypto.PrivateKey
981
-
client *http.Client
267
+
### 2025-10-19: Config as JSON Schema
268
+
**Decision:** Aggregators declare `configSchema` in service record.
988
-
FederatedFrom *FederatedSource
989
-
ContentLabels []string
271
+
- Communities need to know what config options are available
272
+
- JSON Schema is standard and well-supported
273
+
- Enables UI auto-generation (forms from schema)
274
+
- Validation at authorization creation time
275
+
- Flexible: each aggregator has different config needs
992
-
type FederatedSource struct {
993
-
Platform string // "rss", "bluesky", etc.
996
-
OriginalCreatedAt time.Time
999
-
// Helper methods provided by SDK
1000
-
func (a *BaseAggregator) CreatePost(ctx context.Context, communityDID string, post Post) error {
1001
-
// 1. Sign JWT with aggregator's private key
1002
-
token := a.signJWT()
1004
-
// 2. Call social.coves.post.create via XRPC
1005
-
resp, err := a.client.Post(
1006
-
a.CovesAPIURL + "/xrpc/social.coves.post.create",
1008
-
Community: communityDID,
1009
-
Title: post.Title,
1010
-
Content: post.Content,
1011
-
Embed: post.Embed,
1012
-
FederatedFrom: post.FederatedFrom,
1013
-
ContentLabels: post.ContentLabels,
1015
-
&CreatePostOutput{},
281
+
### RSS News Aggregator
282
+
Watches configured RSS feeds, uses LLM for deduplication, posts news articles to community.
284
+
**Community Config Example:**
287
+
"feeds": ["https://techcrunch.com/feed"],
288
+
"topics": ["technology"],
289
+
"dedupeWindow": "6h"
1022
-
func (a *BaseAggregator) GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error) {
1023
-
// Call social.coves.aggregator.getAuthorizations
1024
-
token := a.signJWT()
1026
-
resp, err := a.client.Get(
1027
-
a.CovesAPIURL + "/xrpc/social.coves.aggregator.getAuthorizations",
1028
-
map[string]string{"aggregatorDid": a.DID, "enabledOnly": "true"},
1029
-
&GetAuthorizationsOutput{},
295
+
### Bluesky Post Mirror
296
+
Monitors specific users/hashtags on Bluesky, creates posts in community with original author metadata.
1033
-
return resp.Authorizations, err
298
+
**Community Config Example:**
301
+
"mirrorUsers": ["alice.bsky.social"],
302
+
"hashtags": ["covesalpha"],
1037
-
#### 2.2 SDK Documentation
1038
-
- [ ] README with quickstart guide
1039
-
- [ ] Example aggregators (RSS, Bluesky mirror)
1040
-
- [ ] API reference documentation
1041
-
- [ ] Configuration schema guide
1043
-
**Milestone:** Third parties can build aggregators using SDK
1047
-
### Phase 3: Reference Aggregator (RSS)
1049
-
**Goal:** Build working RSS aggregator as reference implementation
1051
-
#### 3.1 RSS Aggregator Implementation
1053
-
// github.com/coves-social/rss-aggregator
1057
-
import "github.com/coves-social/aggregator-sdk-go"
1059
-
type RSSAggregator struct {
1060
-
*aggregator.BaseAggregator
1062
-
// RSS-specific config
1063
-
pollInterval time.Duration
1064
-
llmClient *openai.Client
1067
-
func (r *RSSAggregator) Start(ctx context.Context) error {
1068
-
// 1. Get authorized communities
1069
-
communities, err := r.GetAuthorizedCommunities(ctx)
1074
-
// 2. Start polling loop
1075
-
ticker := time.NewTicker(r.pollInterval)
1076
-
defer ticker.Stop()
1081
-
r.pollFeeds(ctx, communities)
1082
-
case <-ctx.Done():
1088
-
func (r *RSSAggregator) pollFeeds(ctx context.Context, communities []*CommunityAuth) {
1089
-
for _, comm := range communities {
1090
-
// Get RSS feeds from community config
1091
-
feeds := comm.Config["feeds"].([]string)
1093
-
for _, feedURL := range feeds {
1094
-
items, err := r.fetchFeed(feedURL)
1099
-
// Process new items
1100
-
for _, item := range items {
1101
-
// Check if already posted
1102
-
if r.alreadyPosted(item.GUID) {
1106
-
// LLM deduplication logic
1107
-
duplicate := r.findDuplicate(item, comm.CommunityDID)
1108
-
if duplicate != nil {
1109
-
r.addToMegathread(duplicate, item)
1113
-
// Create new post
1114
-
post := aggregator.Post{
1115
-
Title: item.Title,
1116
-
Content: r.summarize(item),
1117
-
FederatedFrom: &aggregator.FederatedSource{
1120
-
OriginalCreatedAt: item.PublishedAt,
1124
-
err = r.CreatePost(ctx, comm.CommunityDID, post)
1126
-
log.Printf("Failed to create post: %v", err)
1130
-
r.markPosted(item.GUID)
1136
-
func (r *RSSAggregator) summarize(item *RSSItem) string {
1137
-
// Use LLM to create unbiased summary
1138
-
prompt := fmt.Sprintf("Summarize this news article in 2-3 sentences: %s", item.Description)
1139
-
summary, _ := r.llmClient.Complete(prompt)
310
+
Monitors sports APIs, creates post-game threads with scores and stats.
1143
-
func (r *RSSAggregator) findDuplicate(item *RSSItem, communityDID string) *Post {
1144
-
// Use LLM to detect semantic duplicates
1145
-
// Query recent posts in community
1146
-
// Compare with embeddings/similarity
1147
-
return nil // or duplicate post
312
+
**Community Config Example:**
316
+
"teams": ["Lakers", "Warriors"],
317
+
"includeStats": true
1151
-
#### 3.2 Deployment
1152
-
- [ ] Dockerfile for RSS aggregator
1153
-
- [ ] Kubernetes manifests (for Coves-hosted instance)
1154
-
- [ ] Environment configuration guide
1155
-
- [ ] Monitoring and logging setup
1158
-
- [ ] Unit tests for feed parsing
1159
-
- [ ] Integration tests with mock Coves API
1160
-
- [ ] E2E test with real Coves instance
1161
-
- [ ] LLM deduplication accuracy tests
1163
-
**Milestone:** RSS aggregator running in production for select communities
1167
-
### Phase 4: Additional Aggregators
1169
-
#### 4.1 Bluesky Mirror Aggregator
1170
-
- [ ] Monitor Jetstream for specific users/hashtags
1171
-
- [ ] Preserve `originalAuthor` metadata
1172
-
- [ ] Link back to original Bluesky post
1173
-
- [ ] Rate limiting (don't flood community)
1175
-
#### 4.2 Sports Aggregator
1176
-
- [ ] Integrate with ESPN/TheSportsDB APIs
1177
-
- [ ] Monitor game completions
1178
-
- [ ] Create post-game threads with stats
1179
-
- [ ] Auto-pin major games
1181
-
#### 4.3 Media (TV/Movie) Aggregator
1182
-
- [ ] Integrate with TMDB API
1183
-
- [ ] Track show release schedules
1184
-
- [ ] Create episode discussion threads
1185
-
- [ ] Spoiler protection tags
1187
-
**Milestone:** Multiple official aggregators available for communities
1191
-
## Security Considerations
1193
-
### Authentication
1194
-
✅ **DID-based Authentication**
1195
-
- Aggregators sign JWTs with their private keys
1196
-
- Server validates JWT signature against DID document
1197
-
- No shared secrets or API keys
1199
-
✅ **Scoped Authorization**
1200
-
- Authorization records are per-community
1201
-
- Aggregator can only post to authorized communities
1202
-
- Communities can revoke at any time
1205
-
✅ **Per-Aggregator Limits**
1206
-
- 10 posts/hour per community (configurable)
1207
-
- Prevents aggregator spam
1208
-
- Separate from user rate limits
1210
-
✅ **Global Limits**
1211
-
- Total posts across all communities: 100/hour
1212
-
- Prevents runaway aggregators
1214
-
### Content Validation
1215
-
✅ **Community Rules**
1216
-
- Aggregator posts validated against community content rules
1217
-
- No special exemptions (same rules as users)
1218
-
- Community can ban specific content patterns
1220
-
✅ **Config Validation**
1221
-
- Authorization config validated against aggregator's configSchema
1222
-
- Prevents injection attacks via config
1223
-
- JSON schema validation
1225
-
### Monitoring & Auditing
1227
-
- All aggregator posts logged
1228
-
- `created_by` tracks which moderator authorized
1229
-
- `disabled_by` tracks who revoked access
1230
-
- Full history preserved
1232
-
✅ **Abuse Detection**
1233
-
- Monitor for spam patterns
1234
-
- Alert if aggregator posts rejected repeatedly
1235
-
- Auto-disable after threshold violations
1239
-
- Official aggregators open source
1240
-
- Source URL in service declaration
1241
-
- Community can audit behavior
1244
-
- Posts clearly show aggregator authorship
1245
-
- UI shows "🤖 Posted by [Aggregator Name]"
1246
-
- No attempt to impersonate users
1250
-
## UI/UX Considerations
1252
-
### Community Settings
1253
-
**Aggregator Management Page:**
1254
-
- List of available aggregators (with descriptions, types)
1255
-
- "Enable" button opens config modal
1256
-
- Config form generated from aggregator's configSchema
1257
-
- Toggle to enable/disable without deleting config
1258
-
- Stats: posts created, last active
1261
-
- Posts from aggregators have bot badge: "🤖"
1262
-
- Shows aggregator name (e.g., "Posted by RSS News Bot")
1263
-
- `federatedFrom` shows original source
1264
-
- Link to original content (RSS article, Bluesky post, etc.)
1266
-
### User Preferences
1267
-
- Option to hide all aggregator posts
1268
-
- Option to hide specific aggregators
1269
-
- Filter posts by "user-created only" or "include bots"
1275
-
### Pre-Launch Checklist
1276
-
- [ ] Lexicons defined and validated
1277
-
- [ ] Database migrations tested
1278
-
- [ ] Jetstream consumer indexes aggregator records
1279
-
- [ ] Post creation handler validates aggregator auth
1280
-
- [ ] Rate limiting prevents spam
1281
-
- [ ] SDK published and documented
1282
-
- [ ] Reference RSS aggregator working
1283
-
- [ ] E2E tests passing
1284
-
- [ ] Security audit completed
1287
-
- 3+ official aggregators (RSS, Bluesky mirror, sports)
1288
-
- 10+ communities using aggregators
1289
-
- < 0.1% spam posts (false positives)
1290
-
- Aggregator posts appear in feed within 1 minute
326
+
- ✅ Lexicons validated
327
+
- ✅ Database migrations tested
328
+
- ⏳ Jetstream consumer indexes records
329
+
- ⏳ Post creation validates aggregator auth
330
+
- ⏳ Rate limiting prevents spam
331
+
- ⏳ Integration tests passing
1293
-
- Third-party aggregators launched
1294
-
- 50+ communities using aggregators
1295
-
- Developer documentation complete
1296
-
- Marketplace/directory for discovery
333
+
### Beta Goals (Future)
334
+
- First aggregator deployed in production
335
+
- 3+ communities using aggregators
336
+
- < 0.1% spam posts
337
+
- Third-party developer documentation
1300
-
## Out of Scope (Future Versions)
341
+
## Out of Scope (Future)
1302
-
### Aggregator Marketplace
1303
-
- [ ] Community ratings/reviews for aggregators
1304
-
- [ ] Featured aggregators
1305
-
- [ ] Paid aggregators (premium features)
1306
-
- [ ] Aggregator analytics dashboard
1308
-
### Advanced Features
1309
-
- [ ] Scheduled posts (post at specific time)
1310
-
- [ ] Content moderation integration (auto-label NSFW)
1311
-
- [ ] Multi-community posting (single post to multiple communities)
1312
-
- [ ] Interactive aggregators (respond to comments)
1313
-
- [ ] Aggregator-to-aggregator communication (chains)
1316
-
- [ ] Cross-instance aggregator discovery
1317
-
- [ ] Aggregator migration (change hosting provider)
1318
-
- [ ] Federated aggregator authorization (trust other instances' aggregators)
1322
-
## Technical Decisions Log
1324
-
### 2025-10-19: Reuse `social.coves.post.create` Endpoint
1326
-
**Decision:** Aggregators use existing post creation endpoint, not a separate `social.coves.aggregator.post.create`
1329
-
- Post record already server-populates `author` field from JWT
1330
-
- Aggregators authenticate as themselves → `author = aggregator DID`
1331
-
- Simpler: one code path for all post creation
1332
-
- Follows atProto principle: actors are actors (users, bots, aggregators)
1333
-
- `federatedFrom` field already handles external content attribution
1335
-
**Implementation:**
1336
-
- Add authorization check to `social.coves.post.create` handler
1337
-
- Check if authenticated DID is aggregator
1338
-
- Validate authorization record exists and enabled
1339
-
- Apply aggregator-specific rate limits
1340
-
- Otherwise same logic as user posts
1342
-
**Trade-offs Accepted:**
1343
-
- Post creation handler has branching logic (user vs aggregator)
1344
-
- But: keeps lexicon simple, reuses existing validation
1348
-
### 2025-10-19: Hybrid Hosting Model
1350
-
**Decision:** Support both Coves-hosted and third-party aggregators
1353
-
- Coves can provide high-quality official aggregators (RSS, sports, media)
1354
-
- Third parties can build specialized aggregators (niche communities)
1355
-
- SDK makes it easy to build custom aggregators
1356
-
- Follows feed generator model (anyone can run one)
1357
-
- Decentralization-friendly
1360
-
- SDK must be well-documented and maintained
1361
-
- Authorization system must be DID-agnostic (works for any DID)
1362
-
- Discovery system shows all aggregators (official + third-party)
1366
-
### 2025-10-19: Config as JSON Schema
1368
-
**Decision:** Aggregators declare configSchema in their service record
1371
-
- Communities need to know what config options are available
1372
-
- JSON Schema is standard, well-supported
1373
-
- Enables UI auto-generation (forms from schema)
1374
-
- Validation at authorization creation time
1375
-
- Flexible: each aggregator can have different config structure
1385
-
"items": { "type": "string", "format": "uri" },
1386
-
"description": "RSS feed URLs to monitor"
1390
-
"items": { "type": "string" },
1391
-
"description": "Topics to filter posts by"
1394
-
"required": ["feeds"]
1399
-
**Trade-offs Accepted:**
1400
-
- More complex than simple key-value config
1401
-
- But: better UX (self-documenting), prevents errors
343
+
- Aggregator marketplace with ratings/reviews
344
+
- UI for aggregator management (alpha uses XRPC only)
346
+
- Interactive aggregators (respond to comments)
347
+
- Cross-instance aggregator discovery
348
+
- SDK (deferred until post-alpha)
349
+
- LLM features (deferred)
- atProto Lexicon Spec: https://atproto.com/specs/lexicon
1408
-
- Feed Generator Starter Kit: https://github.com/bluesky-social/feed-generator
1409
-
- Labeler Implementation: https://github.com/bluesky-social/atproto/tree/main/packages/ozone
1410
-
- JSON Schema Spec: https://json-schema.org/
1411
-
- Coves Communities PRD: [PRD_COMMUNITIES.md](PRD_COMMUNITIES.md)
1412
-
- Coves Posts Implementation: [IMPLEMENTATION_POST_CREATION.md](IMPLEMENTATION_POST_CREATION.md)
356
+
- Feed Generator Pattern: https://github.com/bluesky-social/feed-generator
357
+
- Labeler Pattern: https://github.com/bluesky-social/atproto/tree/main/packages/ozone
358
+
- JSON Schema: https://json-schema.org/
bretton.dev