commit 87037e024f971b1b90b4355d26c8aacc2925e07a · bretton.dev/coves

+795

docs/COMMUNITY_FEEDS.md

···

       1
       1
       +
       # Community Feeds Implementation

     

       2
       2
       +
       

     

       3
       3
       +
       **Status:** ✅ Implemented (Alpha)

     

       4
       4
       +
       **PR:** #1 - Community Feed Discovery

     

       5
       5
       +
       **Date:** October 2025

     

       6
       6
       +
       

     

       7
       7
       +
       ---

     

       8
       8
       +
       

     

       9
       9
       +
       ## Problem Statement

     

       10
       10
       +
       

     

       11
       11
       +
       ### What We're Solving

     

       12
       12
       +
       

     

       13
       13
       +
       Users need a way to **browse and discover posts** in communities. Before this implementation:

     

       14
       14
       +
       

     

       15
       15
       +
       ❌ **No way to see what's in a community**

     

       16
       16
       +
       - Users could create posts, but couldn't view them

     

       17
       17
       +
       - No community browsing experience

     

       18
       18
       +
       - No sorting or ranking algorithms

     

       19
       19
       +
       - No pagination for large feeds

     

       20
       20
       +
       

     

       21
       21
       +
       ❌ **Missing core forum functionality**

     

       22
       22
       +
       - Forums need "Hot", "Top", "New" sorting

     

       23
       23
       +
       - Users expect Reddit-style ranking

     

       24
       24
       +
       - Need to discover trending content

     

       25
       25
       +
       - Must handle thousands of posts per community

     

       26
       26
       +
       

     

       27
       27
       +
       ### User Stories

     

       28
       28
       +
       

     

       29
       29
       +
       1. **As a user**, I want to browse /c/gaming and see the hottest posts

     

       30
       30
       +
       2. **As a user**, I want to see top posts from this week in /c/cooking

     

       31
       31
       +
       3. **As a user**, I want to see newest posts in /c/music

     

       32
       32
       +
       4. **As a moderator**, I want posts ranked by engagement to surface quality content

     

       33
       33
       +
       

     

       34
       34
       +
       ---

     

       35
       35
       +
       

     

       36
       36
       +
       ## Solution: Hydrated Community Feeds

     

       37
       37
       +
       

     

       38
       38
       +
       ### Architecture Decision

     

       39
       39
       +
       

     

       40
       40
       +
       We chose **hydrated feeds** over Bluesky's skeleton pattern for Alpha:

     

       41
       41
       +
       

     

       42
       42
       +
       ```

     

       43
       43
       +
       ┌────────────┐

     

       44
       44
       +
       │   Client   │

     

       45
       45
       +
       └─────┬──────┘

     

       46
       46
       +
             │ GET /xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot

     

       47
       47
       +
             ▼

     

       48
       48
       +
       ┌─────────────────────┐

     

       49
       49
       +
       │   Feed Service      │ ← Validates request, resolves community DID

     

       50
       50
       +
       └─────────┬───────────┘

     

       51
       51
       +
                 ▼

     

       52
       52
       +
       ┌─────────────────────┐

     

       53
       53
       +
       │   Feed Repository   │ ← Single SQL query with JOINs

     

       54
       54
       +
       │   (PostgreSQL)      │    Returns fully hydrated posts

     

       55
       55
       +
       └─────────┬───────────┘

     

       56
       56
       +
                 ▼

     

       57
       57
       +
           [Full PostViews with author, community, stats]

     

       58
       58
       +
       ```

     

       59
       59
       +
       

     

       60
       60
       +
       **Why hydrated instead of skeleton + hydration?**

     

       61
       61
       +
       

     

       62
       62
       +
       | Criterion | Hydrated (Our Choice) | Skeleton Pattern |

     

       63
       63
       +
       |-----------|----------------------|------------------|

     

       64
       64
       +
       | **Requests** | 1 | 2 (skeleton → hydrate) |

     

       65
       65
       +
       | **Latency** | Lower | Higher |

     

       66
       66
       +
       | **Complexity** | Simple | Complex |

     

       67
       67
       +
       | **Flexibility** | Fixed algorithms | Custom feed generators |

     

       68
       68
       +
       | **Right for Alpha?** | ✅ Yes | ❌ Overkill |

     

       69
       69
       +
       | **Future-proof?** | ✅ Can add later | N/A |

     

       70
       70
       +
       

     

       71
       71
       +
       **Decision:** Ship fast with hydrated feeds now, add skeleton pattern in Beta when users request custom algorithms.

     

       72
       72
       +
       

     

       73
       73
       +
       **Alpha Scope (YAGNI):**

     

       74
       74
       +
       - ✅ Basic community sorting (hot, top, new)

     

       75
       75
       +
       - ✅ Public feeds only (no authentication required)

     

       76
       76
       +
       - ❌ Viewer state (deferred to feed generator phase)

     

       77
       77
       +
       - ❌ Custom feed algorithms (deferred to Beta)

     

       78
       78
       +
       

     

       79
       79
       +
       This keeps Alpha simple and focused on core browsing functionality.

     

       80
       80
       +
       

     

       81
       81
       +
       ---

     

       82
       82
       +
       

     

       83
       83
       +
       ## Implementation Details

     

       84
       84
       +
       

     

       85
       85
       +
       ### 1. Sorting Algorithms

     

       86
       86
       +
       

     

       87
       87
       +
       #### **Hot (Reddit Algorithm)**

     

       88
       88
       +
       

     

       89
       89
       +
       Balances score and recency for discovery:

     

       90
       90
       +
       

     

       91
       91
       +
       ```sql

     

       92
       92
       +
       ORDER BY (score / POWER(age_hours + 2, 1.5)) DESC

     

       93
       93
       +
       ```

     

       94
       94
       +
       

     

       95
       95
       +
       **How it works:**

     

       96
       96
       +
       - New posts with low scores can outrank old posts with high scores

     

       97
       97
       +
       - Decay factor (1.5) tuned for forum dynamics

     

       98
       98
       +
       - Posts "age out" naturally over time

     

       99
       99
       +
       

     

       100
       100
       +
       **Example:**

     

       101
       101
       +
       - Post A: 100 upvotes, 1 day old → Rank: 10.4

     

       102
       102
       +
       - Post B: 10 upvotes, 1 hour old → Rank: 3.5

     

       103
       103
       +
       - Post C: 50 upvotes, 12 hours old → Rank: 5.1

     

       104
       104
       +
       

     

       105
       105
       +
       **Result:** Fresh content surfaces while respecting engagement

     

       106
       106
       +
       

     

       107
       107
       +
       #### **Top (Score-Based)**

     

       108
       108
       +
       

     

       109
       109
       +
       Pure engagement ranking with timeframe filtering:

     

       110
       110
       +
       

     

       111
       111
       +
       ```sql

     

       112
       112
       +
       WHERE created_at > NOW() - INTERVAL '1 day'

     

       113
       113
       +
       ORDER BY score DESC

     

       114
       114
       +
       ```

     

       115
       115
       +
       

     

       116
       116
       +
       **Timeframes:**

     

       117
       117
       +
       - `hour` - Last 60 minutes

     

       118
       118
       +
       - `day` - Last 24 hours (default)

     

       119
       119
       +
       - `week` - Last 7 days

     

       120
       120
       +
       - `month` - Last 30 days

     

       121
       121
       +
       - `year` - Last 365 days

     

       122
       122
       +
       - `all` - All time

     

       123
       123
       +
       

     

       124
       124
       +
       #### **New (Chronological)**

     

       125
       125
       +
       

     

       126
       126
       +
       Latest first, simple and predictable:

     

       127
       127
       +
       

     

       128
       128
       +
       ```sql

     

       129
       129
       +
       ORDER BY created_at DESC

     

       130
       130
       +
       ```

     

       131
       131
       +
       

     

       132
       132
       +
       ### 2. Pagination

     

       133
       133
       +
       

     

       134
       134
       +
       **Keyset pagination** for stability:

     

       135
       135
       +
       

     

       136
       136
       +
       ```

     

       137
       137
       +
       Cursor format (base64): "score::created_at::uri"

     

       138
       138
       +
       Delimiter: :: (following Bluesky convention)

     

       139
       139
       +
       ```

     

       140
       140
       +
       

     

       141
       141
       +
       **Why keyset over offset?**

     

       142
       142
       +
       - ✅ No duplicates when new posts appear

     

       143
       143
       +
       - ✅ No skipped posts when posts are deleted

     

       144
       144
       +
       - ✅ Consistent performance at any page depth

     

       145
       145
       +
       - ✅ Works with all sort orders

     

       146
       146
       +
       

     

       147
       147
       +
       **Cursor formats by sort type:**

     

       148
       148
       +
       - `new`: `timestamp::uri` (e.g., `2025-10-20T12:00:00Z::at://...`)

     

       149
       149
       +
       - `top`/`hot`: `score::timestamp::uri` (e.g., `100::2025-10-20T12:00:00Z::at://...`)

     

       150
       150
       +
       

     

       151
       151
       +
       **Why `::` delimiter?**

     

       152
       152
       +
       - Doesn't appear in ISO timestamps (which contain single `:`)

     

       153
       153
       +
       - Doesn't appear in AT-URIs

     

       154
       154
       +
       - Bluesky convention for cursor pagination

     

       155
       155
       +
       - Prevents parsing ambiguity

     

       156
       156
       +
       

     

       157
       157
       +
       **Example cursor flow:**

     

       158
       158
       +
       ```

     

       159
       159
       +
       Page 1: No cursor

     

       160
       160
       +
         → Returns posts 1-25 + cursor="100::2025-10-20T12:00:00Z::at://..."

     

       161
       161
       +
       

     

       162
       162
       +
       Page 2: cursor from page 1

     

       163
       163
       +
         → Returns posts 26-50 + cursor="85::2025-10-20T11:30:00Z::at://..."

     

       164
       164
       +
       

     

       165
       165
       +
       Page 3: cursor from page 2

     

       166
       166
       +
         → Returns posts 51-75 + cursor (or null if end)

     

       167
       167
       +
       ```

     

       168
       168
       +
       

     

       169
       169
       +
       ### 3. Data Model

     

       170
       170
       +
       

     

       171
       171
       +
       #### **FeedViewPost** (Wrapper)

     

       172
       172
       +
       

     

       173
       173
       +
       ```go

     

       174
       174
       +
       type FeedViewPost struct {

     

       175
       175
       +
           Post   *PostView   // Full post with all metadata

     

       176
       176
       +
           Reason *FeedReason // Why in feed (pin, repost) - Beta

     

       177
       177
       +
           Reply  *ReplyRef   // Reply context - Beta

     

       178
       178
       +
       }

     

       179
       179
       +
       ```

     

       180
       180
       +
       

     

       181
       181
       +
       #### **PostView** (Hydrated Post)

     

       182
       182
       +
       

     

       183
       183
       +
       ```go

     

       184
       184
       +
       type PostView struct {

     

       185
       185
       +
           URI        string         // at://did:plc:abc/social.coves.post.record/123

     

       186
       186
       +
           CID        string         // Content ID

     

       187
       187
       +
           RKey       string         // Record key (TID)

     

       188
       188
       +
           Author     *AuthorView    // Author with handle, avatar, reputation

     

       189
       189
       +
           Community  *CommunityRef  // Community with name, avatar

     

       190
       190
       +
           Title      *string        // Post title

     

       191
       191
       +
           Text       *string        // Post content

     

       192
       192
       +
           TextFacets []interface{}  // Rich text (bold, mentions, links)

     

       193
       193
       +
           Embed      interface{}    // Union: images/video/external/quote

     

       194
       194
       +
           CreatedAt  time.Time      // When posted

     

       195
       195
       +
           IndexedAt  time.Time      // When AppView indexed it

     

       196
       196
       +
           Stats      *PostStats     // Upvotes, downvotes, score, comments

     

       197
       197
       +
           // Viewer: Not included in Alpha (deferred to feed generator phase)

     

       198
       198
       +
       }

     

       199
       199
       +
       ```

     

       200
       200
       +
       

     

       201
       201
       +
       #### **SQL Query** (Single Query Performance)

     

       202
       202
       +
       

     

       203
       203
       +
       ```sql

     

       204
       204
       +
       SELECT

     

       205
       205
       +
           p.uri, p.cid, p.rkey,

     

       206
       206
       +
           p.author_did, u.handle, u.display_name, u.avatar,  -- Author

     

       207
       207
       +
           p.community_did, c.name, c.avatar,                  -- Community

     

       208
       208
       +
           p.title, p.content, p.content_facets, p.embed,      -- Content

     

       209
       209
       +
           p.created_at, p.indexed_at,

     

       210
       210
       +
           p.upvote_count, p.downvote_count, p.score, p.comment_count

     

       211
       211
       +
       FROM posts p

     

       212
       212
       +
       INNER JOIN users u ON p.author_did = u.did

     

       213
       213
       +
       INNER JOIN communities c ON p.community_did = c.did

     

       214
       214
       +
       WHERE p.community_did = $1

     

       215
       215
       +
           AND p.deleted_at IS NULL

     

       216
       216
       +
           AND (cursor_filter)

     

       217
       217
       +
       ORDER BY (hot_rank) DESC

     

       218
       218
       +
       LIMIT 25

     

       219
       219
       +
       ```

     

       220
       220
       +
       

     

       221
       221
       +
       **Performance:** One query returns everything - no N+1, no second hydration call.

     

       222
       222
       +
       

     

       223
       223
       +
       ---

     

       224
       224
       +
       

     

       225
       225
       +
       ## API Specification

     

       226
       226
       +
       

     

       227
       227
       +
       ### Endpoint

     

       228
       228
       +
       

     

       229
       229
       +
       ```

     

       230
       230
       +
       GET /xrpc/social.coves.feed.getCommunity

     

       231
       231
       +
       ```

     

       232
       232
       +
       

     

       233
       233
       +
       ### Request Parameters

     

       234
       234
       +
       

     

       235
       235
       +
       | Parameter | Type | Required | Default | Description |

     

       236
       236
       +
       |-----------|------|----------|---------|-------------|

     

       237
       237
       +
       | `community` | string | ✅ Yes | - | Community DID or handle |

     

       238
       238
       +
       | `sort` | string | ❌ No | `"hot"` | Sort order: `hot`, `top`, `new` |

     

       239
       239
       +
       | `timeframe` | string | ❌ No | `"day"` | For `top` sort: `hour`, `day`, `week`, `month`, `year`, `all` |

     

       240
       240
       +
       | `limit` | integer | ❌ No | `15` | Posts per page (max: 50) |

     

       241
       241
       +
       | `cursor` | string | ❌ No | - | Pagination cursor from previous response |

     

       242
       242
       +
       

     

       243
       243
       +
       ### Response

     

       244
       244
       +
       

     

       245
       245
       +
       ```json

     

       246
       246
       +
       {

     

       247
       247
       +
         "feed": [

     

       248
       248
       +
           {

     

       249
       249
       +
             "post": {

     

       250
       250
       +
               "uri": "at://did:plc:gaming123/social.coves.post.record/abc",

     

       251
       251
       +
               "cid": "bafyrei...",

     

       252
       252
       +
               "author": {

     

       253
       253
       +
                 "did": "did:plc:alice",

     

       254
       254
       +
                 "handle": "alice.bsky.social",

     

       255
       255
       +
                 "displayName": "Alice",

     

       256
       256
       +
                 "avatar": "https://cdn.bsky.app/avatar/..."

     

       257
       257
       +
               },

     

       258
       258
       +
               "community": {

     

       259
       259
       +
                 "did": "did:plc:gaming123",

     

       260
       260
       +
                 "name": "gaming",

     

       261
       261
       +
                 "avatar": "https://..."

     

       262
       262
       +
               },

     

       263
       263
       +
               "title": "Just finished Elden Ring!",

     

       264
       264
       +
               "text": "What an incredible journey...",

     

       265
       265
       +
               "embed": {

     

       266
       266
       +
                 "$type": "social.coves.embed.images#view",

     

       267
       267
       +
                 "images": [

     

       268
       268
       +
                   {"fullsize": "https://...", "alt": "Final boss screenshot"}

     

       269
       269
       +
                 ]

     

       270
       270
       +
               },

     

       271
       271
       +
               "createdAt": "2025-10-20T12:00:00Z",

     

       272
       272
       +
               "indexedAt": "2025-10-20T12:00:05Z",

     

       273
       273
       +
               "stats": {

     

       274
       274
       +
                 "upvotes": 42,

     

       275
       275
       +
                 "downvotes": 3,

     

       276
       276
       +
                 "score": 39,

     

       277
       277
       +
                 "commentCount": 15

     

       278
       278
       +
               }

     

       279
       279
       +
             }

     

       280
       280
       +
           }

     

       281
       281
       +
           // ... 24 more posts

     

       282
       282
       +
         ],

     

       283
       283
       +
         "cursor": "Mzk6MjAyNS0xMC0yMFQxMjowMDowMFo6YXQ6Ly8uLi4="

     

       284
       284
       +
       }

     

       285
       285
       +
       ```

     

       286
       286
       +
       

     

       287
       287
       +
       ### Example Requests

     

       288
       288
       +
       

     

       289
       289
       +
       #### Browse hot posts in /c/gaming

     

       290
       290
       +
       ```bash

     

       291
       291
       +
       curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot&limit=25'

     

       292
       292
       +
       ```

     

       293
       293
       +
       

     

       294
       294
       +
       #### Top posts this week in /c/cooking

     

       295
       295
       +
       ```bash

     

       296
       296
       +
       curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=did:plc:cooking&sort=top&timeframe=week'

     

       297
       297
       +
       ```

     

       298
       298
       +
       

     

       299
       299
       +
       #### Page 2 of new posts

     

       300
       300
       +
       ```bash

     

       301
       301
       +
       curl 'http://localhost:8081/xrpc/social.coves.feed.getCommunity?community=gaming&sort=new&cursor=Mzk6...'

     

       302
       302
       +
       ```

     

       303
       303
       +
       

     

       304
       304
       +
       ---

     

       305
       305
       +
       

     

       306
       306
       +
       ## Error Handling

     

       307
       307
       +
       

     

       308
       308
       +
       ### Error Responses

     

       309
       309
       +
       

     

       310
       310
       +
       | Error | Status | When |

     

       311
       311
       +
       |-------|--------|------|

     

       312
       312
       +
       | `CommunityNotFound` | 404 | Community doesn't exist |

     

       313
       313
       +
       | `InvalidRequest` | 400 | Invalid parameters |

     

       314
       314
       +
       | `InvalidCursor` | 400 | Malformed pagination cursor |

     

       315
       315
       +
       | `InternalServerError` | 500 | Database or system error |

     

       316
       316
       +
       

     

       317
       317
       +
       ### Example Error

     

       318
       318
       +
       

     

       319
       319
       +
       ```json

     

       320
       320
       +
       {

     

       321
       321
       +
         "error": "CommunityNotFound",

     

       322
       322
       +
         "message": "Community not found"

     

       323
       323
       +
       }

     

       324
       324
       +
       ```

     

       325
       325
       +
       

     

       326
       326
       +
       ---

     

       327
       327
       +
       

     

       328
       328
       +
       ## Code Structure

     

       329
       329
       +
       

     

       330
       330
       +
       ### Package Organization

     

       331
       331
       +
       

     

       332
       332
       +
       ```

     

       333
       333
       +
       internal/

     

       334
       334
       +
       ├── core/feeds/                    # Business logic

     

       335
       335
       +
       │   ├── interfaces.go              # Service & Repository contracts

     

       336
       336
       +
       │   ├── service.go                 # Validation, community resolution

     

       337
       337
       +
       │   ├── types.go                   # Request/Response models

     

       338
       338
       +
       │   └── errors.go                  # Error types

     

       339
       339
       +
       ├── db/postgres/

     

       340
       340
       +
       │   └── feed_repo.go               # SQL queries, sorting algorithms

     

       341
       341
       +
       └── api/

     

       342
       342
       +
           ├── handlers/feed/

     

       343
       343
       +
           │   ├── get_community.go       # HTTP handler

     

       344
       344
       +
           │   └── errors.go              # Error mapping

     

       345
       345
       +
           └── routes/

     

       346
       346
       +
               └── feed.go                # Route registration

     

       347
       347
       +
       ```

     

       348
       348
       +
       

     

       349
       349
       +
       ### Service Layer Flow

     

       350
       350
       +
       

     

       351
       351
       +
       ```

     

       352
       352
       +
       1. HandleGetCommunity (HTTP handler)

     

       353
       353
       +
          ↓ Parse query params

     

       354
       354
       +
       

     

       355
       355
       +
       2. FeedService.GetCommunityFeed

     

       356
       356
       +
          ↓ Validate request (sort, limit, timeframe)

     

       357
       357
       +
          ↓ Resolve community identifier (handle → DID)

     

       358
       358
       +
       

     

       359
       359
       +
       3. FeedRepository.GetCommunityFeed

     

       360
       360
       +
          ↓ Build SQL query (ORDER BY based on sort)

     

       361
       361
       +
          ↓ Apply timeframe filter (for top)

     

       362
       362
       +
          ↓ Apply cursor pagination

     

       363
       363
       +
          ↓ Execute single query with JOINs

     

       364
       364
       +
          ↓ Scan rows into PostView structs

     

       365
       365
       +
          ↓ Build pagination cursor from last post

     

       366
       366
       +
       

     

       367
       367
       +
       4. Return FeedResponse

     

       368
       368
       +
          ↓ Array of FeedViewPost

     

       369
       369
       +
          ↓ Cursor for next page (if more results)

     

       370
       370
       +
       ```

     

       371
       371
       +
       

     

       372
       372
       +
       ---

     

       373
       373
       +
       

     

       374
       374
       +
       ## Testing Strategy

     

       375
       375
       +
       

     

       376
       376
       +
       ### Unit Tests (Future)

     

       377
       377
       +
       

     

       378
       378
       +
       - [ ] Feed service validation logic

     

       379
       379
       +
       - [ ] Cursor encoding/decoding

     

       380
       380
       +
       - [ ] Sort clause generation

     

       381
       381
       +
       - [ ] Timeframe filtering

     

       382
       382
       +
       

     

       383
       383
       +
       ### Integration Tests (Required)

     

       384
       384
       +
       

     

       385
       385
       +
       - [x] Test hot/top/new sorting with real posts

     

       386
       386
       +
       - [x] Test pagination (3 pages, verify no duplicates)

     

       387
       387
       +
       - [x] Test community resolution (handle → DID)

     

       388
       388
       +
       - [x] Test error cases (invalid community, bad cursor)

     

       389
       389
       +
       - [x] Test empty feed (new community)

     

       390
       390
       +
       - [x] Test limit validation (zero, negative, over max)

     

       391
       391
       +
       

     

       392
       392
       +
       ### Integration Test Results

     

       393
       393
       +
       

     

       394
       394
       +
       **All tests passing ✅**

     

       395
       395
       +
       

     

       396
       396
       +
       ```bash

     

       397
       397
       +
       PASS: TestGetCommunityFeed_Hot (0.02s)

     

       398
       398
       +
       PASS: TestGetCommunityFeed_Top_WithTimeframe (0.02s)

     

       399
       399
       +
         PASS: Top_posts_from_last_day (0.00s)

     

       400
       400
       +
         PASS: Top_posts_from_all_time (0.00s)

     

       401
       401
       +
       PASS: TestGetCommunityFeed_New (0.02s)

     

       402
       402
       +
       PASS: TestGetCommunityFeed_Pagination (0.05s)

     

       403
       403
       +
       PASS: TestGetCommunityFeed_InvalidCommunity (0.01s)

     

       404
       404
       +
       PASS: TestGetCommunityFeed_InvalidCursor (0.01s)

     

       405
       405
       +
         PASS: Invalid_base64 (0.00s)

     

       406
       406
       +
         PASS: Malicious_SQL (0.00s)

     

       407
       407
       +
         PASS: Invalid_timestamp (0.00s)

     

       408
       408
       +
         PASS: Invalid_URI_format (0.00s)

     

       409
       409
       +
       PASS: TestGetCommunityFeed_EmptyFeed (0.01s)

     

       410
       410
       +
       PASS: TestGetCommunityFeed_LimitValidation (0.01s)

     

       411
       411
       +
         PASS: Reject_limit_over_50 (0.00s)

     

       412
       412
       +
         PASS: Handle_zero_limit_with_default (0.00s)

     

       413
       413
       +
       

     

       414
       414
       +
       Total: 8 test cases, 12 sub-tests

     

       415
       415
       +
       ```

     

       416
       416
       +
       

     

       417
       417
       +
       **Test Coverage:**

     

       418
       418
       +
       - ✅ Hot algorithm (score decay over time)

     

       419
       419
       +
       - ✅ Top algorithm (timeframe filtering: day, all-time)

     

       420
       420
       +
       - ✅ New algorithm (chronological ordering)

     

       421
       421
       +
       - ✅ Pagination (3 pages, no duplicates, cursor stability)

     

       422
       422
       +
       - ✅ Error handling (invalid community, malformed cursors)

     

       423
       423
       +
       - ✅ Security (cursor injection, SQL injection attempts)

     

       424
       424
       +
       - ✅ Edge cases (empty feeds, zero/negative limits)

     

       425
       425
       +
       

     

       426
       426
       +
       **Location:** `tests/integration/feed_test.go`

     

       427
       427
       +
       

     

       428
       428
       +
       ---

     

       429
       429
       +
       

     

       430
       430
       +
       ## Performance Considerations

     

       431
       431
       +
       

     

       432
       432
       +
       ### Database Indexes

     

       433
       433
       +
       

     

       434
       434
       +
       Required indexes for optimal performance:

     

       435
       435
       +
       

     

       436
       436
       +
       ```sql

     

       437
       437
       +
       -- Hot sorting (uses score and created_at)

     

       438
       438
       +
       CREATE INDEX idx_posts_community_hot

     

       439
       439
       +
       ON posts(community_did, score DESC, created_at DESC)

     

       440
       440
       +
       WHERE deleted_at IS NULL;

     

       441
       441
       +
       

     

       442
       442
       +
       -- Top sorting (score only)

     

       443
       443
       +
       CREATE INDEX idx_posts_community_top

     

       444
       444
       +
       ON posts(community_did, score DESC, created_at DESC)

     

       445
       445
       +
       WHERE deleted_at IS NULL;

     

       446
       446
       +
       

     

       447
       447
       +
       -- New sorting (chronological)

     

       448
       448
       +
       CREATE INDEX idx_posts_community_new

     

       449
       449
       +
       ON posts(community_did, created_at DESC)

     

       450
       450
       +
       WHERE deleted_at IS NULL;

     

       451
       451
       +
       ```

     

       452
       452
       +
       

     

       453
       453
       +
       ### Query Performance

     

       454
       454
       +
       

     

       455
       455
       +
       - **Single query** - No N+1 problems

     

       456
       456
       +
       - **JOINs** - users and communities (always small cardinality)

     

       457
       457
       +
       - **Pagination** - Keyset, no OFFSET scans

     

       458
       458
       +
       - **Filtering** - `deleted_at IS NULL` uses partial index

     

       459
       459
       +
       

     

       460
       460
       +
       **Expected performance:**

     

       461
       461
       +
       - 25 posts with full metadata: **< 50ms**

     

       462
       462
       +
       - 1000+ posts in community: **Still < 50ms** (keyset pagination)

     

       463
       463
       +
       

     

       464
       464
       +
       ---

     

       465
       465
       +
       

     

       466
       466
       +
       ## Future Enhancements (Beta)

     

       467
       467
       +
       

     

       468
       468
       +
       ### 1. Feed Generators (Skeleton Pattern)

     

       469
       469
       +
       

     

       470
       470
       +
       Allow users to create custom algorithms:

     

       471
       471
       +
       

     

       472
       472
       +
       ```

     

       473
       473
       +
       GET /xrpc/social.coves.feed.getSkeleton?feed=at://alice/feed/best-memes

     

       474
       474
       +
         → Returns: [uri1, uri2, uri3, ...]

     

       475
       475
       +
       

     

       476
       476
       +
       GET /xrpc/social.coves.post.get?uris=[...]

     

       477
       477
       +
         → Returns: [full posts]

     

       478
       478
       +
       ```

     

       479
       479
       +
       

     

       480
       480
       +
       **Use cases:**

     

       481
       481
       +
       - User-created feeds ("Best of the week")

     

       482
       482
       +
       - Algorithmic feeds ("Rising posts", "Controversial")

     

       483
       483
       +
       - Filtered feeds ("Gaming news only", "No memes")

     

       484
       484
       +
       

     

       485
       485
       +
       ### 2. Viewer State (Feed Generator Phase)

     

       486
       486
       +
       

     

       487
       487
       +
       **Status:** Deferred - Not needed for Alpha's basic community sorting

     

       488
       488
       +
       

     

       489
       489
       +
       Include viewer's relationship with posts when implementing feed generators:

     

       490
       490
       +
       

     

       491
       491
       +
       ```json

     

       492
       492
       +
       "viewer": {

     

       493
       493
       +
         "vote": "up",

     

       494
       494
       +
         "voteUri": "at://...",

     

       495
       495
       +
         "saved": true,

     

       496
       496
       +
         "savedUri": "at://...",

     

       497
       497
       +
         "tags": ["read-later", "favorite"]

     

       498
       498
       +
       }

     

       499
       499
       +
       ```

     

       500
       500
       +
       

     

       501
       501
       +
       **Implementation Plan:**

     

       502
       502
       +
       - Wire up OptionalAuth middleware to feed routes

     

       503
       503
       +
       - Extract viewer DID from auth context

     

       504
       504
       +
       - Query viewer state tables (votes, saves, blocks)

     

       505
       505
       +
       - Include in PostView response

     

       506
       506
       +
       

     

       507
       507
       +
       **Requires:**

     

       508
       508
       +
       - Votes table (user_did, post_uri, vote_type)

     

       509
       509
       +
       - Saved posts table

     

       510
       510
       +
       - Blocks table

     

       511
       511
       +
       - Tags table

     

       512
       512
       +
       

     

       513
       513
       +
       **Why deferred:** Alpha only needs raw community sorting (hot/new/top). Viewer-specific features like upvote highlighting and saved posts will be implemented when we build the feed generator skeleton.

     

       514
       514
       +
       

     

       515
       515
       +
       ### 3. Post Type Filtering (Feed Generator Phase)

     

       516
       516
       +
       

     

       517
       517
       +
       **Status:** Deferred - Not needed for Alpha's basic community sorting

     

       518
       518
       +
       

     

       519
       519
       +
       Filter by embed type when implementing feed generators:

     

       520
       520
       +
       

     

       521
       521
       +
       ```

     

       522
       522
       +
       GET ...?postTypes=image,video

     

       523
       523
       +
         → Only image and video posts

     

       524
       524
       +
       ```

     

       525
       525
       +
       

     

       526
       526
       +
       **Implementation Plan:**

     

       527
       527
       +
       - Check `embed->>'$type'` in SQL WHERE clause

     

       528
       528
       +
       - Map to friendly types (text, image, video, link, quote)

     

       529
       529
       +
       - Support both single (`postType`) and array (`postTypes`) filtering

     

       530
       530
       +
       

     

       531
       531
       +
       **Why deferred:** Alpha displays all posts without filtering. Post type filtering will be useful in feed generators for specialized feeds (e.g., "images only").

     

       532
       532
       +
       

     

       533
       533
       +
       ### 4. Pinned Posts (Feed Generator Phase)

     

       534
       534
       +
       

     

       535
       535
       +
       Moderators pin important posts to top:

     

       536
       536
       +
       

     

       537
       537
       +
       ```json

     

       538
       538
       +
       "reason": {

     

       539
       539
       +
         "$type": "social.coves.feed.defs#reasonPin",

     

       540
       540
       +
         "community": {"did": "...", "name": "gaming"}

     

       541
       541
       +
       }

     

       542
       542
       +
       ```

     

       543
       543
       +
       

     

       544
       544
       +
       ### 5. Reply Context

     

       545
       545
       +
       

     

       546
       546
       +
       Show post's position in thread:

     

       547
       547
       +
       

     

       548
       548
       +
       ```json

     

       549
       549
       +
       "reply": {

     

       550
       550
       +
         "root": {"uri": "at://...", "cid": "..."},

     

       551
       551
       +
         "parent": {"uri": "at://...", "cid": "..."}

     

       552
       552
       +
       }

     

       553
       553
       +
       ```

     

       554
       554
       +
       

     

       555
       555
       +
       ---

     

       556
       556
       +
       

     

       557
       557
       +
       ## Lexicon Updates

     

       558
       558
       +
       

     

       559
       559
       +
       ### Updated: `social.coves.post.get`

     

       560
       560
       +
       

     

       561
       561
       +
       **Changes:**

     

       562
       562
       +
       1. ✅ Batch URIs: `uri` → `uris[]` (max 25)

     

       563
       563
       +
       2. ✅ Union embed: Matches Bluesky pattern exactly

     

       564
       564
       +
       3. ✅ Error handling: `notFoundPost`, `blockedPost`

     

       565
       565
       +
       

     

       566
       566
       +
       **Before:**

     

       567
       567
       +
       ```json

     

       568
       568
       +
       {

     

       569
       569
       +
         "parameters": {

     

       570
       570
       +
           "uri": "string"

     

       571
       571
       +
         },

     

       572
       572
       +
         "output": {

     

       573
       573
       +
           "post": "#postView"

     

       574
       574
       +
         }

     

       575
       575
       +
       }

     

       576
       576
       +
       ```

     

       577
       577
       +
       

     

       578
       578
       +
       **After:**

     

       579
       579
       +
       ```json

     

       580
       580
       +
       {

     

       581
       581
       +
         "parameters": {

     

       582
       582
       +
           "uris": ["string"]  // Array, max 25

     

       583
       583
       +
         },

     

       584
       584
       +
         "output": {

     

       585
       585
       +
           "posts": [

     

       586
       586
       +
             "union": ["#postView", "#notFoundPost", "#blockedPost"]

     

       587
       587
       +
           ]

     

       588
       588
       +
         }

     

       589
       589
       +
       }

     

       590
       590
       +
       ```

     

       591
       591
       +
       

     

       592
       592
       +
       **Why?**

     

       593
       593
       +
       - Batch fetching for feed hydration (future)

     

       594
       594
       +
       - Handle missing/blocked posts gracefully

     

       595
       595
       +
       - Bluesky compatibility

     

       596
       596
       +
       

     

       597
       597
       +
       ### Using: `social.coves.feed.getCommunity`

     

       598
       598
       +
       

     

       599
       599
       +
       Already exists, matches our implementation:

     

       600
       600
       +
       

     

       601
       601
       +
       ```json

     

       602
       602
       +
       {

     

       603
       603
       +
         "id": "social.coves.feed.getCommunity",

     

       604
       604
       +
         "parameters": {

     

       605
       605
       +
           "community": "at-identifier",

     

       606
       606
       +
           "sort": "hot|top|new",

     

       607
       607
       +
           "timeframe": "hour|day|week|month|year|all",

     

       608
       608
       +
           "limit": 1-50,

     

       609
       609
       +
           "cursor": "string"

     

       610
       610
       +
         },

     

       611
       611
       +
         "output": {

     

       612
       612
       +
           "feed": ["#feedViewPost"],

     

       613
       613
       +
           "cursor": "string"

     

       614
       614
       +
         }

     

       615
       615
       +
       }

     

       616
       616
       +
       ```

     

       617
       617
       +
       

     

       618
       618
       +
       ---

     

       619
       619
       +
       

     

       620
       620
       +
       ## Migration Path

     

       621
       621
       +
       

     

       622
       622
       +
       ### Alpha → Beta: Adding Feed Generators

     

       623
       623
       +
       

     

       624
       624
       +
       **Good news:** No breaking changes needed!

     

       625
       625
       +
       

     

       626
       626
       +
       **Approach:**

     

       627
       627
       +
       1. Keep `getCommunity` for standard sorting

     

       628
       628
       +
       2. Add `getFeedSkeleton` for custom algorithms

     

       629
       629
       +
       3. Add `post.get` batch support (already lexicon-ready)

     

       630
       630
       +
       4. Users choose: fast hydrated OR flexible skeleton

     

       631
       631
       +
       

     

       632
       632
       +
       **Both coexist:**

     

       633
       633
       +
       ```

     

       634
       634
       +
       // Standard community browsing (most users)

     

       635
       635
       +
       GET /xrpc/social.coves.feed.getCommunity?community=gaming&sort=hot

     

       636
       636
       +
         → One request, hydrated posts

     

       637
       637
       +
       

     

       638
       638
       +
       // Custom feed (power users)

     

       639
       639
       +
       GET /xrpc/social.coves.feed.getSkeleton?feed=at://alice/feed/best-memes

     

       640
       640
       +
         → Returns URIs

     

       641
       641
       +
       GET /xrpc/social.coves.post.get?uris=[...]

     

       642
       642
       +
         → Hydrates posts

     

       643
       643
       +
       ```

     

       644
       644
       +
       

     

       645
       645
       +
       ---

     

       646
       646
       +
       

     

       647
       647
       +
       ## Success Metrics

     

       648
       648
       +
       

     

       649
       649
       +
       ### Alpha Launch

     

       650
       650
       +
       

     

       651
       651
       +
       - [ ] Users can browse communities

     

       652
       652
       +
       - [ ] Hot/top/new sorting works correctly

     

       653
       653
       +
       - [ ] Pagination stable across 3+ pages

     

       654
       654
       +
       - [ ] Performance < 100ms for 25 posts

     

       655
       655
       +
       - [ ] Handles 1000+ posts per community

     

       656
       656
       +
       

     

       657
       657
       +
       ### Future KPIs

     

       658
       658
       +
       

     

       659
       659
       +
       - Feed load time (target: < 50ms)

     

       660
       660
       +
       - Cache hit rate (future: Redis cache)

     

       661
       661
       +
       - Custom feed adoption (Beta)

     

       662
       662
       +
       - User engagement (time in feed, clicks)

     

       663
       663
       +
       

     

       664
       664
       +
       ---

     

       665
       665
       +
       

     

       666
       666
       +
       ## Dependencies

     

       667
       667
       +
       

     

       668
       668
       +
       ### Required Services

     

       669
       669
       +
       

     

       670
       670
       +
       - ✅ PostgreSQL (AppView database)

     

       671
       671
       +
       - ✅ Posts indexed via Jetstream

     

       672
       672
       +
       - ✅ Users indexed via Jetstream

     

       673
       673
       +
       - ✅ Communities indexed via Jetstream

     

       674
       674
       +
       

     

       675
       675
       +
       ### Optional (Future)

     

       676
       676
       +
       

     

       677
       677
       +
       - Redis (feed caching)

     

       678
       678
       +
       - Feed generator services (custom algorithms)

     

       679
       679
       +
       

     

       680
       680
       +
       ---

     

       681
       681
       +
       

     

       682
       682
       +
       ## Security Considerations

     

       683
       683
       +
       

     

       684
       684
       +
       ### Input Validation

     

       685
       685
       +
       

     

       686
       686
       +
       - ✅ Community identifier format (DID or handle)

     

       687
       687
       +
       - ✅ Sort parameter (enum: hot/top/new)

     

       688
       688
       +
       - ✅ Limit (1-50, default 15, explicit rejection over 50)

     

       689
       689
       +
       - ✅ Cursor (base64 decoding, format validation)

     

       690
       690
       +
       - ✅ **Cursor injection prevention:**

     

       691
       691
       +
         - Timestamp format validation (RFC3339Nano)

     

       692
       692
       +
         - URI format validation (must start with `at://`)

     

       693
       693
       +
         - Score numeric validation

     

       694
       694
       +
         - Part count validation (2 for new, 3 for top/hot)

     

       695
       695
       +
       

     

       696
       696
       +
       ### SQL Injection Prevention

     

       697
       697
       +
       

     

       698
       698
       +
       - ✅ All queries use parameterized statements

     

       699
       699
       +
       - ✅ **Dynamic ORDER BY uses whitelist map** (defense-in-depth)

     

       700
       700
       +
         ```go

     

       701
       701
       +
         var sortClauses = map[string]string{

     

       702
       702
       +
             "hot": `(p.score / POWER(...)) DESC, p.created_at DESC`,

     

       703
       703
       +
             "top": `p.score DESC, p.created_at DESC`,

     

       704
       704
       +
             "new": `p.created_at DESC, p.uri DESC`,

     

       705
       705
       +
         }

     

       706
       706
       +
         ```

     

       707
       707
       +
       - ✅ **Timeframe filter uses hardcoded switch** (no user input in INTERVAL)

     

       708
       708
       +
       - ✅ No string concatenation in SQL

     

       709
       709
       +
       

     

       710
       710
       +
       ### DoS Prevention

     

       711
       711
       +
       

     

       712
       712
       +
       - ✅ **Zero-limit pagination fix:** Guards against `limit=0` causing panic

     

       713
       713
       +
         - Service layer: Sets default limit if ≤ 0

     

       714
       714
       +
         - Repository layer: Additional check before array slicing

     

       715
       715
       +
       - ✅ Limit validation: Explicit error for limits over 50

     

       716
       716
       +
       - ✅ Cursor validation: Rejects malformed cursors early

     

       717
       717
       +
       

     

       718
       718
       +
       ### Rate Limiting

     

       719
       719
       +
       

     

       720
       720
       +
       - ✅ Global rate limiter (100 req/min per IP)

     

       721
       721
       +
       - Future: Per-endpoint limits

     

       722
       722
       +
       

     

       723
       723
       +
       ### Privacy

     

       724
       724
       +
       

     

       725
       725
       +
       - Alpha: All feeds public

     

       726
       726
       +
       - Beta: Respect community visibility (private/unlisted)

     

       727
       727
       +
       - Beta: Block lists (hide posts from blocked users)

     

       728
       728
       +
       

     

       729
       729
       +
       ### Security Audit (PR Review)

     

       730
       730
       +
       

     

       731
       731
       +
       All critical and important issues from PR review have been addressed:

     

       732
       732
       +
       

     

       733
       733
       +
       **P0 - Critical (Fixed):**

     

       734
       734
       +
       1. ✅ Zero-limit DoS vulnerability

     

       735
       735
       +
       2. ✅ Cursor injection attacks

     

       736
       736
       +
       3. ✅ Validation by-value bug

     

       737
       737
       +
       

     

       738
       738
       +
       **Important (Fixed):**

     

       739
       739
       +
       4. ✅ ORDER BY SQL injection hardening

     

       740
       740
       +
       5. ✅ Silent error swallowing in JSON encoding

     

       741
       741
       +
       6. ✅ Limit validation (reject vs silent cap)

     

       742
       742
       +
       

     

       743
       743
       +
       **False Positives (Rejected):**

     

       744
       744
       +
       - ❌ Time filter SQL injection (safe by design)

     

       745
       745
       +
       - ❌ Nil pointer dereference (impossible condition)

     

       746
       746
       +
       

     

       747
       747
       +
       ---

     

       748
       748
       +
       

     

       749
       749
       +
       ## Conclusion

     

       750
       750
       +
       

     

       751
       751
       +
       ### What We Shipped

     

       752
       752
       +
       

     

       753
       753
       +
       ✅ **Complete community feed system (Alpha scope)**

     

       754
       754
       +
       - Hot/top/new sorting algorithms

     

       755
       755
       +
       - Cursor-based pagination

     

       756
       756
       +
       - Single-query performance

     

       757
       757
       +
       - Full post hydration (author, community, stats)

     

       758
       758
       +
       - Error handling

     

       759
       759
       +
       - Production-ready code

     

       760
       760
       +
       - **No viewer state** (YAGNI - deferred to feed generator phase)

     

       761
       761
       +
       

     

       762
       762
       +
       ### Why It Matters

     

       763
       763
       +
       

     

       764
       764
       +
       **Before:** Users could create posts but not see them

     

       765
       765
       +
       **After:** Full community browsing experience

     

       766
       766
       +
       

     

       767
       767
       +
       **Impact:**

     

       768
       768
       +
       - 🎯 Core forum functionality

     

       769
       769
       +
       - 🚀 Fast, scalable implementation

     

       770
       770
       +
       - 🔮 Future-proof architecture

     

       771
       771
       +
       - 🤝 Bluesky-compatible patterns

     

       772
       772
       +
       

     

       773
       773
       +
       ### Next Steps

     

       774
       774
       +
       

     

       775
       775
       +
       1. ~~**Write E2E tests**~~ ✅ Complete (8 test cases, all passing)

     

       776
       776
       +
       2. **Performance testing** (1000+ posts under load)

     

       777
       777
       +
       3. **Add to docs site** (API reference)

     

       778
       778
       +
       4. **Monitor in production** (query performance, cursor stability)

     

       779
       779
       +
       5. **PR #2:** Batch `getPosts` for feed generators (Beta)

     

       780
       780
       +
       

     

       781
       781
       +
       ---

     

       782
       782
       +
       

     

       783
       783
       +
       ## References

     

       784
       784
       +
       

     

       785
       785
       +
       - [PRD: Posts](../PRD_POSTS.md)

     

       786
       786
       +
       - [Lexicon: getCommunity](../internal/atproto/lexicon/social/coves/feed/getCommunity.json)

     

       787
       787
       +
       - [Lexicon: post.get](../internal/atproto/lexicon/social/coves/post/get.json)

     

       788
       788
       +
       - [Bluesky Feed Pattern](https://github.com/bluesky-social/atproto/discussions/4245)

     

       789
       789
       +
       - [Reddit Hot Algorithm](https://medium.com/hacking-and-gonzo/how-reddit-ranking-algorithms-work-ef111e33d0d9)

     

       790
       790
       +
       

     

       791
       791
       +
       ---

     

       792
       792
       +
       

     

       793
       793
       +
       **Document Version:** 1.0

     

       794
       794
       +
       **Last Updated:** October 20, 2025

     

       795
       795
       +
       **Status:** ✅ Implemented, Ready for Testing

+10 -10

docs/PRD_POSTS.md

···

       17
       17
        
       - **at-identifier Support:** All 4 formats (DIDs, canonical, @-prefixed, scoped handles)

     

       18
       18
        
       

     

       19
       19
        
       ### ⚠️ DEFERRED TO BETA

     

       20
       20
       -
       - Content rules validation (text-only, image-only communities)

     

       20
       20
       +
       - Content rules validation (text-only, image-only communities) - Governence

     

       21
       21
        
       - Post read operations (get, list)

     

       22
       22
        
       - Post update/edit operations

     

       23
       23
        
       - Post deletion

     
···

       379
       379
        
       - [x] **Community Integration:** ✅ Posts correctly reference communities via at-identifiers

     

       380
       380
        
       - [x] **at-identifier Support:** ✅ All 4 formats supported (DIDs, canonical, @-prefixed, scoped)

     

       381
       381
        
       - [ ] **Content Rules Validation:** ⚠️ DEFERRED TO BETA - Posts validated against community content rules

     

       382
       382
       -
       - [ ] **Vote System:** ⚠️ DEFERRED TO BETA - Upvote/downvote with community-level controls

     

       382
       382
       +
       - [ ] **Vote System:** - Upvote/downvote with community-level controls

     

       383
       383
        
       - [ ] **Moderator Permissions:** ⚠️ DEFERRED TO BETA - Community moderators can delete posts

     

       384
       384
        
       

     

       385
       385
        
       ### Testing Requirements

     
···

       392
       392
        
       - [x] Handler security: All 4 at-identifier formats accepted ✅

     

       393
       393
        
       - [x] Consumer security: Rejects posts from wrong repository ✅

     

       394
       394
        
       - [x] Consumer security: Verifies community and author exist ✅

     

       395
       395
       -
       - [ ] **Content rules validation:** Text-only community rejects image posts ⚠️ DEFERRED

     

       396
       396
       -
       - [ ] **Content rules validation:** Image community rejects posts without images ⚠️ DEFERRED

     

       397
       397
       -
       - [ ] **Content rules validation:** Post with too-short text rejected ⚠️ DEFERRED

     

       398
       398
       -
       - [ ] **Content rules validation:** Federated post rejected if `allowFederated: false` ⚠️ DEFERRED

     

       395
       395
       +
       - [ ] **Content rules validation:** Text-only community rejects image posts ⚠️ DEFERRED - Governence

     

       396
       396
       +
       - [ ] **Content rules validation:** Image community rejects posts without images ⚠️ DEFERRED - Governence

     

       397
       397
       +
       - [ ] **Content rules validation:** Post with too-short text rejected ⚠️ DEFERRED - Governence

     

       398
       398
       +
       - [ ] **Content rules validation:** Federated post rejected if `allowFederated: false` ⚠️ DEFERRED - Governence

     

       399
       399
        
       - [ ] Update post within 24 hours → Edit reflected ⚠️ DEFERRED

     

       400
       400
        
       - [ ] Delete post as author → Hidden from queries ⚠️ DEFERRED

     

       401
       401
        
       - [ ] Delete post as moderator → Hidden from queries ⚠️ DEFERRED

     

       402
       402
       -
       - [ ] Upvote post → Count increments ⚠️ DEFERRED

     

       403
       403
       -
       - [ ] Downvote post → Count increments (if enabled) ⚠️ DEFERRED

     

       404
       404
       -
       - [ ] Toggle vote → Counts update correctly ⚠️ DEFERRED

     

       405
       405
       -
       - [ ] Community with downvotes disabled → Downvote returns error ⚠️ DEFERRED

     

       402
       402
       +
       - [ ] Upvote post → Count increments

     

       403
       403
       +
       - [ ] Downvote post → Count increments (if enabled)

     

       404
       404
       +
       - [ ] Toggle vote → Counts update correctly ⚠️ DEFERRED - Governence

     

       405
       405
       +
       - [ ] Community with downvotes disabled → Downvote returns error ⚠️ DEFERRED - Governence

     

       406
       406
        
       

     

       407
       407
        
       ---

     

       408
       408

+1412

docs/aggregators/PRD_AGGREGATORS.md

···

       1
       1
       +
       # Aggregators PRD: Automated Content Posting System

     

       2
       2
       +
       

     

       3
       3
       +
       **Status:** Planning / Design Phase

     

       4
       4
       +
       **Owner:** Platform Team

     

       5
       5
       +
       **Last Updated:** 2025-10-19

     

       6
       6
       +
       

     

       7
       7
       +
       ## Overview

     

       8
       8
       +
       

     

       9
       9
       +
       Coves Aggregators are autonomous services that automatically post content to communities. Each aggregator is identified by its own DID and operates as a specialized actor within the atProto ecosystem. This system enables communities to have automated content feeds (RSS, sports results, TV/movie discussion threads, Bluesky mirrors, etc.) while maintaining full community control over which aggregators can post and what content they can create.

     

       10
       10
       +
       

     

       11
       11
       +
       **Key Differentiator:** Unlike other platforms where users manually aggregate content, Coves communities can enable automated aggregators to handle routine posting tasks, creating a more dynamic and up-to-date community experience.

     

       12
       12
       +
       

     

       13
       13
       +
       ## Architecture Principles

     

       14
       14
       +
       

     

       15
       15
       +
       ### ✅ atProto-Compliant Design

     

       16
       16
       +
       

     

       17
       17
       +
       Aggregators follow established atProto patterns for autonomous services:

     

       18
       18
       +
       

     

       19
       19
       +
       **Pattern:** Feed Generators + Labelers Model

     

       20
       20
       +
       - Each aggregator has its own DID (like feed generators)

     

       21
       21
       +
       - Declaration record published in aggregator's repo (like `app.bsky.feed.generator`)

     

       22
       22
       +
       - DID document advertises service endpoint

     

       23
       23
       +
       - Service makes authenticated XRPC calls

     

       24
       24
       +
       - Communities explicitly authorize aggregators (like subscribing to labelers)

     

       25
       25
       +
       

     

       26
       26
       +
       **Key Design Decisions:**

     

       27
       27
       +
       

     

       28
       28
       +
       1. **Aggregators are Actors, Not a Separate System**

     

       29
       29
       +
          - Aggregators authenticate as themselves (their DID)

     

       30
       30
       +
          - Use existing `social.coves.post.create` endpoint

     

       31
       31
       +
          - Post record's `author` field = aggregator DID (server-populated)

     

       32
       32
       +
          - No separate posting API needed

     

       33
       33
       +
       

     

       34
       34
       +
       2. **Community Authorization Model**

     

       35
       35
       +
          - Communities create `social.coves.aggregator.authorization` records

     

       36
       36
       +
          - These records grant specific aggregators permission to post

     

       37
       37
       +
          - Authorizations include configuration (which RSS feeds, which users to mirror, etc.)

     

       38
       38
       +
          - Can be enabled/disabled at any time

     

       39
       39
       +
       

     

       40
       40
       +
       3. **Hybrid Hosting**

     

       41
       41
       +
          - Coves can host official aggregators (RSS, sports, media)

     

       42
       42
       +
          - Third parties can build and host their own aggregators

     

       43
       43
       +
          - SDK provided for easy aggregator development

     

       44
       44
       +
          - All aggregators use same authorization system

     

       45
       45
       +
       

     

       46
       46
       +
       ---

     

       47
       47
       +
       

     

       48
       48
       +
       ## Architecture Overview

     

       49
       49
       +
       

     

       50
       50
       +
       ```

     

       51
       51
       +
       ┌────────────────────────────────────────────────────────────┐

     

       52
       52
       +
       │  Aggregator Service (External)                             │

     

       53
       53
       +
       │  DID: did:web:rss-bot.coves.social                         │

     

       54
       54
       +
       │                                                             │

     

       55
       55
       +
       │  - Watches external data sources (RSS, APIs, etc.)         │

     

       56
       56
       +
       │  - Processes content (LLM deduplication, formatting)       │

     

       57
       57
       +
       │  - Queries which communities have authorized it            │

     

       58
       58
       +
       │  - Creates posts via social.coves.post.create              │

     

       59
       59
       +
       │  - Responds to config queries via XRPC                     │

     

       60
       60
       +
       └────────────────────────────────────────────────────────────┘

     

       61
       61
       +
                                   │

     

       62
       62
       +
                                   │ 1. Authenticate as aggregator DID (JWT)

     

       63
       63
       +
                                   │ 2. Call social.coves.post.create

     

       64
       64
       +
                                   │    {

     

       65
       65
       +
                                   │      community: "did:plc:gaming123",

     

       66
       66
       +
                                   │      title: "...",

     

       67
       67
       +
                                   │      content: "...",

     

       68
       68
       +
                                   │      federatedFrom: { platform: "rss", ... }

     

       69
       69
       +
                                   │    }

     

       70
       70
       +
                                   ▼

     

       71
       71
       +
       ┌────────────────────────────────────────────────────────────┐

     

       72
       72
       +
       │  Coves AppView (social.coves.post.create Handler)          │

     

       73
       73
       +
       │                                                             │

     

       74
       74
       +
       │  1. Extract DID from JWT (aggregator's DID)                │

     

       75
       75
       +
       │  2. Check if DID is registered aggregator                  │

     

       76
       76
       +
       │  3. Validate authorization record exists & enabled         │

     

       77
       77
       +
       │  4. Apply aggregator-specific rate limits                  │

     

       78
       78
       +
       │  5. Validate content against community rules               │

     

       79
       79
       +
       │  6. Create post with author = aggregator DID               │

     

       80
       80
       +
       └────────────────────────────────────────────────────────────┘

     

       81
       81
       +
                                   │

     

       82
       82
       +
                                   │ Post record created:

     

       83
       83
       +
                                   │ {

     

       84
       84
       +
                                   │   $type: "social.coves.post.record",

     

       85
       85
       +
                                   │   author: "did:web:rss-bot.coves.social",

     

       86
       86
       +
                                   │   community: "did:plc:gaming123",

     

       87
       87
       +
                                   │   title: "Tech News Roundup",

     

       88
       88
       +
                                   │   content: "...",

     

       89
       89
       +
                                   │   federatedFrom: {

     

       90
       90
       +
                                   │     platform: "rss",

     

       91
       91
       +
                                   │     uri: "https://techcrunch.com/..."

     

       92
       92
       +
                                   │   }

     

       93
       93
       +
                                   │ }

     

       94
       94
       +
                                   ▼

     

       95
       95
       +
       ┌────────────────────────────────────────────────────────────┐

     

       96
       96
       +
       │  Jetstream → AppView Indexing                              │

     

       97
       97
       +
       │  - Post indexed with aggregator attribution                │

     

       98
       98
       +
       │  - UI shows: "🤖 Posted by RSS Aggregator"                 │

     

       99
       99
       +
       │  - Community feed includes automated posts                 │

     

       100
       100
       +
       └────────────────────────────────────────────────────────────┘

     

       101
       101
       +
       ```

     

       102
       102
       +
       

     

       103
       103
       +
       ---

     

       104
       104
       +
       

     

       105
       105
       +
       ## Use Cases

     

       106
       106
       +
       

     

       107
       107
       +
       ### 1. RSS News Aggregator

     

       108
       108
       +
       **Problem:** Multiple users posting the same breaking news from different sources

     

       109
       109
       +
       **Solution:** RSS aggregator with LLM deduplication

     

       110
       110
       +
       - Watches configured RSS feeds

     

       111
       111
       +
       - Uses LLM to identify duplicate stories from different outlets

     

       112
       112
       +
       - Creates single "megathread" with all sources linked

     

       113
       113
       +
       - Posts unbiased summary of event

     

       114
       114
       +
       - Automatically tags with relevant topics

     

       115
       115
       +
       

     

       116
       116
       +
       **Community Config:**

     

       117
       117
       +
       ```json

     

       118
       118
       +
       {

     

       119
       119
       +
         "aggregatorDid": "did:web:rss-bot.coves.social",

     

       120
       120
       +
         "enabled": true,

     

       121
       121
       +
         "config": {

     

       122
       122
       +
           "feeds": [

     

       123
       123
       +
             "https://techcrunch.com/feed",

     

       124
       124
       +
             "https://arstechnica.com/feed"

     

       125
       125
       +
           ],

     

       126
       126
       +
           "topics": ["technology", "ai"],

     

       127
       127
       +
           "dedupeWindow": "6h",

     

       128
       128
       +
           "minSources": 2

     

       129
       129
       +
         }

     

       130
       130
       +
       }

     

       131
       131
       +
       ```

     

       132
       132
       +
       

     

       133
       133
       +
       ### 2. Bluesky Post Mirror

     

       134
       134
       +
       **Problem:** Want to surface specific Bluesky discussions in community

     

       135
       135
       +
       **Solution:** Bluesky mirror aggregator

     

       136
       136
       +
       - Monitors specific users or hashtags on Bluesky

     

       137
       137
       +
       - Creates posts in community when criteria met

     

       138
       138
       +
       - Preserves `originalAuthor` metadata

     

       139
       139
       +
       - Links back to original Bluesky thread

     

       140
       140
       +
       

     

       141
       141
       +
       **Community Config:**

     

       142
       142
       +
       ```json

     

       143
       143
       +
       {

     

       144
       144
       +
         "aggregatorDid": "did:web:bsky-mirror.coves.social",

     

       145
       145
       +
         "enabled": true,

     

       146
       146
       +
         "config": {

     

       147
       147
       +
           "mirrorUsers": [

     

       148
       148
       +
             "alice.bsky.social",

     

       149
       149
       +
             "bob.bsky.social"

     

       150
       150
       +
           ],

     

       151
       151
       +
           "hashtags": ["covesalpha"],

     

       152
       152
       +
           "minLikes": 10

     

       153
       153
       +
         }

     

       154
       154
       +
       }

     

       155
       155
       +
       ```

     

       156
       156
       +
       

     

       157
       157
       +
       ### 3. Sports Results Aggregator

     

       158
       158
       +
       **Problem:** Need post-game threads created immediately after games end

     

       159
       159
       +
       **Solution:** Sports aggregator watching game APIs

     

       160
       160
       +
       - Monitors sports APIs for game completions

     

       161
       161
       +
       - Creates post-game thread with final score, stats

     

       162
       162
       +
       - Tags with team names and league

     

       163
       163
       +
       - Posts within minutes of game ending

     

       164
       164
       +
       

     

       165
       165
       +
       **Community Config:**

     

       166
       166
       +
       ```json

     

       167
       167
       +
       {

     

       168
       168
       +
         "aggregatorDid": "did:web:sports-bot.coves.social",

     

       169
       169
       +
         "enabled": true,

     

       170
       170
       +
         "config": {

     

       171
       171
       +
           "league": "NBA",

     

       172
       172
       +
           "teams": ["Lakers", "Warriors"],

     

       173
       173
       +
           "includeStats": true,

     

       174
       174
       +
           "autoPin": true

     

       175
       175
       +
         }

     

       176
       176
       +
       }

     

       177
       177
       +
       ```

     

       178
       178
       +
       

     

       179
       179
       +
       ### 4. TV/Movie Discussion Aggregator

     

       180
       180
       +
       **Problem:** Want episode discussion threads created when shows air

     

       181
       181
       +
       **Solution:** Media aggregator tracking release schedules

     

       182
       182
       +
       - Uses TMDB/IMDB APIs for release dates

     

       183
       183
       +
       - Creates discussion threads when episodes/movies release

     

       184
       184
       +
       - Includes metadata (cast, synopsis, ratings)

     

       185
       185
       +
       - Automatically pins for premiere episodes

     

       186
       186
       +
       

     

       187
       187
       +
       **Community Config:**

     

       188
       188
       +
       ```json

     

       189
       189
       +
       {

     

       190
       190
       +
         "aggregatorDid": "did:web:media-bot.coves.social",

     

       191
       191
       +
         "enabled": true,

     

       192
       192
       +
         "config": {

     

       193
       193
       +
           "shows": [

     

       194
       194
       +
             {"tmdbId": "1234", "name": "Breaking Bad"}

     

       195
       195
       +
           ],

     

       196
       196
       +
           "createOn": "airDate",

     

       197
       197
       +
           "timezone": "America/New_York",

     

       198
       198
       +
           "spoilerProtection": true

     

       199
       199
       +
         }

     

       200
       200
       +
       }

     

       201
       201
       +
       ```

     

       202
       202
       +
       

     

       203
       203
       +
       ---

     

       204
       204
       +
       

     

       205
       205
       +
       ## Lexicon Schemas

     

       206
       206
       +
       

     

       207
       207
       +
       ### 1. Aggregator Service Declaration

     

       208
       208
       +
       

     

       209
       209
       +
       **Collection:** `social.coves.aggregator.service`

     

       210
       210
       +
       **Key:** `literal:self` (one per aggregator account)

     

       211
       211
       +
       **Location:** Aggregator's own repository

     

       212
       212
       +
       

     

       213
       213
       +
       This record declares the existence of an aggregator service and provides metadata for discovery.

     

       214
       214
       +
       

     

       215
       215
       +
       ```json

     

       216
       216
       +
       {

     

       217
       217
       +
         "lexicon": 1,

     

       218
       218
       +
         "id": "social.coves.aggregator.service",

     

       219
       219
       +
         "defs": {

     

       220
       220
       +
           "main": {

     

       221
       221
       +
             "type": "record",

     

       222
       222
       +
             "description": "Declaration of an aggregator service that can post to communities",

     

       223
       223
       +
             "key": "literal:self",

     

       224
       224
       +
             "record": {

     

       225
       225
       +
               "type": "object",

     

       226
       226
       +
               "required": ["did", "displayName", "createdAt", "aggregatorType"],

     

       227
       227
       +
               "properties": {

     

       228
       228
       +
                 "did": {

     

       229
       229
       +
                   "type": "string",

     

       230
       230
       +
                   "format": "did",

     

       231
       231
       +
                   "description": "DID of the aggregator service (must match repo DID)"

     

       232
       232
       +
                 },

     

       233
       233
       +
                 "displayName": {

     

       234
       234
       +
                   "type": "string",

     

       235
       235
       +
                   "maxGraphemes": 64,

     

       236
       236
       +
                   "maxLength": 640,

     

       237
       237
       +
                   "description": "Human-readable name (e.g., 'RSS News Aggregator')"

     

       238
       238
       +
                 },

     

       239
       239
       +
                 "description": {

     

       240
       240
       +
                   "type": "string",

     

       241
       241
       +
                   "maxGraphemes": 300,

     

       242
       242
       +
                   "maxLength": 3000,

     

       243
       243
       +
                   "description": "Description of what this aggregator does"

     

       244
       244
       +
                 },

     

       245
       245
       +
                 "avatar": {

     

       246
       246
       +
                   "type": "blob",

     

       247
       247
       +
                   "accept": ["image/png", "image/jpeg"],

     

       248
       248
       +
                   "maxSize": 1000000,

     

       249
       249
       +
                   "description": "Avatar image for bot identity"

     

       250
       250
       +
                 },

     

       251
       251
       +
                 "aggregatorType": {

     

       252
       252
       +
                   "type": "string",

     

       253
       253
       +
                   "knownValues": [

     

       254
       254
       +
                     "social.coves.aggregator.types#rss",

     

       255
       255
       +
                     "social.coves.aggregator.types#blueskyMirror",

     

       256
       256
       +
                     "social.coves.aggregator.types#sports",

     

       257
       257
       +
                     "social.coves.aggregator.types#media",

     

       258
       258
       +
                     "social.coves.aggregator.types#custom"

     

       259
       259
       +
                   ],

     

       260
       260
       +
                   "description": "Type of aggregator for categorization"

     

       261
       261
       +
                 },

     

       262
       262
       +
                 "configSchema": {

     

       263
       263
       +
                   "type": "unknown",

     

       264
       264
       +
                   "description": "JSON Schema describing config options for this aggregator. Communities use this to know what configuration fields are available."

     

       265
       265
       +
                 },

     

       266
       266
       +
                 "sourceUrl": {

     

       267
       267
       +
                   "type": "string",

     

       268
       268
       +
                   "format": "uri",

     

       269
       269
       +
                   "description": "URL to aggregator's source code (for transparency)"

     

       270
       270
       +
                 },

     

       271
       271
       +
                 "maintainer": {

     

       272
       272
       +
                   "type": "string",

     

       273
       273
       +
                   "format": "did",

     

       274
       274
       +
                   "description": "DID of person/organization maintaining this aggregator"

     

       275
       275
       +
                 },

     

       276
       276
       +
                 "createdAt": {

     

       277
       277
       +
                   "type": "string",

     

       278
       278
       +
                   "format": "datetime"

     

       279
       279
       +
                 }

     

       280
       280
       +
               }

     

       281
       281
       +
             }

     

       282
       282
       +
           }

     

       283
       283
       +
         }

     

       284
       284
       +
       }

     

       285
       285
       +
       ```

     

       286
       286
       +
       

     

       287
       287
       +
       **Example Record:**

     

       288
       288
       +
       ```json

     

       289
       289
       +
       {

     

       290
       290
       +
         "$type": "social.coves.aggregator.service",

     

       291
       291
       +
         "did": "did:web:rss-bot.coves.social",

     

       292
       292
       +
         "displayName": "RSS News Aggregator",

     

       293
       293
       +
         "description": "Automatically posts breaking news from configured RSS feeds with LLM-powered deduplication",

     

       294
       294
       +
         "aggregatorType": "social.coves.aggregator.types#rss",

     

       295
       295
       +
         "configSchema": {

     

       296
       296
       +
           "type": "object",

     

       297
       297
       +
           "properties": {

     

       298
       298
       +
             "feeds": {

     

       299
       299
       +
               "type": "array",

     

       300
       300
       +
               "items": { "type": "string", "format": "uri" }

     

       301
       301
       +
             },

     

       302
       302
       +
             "topics": {

     

       303
       303
       +
               "type": "array",

     

       304
       304
       +
               "items": { "type": "string" }

     

       305
       305
       +
             },

     

       306
       306
       +
             "dedupeWindow": { "type": "string" },

     

       307
       307
       +
             "minSources": { "type": "integer", "minimum": 1 }

     

       308
       308
       +
           }

     

       309
       309
       +
         },

     

       310
       310
       +
         "sourceUrl": "https://github.com/coves-social/rss-aggregator",

     

       311
       311
       +
         "maintainer": "did:plc:coves-platform",

     

       312
       312
       +
         "createdAt": "2025-10-19T12:00:00Z"

     

       313
       313
       +
       }

     

       314
       314
       +
       ```

     

       315
       315
       +
       

     

       316
       316
       +
       ---

     

       317
       317
       +
       

     

       318
       318
       +
       ### 2. Community Authorization Record

     

       319
       319
       +
       

     

       320
       320
       +
       **Collection:** `social.coves.aggregator.authorization`

     

       321
       321
       +
       **Key:** `any` (one per aggregator per community)

     

       322
       322
       +
       **Location:** Community's repository

     

       323
       323
       +
       

     

       324
       324
       +
       This record grants an aggregator permission to post to a community and contains aggregator-specific configuration.

     

       325
       325
       +
       

     

       326
       326
       +
       ```json

     

       327
       327
       +
       {

     

       328
       328
       +
         "lexicon": 1,

     

       329
       329
       +
         "id": "social.coves.aggregator.authorization",

     

       330
       330
       +
         "defs": {

     

       331
       331
       +
           "main": {

     

       332
       332
       +
             "type": "record",

     

       333
       333
       +
             "description": "Authorization for an aggregator to post to a community with specific configuration",

     

       334
       334
       +
             "key": "any",

     

       335
       335
       +
             "record": {

     

       336
       336
       +
               "type": "object",

     

       337
       337
       +
               "required": ["aggregatorDid", "communityDid", "createdAt", "enabled"],

     

       338
       338
       +
               "properties": {

     

       339
       339
       +
                 "aggregatorDid": {

     

       340
       340
       +
                   "type": "string",

     

       341
       341
       +
                   "format": "did",

     

       342
       342
       +
                   "description": "DID of the authorized aggregator"

     

       343
       343
       +
                 },

     

       344
       344
       +
                 "communityDid": {

     

       345
       345
       +
                   "type": "string",

     

       346
       346
       +
                   "format": "did",

     

       347
       347
       +
                   "description": "DID of the community granting access (must match repo DID)"

     

       348
       348
       +
                 },

     

       349
       349
       +
                 "enabled": {

     

       350
       350
       +
                   "type": "boolean",

     

       351
       351
       +
                   "description": "Whether this aggregator is currently active. Can be toggled without deleting the record."

     

       352
       352
       +
                 },

     

       353
       353
       +
                 "config": {

     

       354
       354
       +
                   "type": "unknown",

     

       355
       355
       +
                   "description": "Aggregator-specific configuration. Must conform to the aggregator's configSchema."

     

       356
       356
       +
                 },

     

       357
       357
       +
                 "createdAt": {

     

       358
       358
       +
                   "type": "string",

     

       359
       359
       +
                   "format": "datetime"

     

       360
       360
       +
                 },

     

       361
       361
       +
                 "createdBy": {

     

       362
       362
       +
                   "type": "string",

     

       363
       363
       +
                   "format": "did",

     

       364
       364
       +
                   "description": "DID of moderator who authorized this aggregator"

     

       365
       365
       +
                 },

     

       366
       366
       +
                 "disabledAt": {

     

       367
       367
       +
                   "type": "string",

     

       368
       368
       +
                   "format": "datetime",

     

       369
       369
       +
                   "description": "When this authorization was disabled (if enabled=false)"

     

       370
       370
       +
                 },

     

       371
       371
       +
                 "disabledBy": {

     

       372
       372
       +
                   "type": "string",

     

       373
       373
       +
                   "format": "did",

     

       374
       374
       +
                   "description": "DID of moderator who disabled this aggregator"

     

       375
       375
       +
                 }

     

       376
       376
       +
               }

     

       377
       377
       +
             }

     

       378
       378
       +
           }

     

       379
       379
       +
         }

     

       380
       380
       +
       }

     

       381
       381
       +
       ```

     

       382
       382
       +
       

     

       383
       383
       +
       **Example Record:**

     

       384
       384
       +
       ```json

     

       385
       385
       +
       {

     

       386
       386
       +
         "$type": "social.coves.aggregator.authorization",

     

       387
       387
       +
         "aggregatorDid": "did:web:rss-bot.coves.social",

     

       388
       388
       +
         "communityDid": "did:plc:gaming123",

     

       389
       389
       +
         "enabled": true,

     

       390
       390
       +
         "config": {

     

       391
       391
       +
           "feeds": [

     

       392
       392
       +
             "https://techcrunch.com/feed",

     

       393
       393
       +
             "https://arstechnica.com/feed"

     

       394
       394
       +
           ],

     

       395
       395
       +
           "topics": ["technology", "ai", "gaming"],

     

       396
       396
       +
           "dedupeWindow": "6h",

     

       397
       397
       +
           "minSources": 2

     

       398
       398
       +
         },

     

       399
       399
       +
         "createdAt": "2025-10-19T14:00:00Z",

     

       400
       400
       +
         "createdBy": "did:plc:alice123"

     

       401
       401
       +
       }

     

       402
       402
       +
       ```

     

       403
       403
       +
       

     

       404
       404
       +
       ---

     

       405
       405
       +
       

     

       406
       406
       +
       ### 3. Aggregator Type Definitions

     

       407
       407
       +
       

     

       408
       408
       +
       **Collection:** `social.coves.aggregator.types`

     

       409
       409
       +
       **Purpose:** Define known aggregator types for categorization

     

       410
       410
       +
       

     

       411
       411
       +
       ```json

     

       412
       412
       +
       {

     

       413
       413
       +
         "lexicon": 1,

     

       414
       414
       +
         "id": "social.coves.aggregator.types",

     

       415
       415
       +
         "defs": {

     

       416
       416
       +
           "rss": {

     

       417
       417
       +
             "type": "string",

     

       418
       418
       +
             "description": "Aggregator that monitors RSS/Atom feeds"

     

       419
       419
       +
           },

     

       420
       420
       +
           "blueskyMirror": {

     

       421
       421
       +
             "type": "string",

     

       422
       422
       +
             "description": "Aggregator that mirrors Bluesky posts"

     

       423
       423
       +
           },

     

       424
       424
       +
           "sports": {

     

       425
       425
       +
             "type": "string",

     

       426
       426
       +
             "description": "Aggregator for sports scores and game threads"

     

       427
       427
       +
           },

     

       428
       428
       +
           "media": {

     

       429
       429
       +
             "type": "string",

     

       430
       430
       +
             "description": "Aggregator for TV/movie discussion threads"

     

       431
       431
       +
           },

     

       432
       432
       +
           "custom": {

     

       433
       433
       +
             "type": "string",

     

       434
       434
       +
             "description": "Custom third-party aggregator"

     

       435
       435
       +
           }

     

       436
       436
       +
         }

     

       437
       437
       +
       }

     

       438
       438
       +
       ```

     

       439
       439
       +
       

     

       440
       440
       +
       ---

     

       441
       441
       +
       

     

       442
       442
       +
       ## XRPC Methods

     

       443
       443
       +
       

     

       444
       444
       +
       ### For Communities (Moderators)

     

       445
       445
       +
       

     

       446
       446
       +
       #### `social.coves.aggregator.enable`

     

       447
       447
       +
       Enable an aggregator for a community

     

       448
       448
       +
       

     

       449
       449
       +
       **Input:**

     

       450
       450
       +
       ```json

     

       451
       451
       +
       {

     

       452
       452
       +
         "aggregatorDid": "did:web:rss-bot.coves.social",

     

       453
       453
       +
         "config": {

     

       454
       454
       +
           "feeds": ["https://techcrunch.com/feed"],

     

       455
       455
       +
           "topics": ["technology"]

     

       456
       456
       +
         }

     

       457
       457
       +
       }

     

       458
       458
       +
       ```

     

       459
       459
       +
       

     

       460
       460
       +
       **Output:**

     

       461
       461
       +
       ```json

     

       462
       462
       +
       {

     

       463
       463
       +
         "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",

     

       464
       464
       +
         "cid": "bafyreif5...",

     

       465
       465
       +
         "authorization": {

     

       466
       466
       +
           "aggregatorDid": "did:web:rss-bot.coves.social",

     

       467
       467
       +
           "communityDid": "did:plc:gaming123",

     

       468
       468
       +
           "enabled": true,

     

       469
       469
       +
           "config": {...},

     

       470
       470
       +
           "createdAt": "2025-10-19T14:00:00Z"

     

       471
       471
       +
         }

     

       472
       472
       +
       }

     

       473
       473
       +
       ```

     

       474
       474
       +
       

     

       475
       475
       +
       **Behavior:**

     

       476
       476
       +
       - Validates caller is community moderator

     

       477
       477
       +
       - Validates aggregator exists and has service declaration

     

       478
       478
       +
       - Validates config against aggregator's configSchema

     

       479
       479
       +
       - Creates authorization record in community's repo

     

       480
       480
       +
       - Indexes to AppView for authorization checks

     

       481
       481
       +
       

     

       482
       482
       +
       **Errors:**

     

       483
       483
       +
       - `NotAuthorized` - Caller is not a moderator

     

       484
       484
       +
       - `AggregatorNotFound` - Aggregator DID doesn't exist

     

       485
       485
       +
       - `InvalidConfig` - Config doesn't match configSchema

     

       486
       486
       +
       

     

       487
       487
       +
       ---

     

       488
       488
       +
       

     

       489
       489
       +
       #### `social.coves.aggregator.disable`

     

       490
       490
       +
       Disable an aggregator for a community

     

       491
       491
       +
       

     

       492
       492
       +
       **Input:**

     

       493
       493
       +
       ```json

     

       494
       494
       +
       {

     

       495
       495
       +
         "aggregatorDid": "did:web:rss-bot.coves.social"

     

       496
       496
       +
       }

     

       497
       497
       +
       ```

     

       498
       498
       +
       

     

       499
       499
       +
       **Output:**

     

       500
       500
       +
       ```json

     

       501
       501
       +
       {

     

       502
       502
       +
         "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",

     

       503
       503
       +
         "disabled": true,

     

       504
       504
       +
         "disabledAt": "2025-10-19T15:00:00Z"

     

       505
       505
       +
       }

     

       506
       506
       +
       ```

     

       507
       507
       +
       

     

       508
       508
       +
       **Behavior:**

     

       509
       509
       +
       - Validates caller is community moderator

     

       510
       510
       +
       - Updates authorization record (sets `enabled=false`, `disabledAt`, `disabledBy`)

     

       511
       511
       +
       - Aggregator can no longer post until re-enabled

     

       512
       512
       +
       

     

       513
       513
       +
       ---

     

       514
       514
       +
       

     

       515
       515
       +
       #### `social.coves.aggregator.updateConfig`

     

       516
       516
       +
       Update configuration for an enabled aggregator

     

       517
       517
       +
       

     

       518
       518
       +
       **Input:**

     

       519
       519
       +
       ```json

     

       520
       520
       +
       {

     

       521
       521
       +
         "aggregatorDid": "did:web:rss-bot.coves.social",

     

       522
       522
       +
         "config": {

     

       523
       523
       +
           "feeds": ["https://techcrunch.com/feed", "https://arstechnica.com/feed"],

     

       524
       524
       +
           "topics": ["technology", "gaming"]

     

       525
       525
       +
         }

     

       526
       526
       +
       }

     

       527
       527
       +
       ```

     

       528
       528
       +
       

     

       529
       529
       +
       **Output:**

     

       530
       530
       +
       ```json

     

       531
       531
       +
       {

     

       532
       532
       +
         "uri": "at://did:plc:gaming123/social.coves.aggregator.authorization/3jui7kd58dt2g",

     

       533
       533
       +
         "cid": "bafyreif6...",

     

       534
       534
       +
         "config": {...}

     

       535
       535
       +
       }

     

       536
       536
       +
       ```

     

       537
       537
       +
       

     

       538
       538
       +
       ---

     

       539
       539
       +
       

     

       540
       540
       +
       #### `social.coves.aggregator.listForCommunity`

     

       541
       541
       +
       List all aggregators (enabled and disabled) for a community

     

       542
       542
       +
       

     

       543
       543
       +
       **Input:**

     

       544
       544
       +
       ```json

     

       545
       545
       +
       {

     

       546
       546
       +
         "community": "did:plc:gaming123",

     

       547
       547
       +
         "enabledOnly": false,

     

       548
       548
       +
         "limit": 50,

     

       549
       549
       +
         "cursor": "..."

     

       550
       550
       +
       }

     

       551
       551
       +
       ```

     

       552
       552
       +
       

     

       553
       553
       +
       **Output:**

     

       554
       554
       +
       ```json

     

       555
       555
       +
       {

     

       556
       556
       +
         "aggregators": [

     

       557
       557
       +
           {

     

       558
       558
       +
             "aggregatorDid": "did:web:rss-bot.coves.social",

     

       559
       559
       +
             "displayName": "RSS News Aggregator",

     

       560
       560
       +
             "description": "...",

     

       561
       561
       +
             "aggregatorType": "social.coves.aggregator.types#rss",

     

       562
       562
       +
             "enabled": true,

     

       563
       563
       +
             "config": {...},

     

       564
       564
       +
             "createdAt": "2025-10-19T14:00:00Z"

     

       565
       565
       +
           }

     

       566
       566
       +
         ],

     

       567
       567
       +
         "cursor": "..."

     

       568
       568
       +
       }

     

       569
       569
       +
       ```

     

       570
       570
       +
       

     

       571
       571
       +
       ---

     

       572
       572
       +
       

     

       573
       573
       +
       ### For Aggregators

     

       574
       574
       +
       

     

       575
       575
       +
       #### Existing: `social.coves.post.create`

     

       576
       576
       +
       **Modified Behavior:** Now handles aggregator authentication

     

       577
       577
       +
       

     

       578
       578
       +
       **Authorization Flow:**

     

       579
       579
       +
       1. Extract DID from JWT

     

       580
       580
       +
       2. Check if DID is registered aggregator (query `aggregators` table)

     

       581
       581
       +
       3. If aggregator:

     

       582
       582
       +
          - Validate authorization record exists for this community

     

       583
       583
       +
          - Check `enabled=true`

     

       584
       584
       +
          - Apply aggregator rate limits (e.g., 10 posts/hour)

     

       585
       585
       +
       4. If regular user:

     

       586
       586
       +
          - Validate membership, bans, etc. (existing logic)

     

       587
       587
       +
       5. Create post with `author = actorDID`

     

       588
       588
       +
       

     

       589
       589
       +
       **Rate Limits:**

     

       590
       590
       +
       - Regular users: 20 posts/hour per community

     

       591
       591
       +
       - Aggregators: 10 posts/hour per community (to prevent spam)

     

       592
       592
       +
       

     

       593
       593
       +
       ---

     

       594
       594
       +
       

     

       595
       595
       +
       #### `social.coves.aggregator.getAuthorizations`

     

       596
       596
       +
       Get list of communities that have authorized this aggregator

     

       597
       597
       +
       

     

       598
       598
       +
       **Input:**

     

       599
       599
       +
       ```json

     

       600
       600
       +
       {

     

       601
       601
       +
         "aggregatorDid": "did:web:rss-bot.coves.social",

     

       602
       602
       +
         "enabledOnly": true,

     

       603
       603
       +
         "limit": 100,

     

       604
       604
       +
         "cursor": "..."

     

       605
       605
       +
       }

     

       606
       606
       +
       ```

     

       607
       607
       +
       

     

       608
       608
       +
       **Output:**

     

       609
       609
       +
       ```json

     

       610
       610
       +
       {

     

       611
       611
       +
         "authorizations": [

     

       612
       612
       +
           {

     

       613
       613
       +
             "communityDid": "did:plc:gaming123",

     

       614
       614
       +
             "communityName": "Gaming News",

     

       615
       615
       +
             "enabled": true,

     

       616
       616
       +
             "config": {...},

     

       617
       617
       +
             "createdAt": "2025-10-19T14:00:00Z"

     

       618
       618
       +
           }

     

       619
       619
       +
         ],

     

       620
       620
       +
         "cursor": "..."

     

       621
       621
       +
       }

     

       622
       622
       +
       ```

     

       623
       623
       +
       

     

       624
       624
       +
       **Use Case:** Aggregator queries this to know which communities to post to

     

       625
       625
       +
       

     

       626
       626
       +
       ---

     

       627
       627
       +
       

     

       628
       628
       +
       ### For Discovery

     

       629
       629
       +
       

     

       630
       630
       +
       #### `social.coves.aggregator.list`

     

       631
       631
       +
       List all available aggregators

     

       632
       632
       +
       

     

       633
       633
       +
       **Input:**

     

       634
       634
       +
       ```json

     

       635
       635
       +
       {

     

       636
       636
       +
         "type": "social.coves.aggregator.types#rss",

     

       637
       637
       +
         "limit": 50,

     

       638
       638
       +
         "cursor": "..."

     

       639
       639
       +
       }

     

       640
       640
       +
       ```

     

       641
       641
       +
       

     

       642
       642
       +
       **Output:**

     

       643
       643
       +
       ```json

     

       644
       644
       +
       {

     

       645
       645
       +
         "aggregators": [

     

       646
       646
       +
           {

     

       647
       647
       +
             "did": "did:web:rss-bot.coves.social",

     

       648
       648
       +
             "displayName": "RSS News Aggregator",

     

       649
       649
       +
             "description": "...",

     

       650
       650
       +
             "aggregatorType": "social.coves.aggregator.types#rss",

     

       651
       651
       +
             "avatar": "...",

     

       652
       652
       +
             "maintainer": "did:plc:coves-platform",

     

       653
       653
       +
             "sourceUrl": "https://github.com/coves-social/rss-aggregator"

     

       654
       654
       +
           }

     

       655
       655
       +
         ],

     

       656
       656
       +
         "cursor": "..."

     

       657
       657
       +
       }

     

       658
       658
       +
       ```

     

       659
       659
       +
       

     

       660
       660
       +
       ---

     

       661
       661
       +
       

     

       662
       662
       +
       #### `social.coves.aggregator.get`

     

       663
       663
       +
       Get detailed information about a specific aggregator

     

       664
       664
       +
       

     

       665
       665
       +
       **Input:**

     

       666
       666
       +
       ```json

     

       667
       667
       +
       {

     

       668
       668
       +
         "aggregatorDid": "did:web:rss-bot.coves.social"

     

       669
       669
       +
       }

     

       670
       670
       +
       ```

     

       671
       671
       +
       

     

       672
       672
       +
       **Output:**

     

       673
       673
       +
       ```json

     

       674
       674
       +
       {

     

       675
       675
       +
         "did": "did:web:rss-bot.coves.social",

     

       676
       676
       +
         "displayName": "RSS News Aggregator",

     

       677
       677
       +
         "description": "...",

     

       678
       678
       +
         "aggregatorType": "social.coves.aggregator.types#rss",

     

       679
       679
       +
         "configSchema": {...},

     

       680
       680
       +
         "sourceUrl": "...",

     

       681
       681
       +
         "maintainer": "...",

     

       682
       682
       +
         "stats": {

     

       683
       683
       +
           "communitiesUsing": 42,

     

       684
       684
       +
           "postsCreated": 1337,

     

       685
       685
       +
           "createdAt": "2025-10-19T12:00:00Z"

     

       686
       686
       +
         }

     

       687
       687
       +
       }

     

       688
       688
       +
       ```

     

       689
       689
       +
       

     

       690
       690
       +
       ---

     

       691
       691
       +
       

     

       692
       692
       +
       ## Database Schema

     

       693
       693
       +
       

     

       694
       694
       +
       ### `aggregators` Table

     

       695
       695
       +
       Indexed aggregator service declarations from Jetstream

     

       696
       696
       +
       

     

       697
       697
       +
       ```sql

     

       698
       698
       +
       CREATE TABLE aggregators (

     

       699
       699
       +
         did TEXT PRIMARY KEY,

     

       700
       700
       +
         display_name TEXT NOT NULL,

     

       701
       701
       +
         description TEXT,

     

       702
       702
       +
         aggregator_type TEXT NOT NULL,

     

       703
       703
       +
         config_schema JSONB,

     

       704
       704
       +
         avatar_url TEXT,

     

       705
       705
       +
         source_url TEXT,

     

       706
       706
       +
         maintainer_did TEXT,

     

       707
       707
       +
       

     

       708
       708
       +
         -- Indexing metadata

     

       709
       709
       +
         record_uri TEXT NOT NULL,

     

       710
       710
       +
         record_cid TEXT NOT NULL,

     

       711
       711
       +
         indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

     

       712
       712
       +
       

     

       713
       713
       +
         -- Stats (cached)

     

       714
       714
       +
         communities_using INTEGER NOT NULL DEFAULT 0,

     

       715
       715
       +
         posts_created BIGINT NOT NULL DEFAULT 0,

     

       716
       716
       +
       

     

       717
       717
       +
         CONSTRAINT aggregators_type_check CHECK (

     

       718
       718
       +
           aggregator_type IN (

     

       719
       719
       +
             'social.coves.aggregator.types#rss',

     

       720
       720
       +
             'social.coves.aggregator.types#blueskyMirror',

     

       721
       721
       +
             'social.coves.aggregator.types#sports',

     

       722
       722
       +
             'social.coves.aggregator.types#media',

     

       723
       723
       +
             'social.coves.aggregator.types#custom'

     

       724
       724
       +
           )

     

       725
       725
       +
         )

     

       726
       726
       +
       );

     

       727
       727
       +
       

     

       728
       728
       +
       CREATE INDEX idx_aggregators_type ON aggregators(aggregator_type);

     

       729
       729
       +
       CREATE INDEX idx_aggregators_indexed_at ON aggregators(indexed_at DESC);

     

       730
       730
       +
       ```

     

       731
       731
       +
       

     

       732
       732
       +
       ---

     

       733
       733
       +
       

     

       734
       734
       +
       ### `aggregator_authorizations` Table

     

       735
       735
       +
       Indexed authorization records from communities

     

       736
       736
       +
       

     

       737
       737
       +
       ```sql

     

       738
       738
       +
       CREATE TABLE aggregator_authorizations (

     

       739
       739
       +
         id BIGSERIAL PRIMARY KEY,

     

       740
       740
       +
       

     

       741
       741
       +
         -- Authorization identity

     

       742
       742
       +
         aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,

     

       743
       743
       +
         community_did TEXT NOT NULL,

     

       744
       744
       +
       

     

       745
       745
       +
         -- Authorization state

     

       746
       746
       +
         enabled BOOLEAN NOT NULL DEFAULT true,

     

       747
       747
       +
         config JSONB,

     

       748
       748
       +
       

     

       749
       749
       +
         -- Audit trail

     

       750
       750
       +
         created_at TIMESTAMPTZ NOT NULL,

     

       751
       751
       +
         created_by TEXT NOT NULL, -- DID of moderator

     

       752
       752
       +
         disabled_at TIMESTAMPTZ,

     

       753
       753
       +
         disabled_by TEXT, -- DID of moderator

     

       754
       754
       +
       

     

       755
       755
       +
         -- atProto record metadata

     

       756
       756
       +
         record_uri TEXT NOT NULL UNIQUE,

     

       757
       757
       +
         record_cid TEXT NOT NULL,

     

       758
       758
       +
         indexed_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

     

       759
       759
       +
       

     

       760
       760
       +
         UNIQUE(aggregator_did, community_did)

     

       761
       761
       +
       );

     

       762
       762
       +
       

     

       763
       763
       +
       CREATE INDEX idx_aggregator_auth_agg_did ON aggregator_authorizations(aggregator_did) WHERE enabled = true;

     

       764
       764
       +
       CREATE INDEX idx_aggregator_auth_comm_did ON aggregator_authorizations(community_did) WHERE enabled = true;

     

       765
       765
       +
       CREATE INDEX idx_aggregator_auth_enabled ON aggregator_authorizations(enabled);

     

       766
       766
       +
       ```

     

       767
       767
       +
       

     

       768
       768
       +
       ---

     

       769
       769
       +
       

     

       770
       770
       +
       ### `aggregator_posts` Table

     

       771
       771
       +
       Track posts created by aggregators (for rate limiting and stats)

     

       772
       772
       +
       

     

       773
       773
       +
       ```sql

     

       774
       774
       +
       CREATE TABLE aggregator_posts (

     

       775
       775
       +
         id BIGSERIAL PRIMARY KEY,

     

       776
       776
       +
       

     

       777
       777
       +
         aggregator_did TEXT NOT NULL REFERENCES aggregators(did) ON DELETE CASCADE,

     

       778
       778
       +
         community_did TEXT NOT NULL,

     

       779
       779
       +
         post_uri TEXT NOT NULL,

     

       780
       780
       +
         post_cid TEXT NOT NULL,

     

       781
       781
       +
       

     

       782
       782
       +
         created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),

     

       783
       783
       +
       

     

       784
       784
       +
         UNIQUE(post_uri)

     

       785
       785
       +
       );

     

       786
       786
       +
       

     

       787
       787
       +
       CREATE INDEX idx_aggregator_posts_agg_did_created ON aggregator_posts(aggregator_did, created_at DESC);

     

       788
       788
       +
       CREATE INDEX idx_aggregator_posts_comm_did_created ON aggregator_posts(community_did, created_at DESC);

     

       789
       789
       +
       

     

       790
       790
       +
       -- For rate limiting: count posts in last hour

     

       791
       791
       +
       CREATE INDEX idx_aggregator_posts_rate_limit ON aggregator_posts(aggregator_did, community_did, created_at DESC);

     

       792
       792
       +
       ```

     

       793
       793
       +
       

     

       794
       794
       +
       ---

     

       795
       795
       +
       

     

       796
       796
       +
       ## Implementation Plan

     

       797
       797
       +
       

     

       798
       798
       +
       ### Phase 1: Core Infrastructure (Coves AppView)

     

       799
       799
       +
       

     

       800
       800
       +
       **Goal:** Enable aggregator authentication and authorization

     

       801
       801
       +
       

     

       802
       802
       +
       #### 1.1 Database Setup

     

       803
       803
       +
       - [ ] Create migration for `aggregators` table

     

       804
       804
       +
       - [ ] Create migration for `aggregator_authorizations` table

     

       805
       805
       +
       - [ ] Create migration for `aggregator_posts` table

     

       806
       806
       +
       

     

       807
       807
       +
       #### 1.2 Lexicon Definitions

     

       808
       808
       +
       - [ ] Create `social.coves.aggregator.service.json`

     

       809
       809
       +
       - [ ] Create `social.coves.aggregator.authorization.json`

     

       810
       810
       +
       - [ ] Create `social.coves.aggregator.types.json`

     

       811
       811
       +
       - [ ] Generate Go types from lexicons

     

       812
       812
       +
       

     

       813
       813
       +
       #### 1.3 Repository Layer

     

       814
       814
       +
       ```go

     

       815
       815
       +
       // internal/core/aggregators/repository.go

     

       816
       816
       +
       

     

       817
       817
       +
       type Repository interface {

     

       818
       818
       +
           // Aggregator management

     

       819
       819
       +
           CreateAggregator(ctx context.Context, agg *Aggregator) error

     

       820
       820
       +
           GetAggregator(ctx context.Context, did string) (*Aggregator, error)

     

       821
       821
       +
           ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)

     

       822
       822
       +
           UpdateAggregatorStats(ctx context.Context, did string, stats Stats) error

     

       823
       823
       +
       

     

       824
       824
       +
           // Authorization management

     

       825
       825
       +
           CreateAuthorization(ctx context.Context, auth *Authorization) error

     

       826
       826
       +
           GetAuthorization(ctx context.Context, aggDID, commDID string) (*Authorization, error)

     

       827
       827
       +
           ListAuthorizationsForAggregator(ctx context.Context, aggDID string, enabledOnly bool) ([]*Authorization, error)

     

       828
       828
       +
           ListAuthorizationsForCommunity(ctx context.Context, commDID string) ([]*Authorization, error)

     

       829
       829
       +
           UpdateAuthorization(ctx context.Context, auth *Authorization) error

     

       830
       830
       +
           IsAuthorized(ctx context.Context, aggDID, commDID string) (bool, error)

     

       831
       831
       +
       

     

       832
       832
       +
           // Post tracking (for rate limiting)

     

       833
       833
       +
           RecordAggregatorPost(ctx context.Context, aggDID, commDID, postURI string) error

     

       834
       834
       +
           CountRecentPosts(ctx context.Context, aggDID, commDID string, since time.Time) (int, error)

     

       835
       835
       +
       }

     

       836
       836
       +
       ```

     

       837
       837
       +
       

     

       838
       838
       +
       #### 1.4 Service Layer

     

       839
       839
       +
       ```go

     

       840
       840
       +
       // internal/core/aggregators/service.go

     

       841
       841
       +
       

     

       842
       842
       +
       type Service interface {

     

       843
       843
       +
           // For communities (moderators)

     

       844
       844
       +
           EnableAggregator(ctx context.Context, commDID, aggDID string, config map[string]interface{}) (*Authorization, error)

     

       845
       845
       +
           DisableAggregator(ctx context.Context, commDID, aggDID string) error

     

       846
       846
       +
           UpdateAggregatorConfig(ctx context.Context, commDID, aggDID string, config map[string]interface{}) error

     

       847
       847
       +
           ListCommunityAggregators(ctx context.Context, commDID string, enabledOnly bool) ([]*AggregatorInfo, error)

     

       848
       848
       +
       

     

       849
       849
       +
           // For aggregators

     

       850
       850
       +
           GetAuthorizedCommunities(ctx context.Context, aggDID string) ([]*CommunityAuth, error)

     

       851
       851
       +
       

     

       852
       852
       +
           // For discovery

     

       853
       853
       +
           ListAggregators(ctx context.Context, filter AggregatorFilter) ([]*Aggregator, error)

     

       854
       854
       +
           GetAggregator(ctx context.Context, did string) (*AggregatorDetail, error)

     

       855
       855
       +
       

     

       856
       856
       +
           // Internal: called by post creation handler

     

       857
       857
       +
           ValidateAggregatorPost(ctx context.Context, aggDID, commDID string) error

     

       858
       858
       +
       }

     

       859
       859
       +
       ```

     

       860
       860
       +
       

     

       861
       861
       +
       #### 1.5 Modify Post Creation Handler

     

       862
       862
       +
       ```go

     

       863
       863
       +
       // internal/api/handlers/post/create.go

     

       864
       864
       +
       

     

       865
       865
       +
       func CreatePost(ctx context.Context, input *CreatePostInput) (*CreatePostOutput, error) {

     

       866
       866
       +
           actorDID := GetDIDFromAuth(ctx)

     

       867
       867
       +
       

     

       868
       868
       +
           // Check if actor is an aggregator

     

       869
       869
       +
           if isAggregator, _ := aggregatorService.IsAggregator(ctx, actorDID); isAggregator {

     

       870
       870
       +
               // Validate aggregator authorization

     

       871
       871
       +
               if err := aggregatorService.ValidateAggregatorPost(ctx, actorDID, input.Community); err != nil {

     

       872
       872
       +
                   return nil, err

     

       873
       873
       +
               }

     

       874
       874
       +
       

     

       875
       875
       +
               // Apply aggregator rate limits

     

       876
       876
       +
               if err := rateLimitAggregator(ctx, actorDID, input.Community); err != nil {

     

       877
       877
       +
                   return nil, ErrRateLimitExceeded

     

       878
       878
       +
               }

     

       879
       879
       +
           } else {

     

       880
       880
       +
               // Regular user validation (existing logic)

     

       881
       881
       +
               // ... membership checks, ban checks, etc.

     

       882
       882
       +
           }

     

       883
       883
       +
       

     

       884
       884
       +
           // Create post (author will be actorDID - either user or aggregator)

     

       885
       885
       +
           post, err := postService.CreatePost(ctx, actorDID, input)

     

       886
       886
       +
           if err != nil {

     

       887
       887
       +
               return nil, err

     

       888
       888
       +
           }

     

       889
       889
       +
       

     

       890
       890
       +
           // If aggregator, track the post

     

       891
       891
       +
           if isAggregator {

     

       892
       892
       +
               _ = aggregatorService.RecordPost(ctx, actorDID, input.Community, post.URI)

     

       893
       893
       +
           }

     

       894
       894
       +
       

     

       895
       895
       +
           return post, nil

     

       896
       896
       +
       }

     

       897
       897
       +
       ```

     

       898
       898
       +
       

     

       899
       899
       +
       #### 1.6 XRPC Handlers

     

       900
       900
       +
       - [ ] `social.coves.aggregator.enable` handler

     

       901
       901
       +
       - [ ] `social.coves.aggregator.disable` handler

     

       902
       902
       +
       - [ ] `social.coves.aggregator.updateConfig` handler

     

       903
       903
       +
       - [ ] `social.coves.aggregator.listForCommunity` handler

     

       904
       904
       +
       - [ ] `social.coves.aggregator.getAuthorizations` handler

     

       905
       905
       +
       - [ ] `social.coves.aggregator.list` handler

     

       906
       906
       +
       - [ ] `social.coves.aggregator.get` handler

     

       907
       907
       +
       

     

       908
       908
       +
       #### 1.7 Jetstream Consumer

     

       909
       909
       +
       ```go

     

       910
       910
       +
       // internal/atproto/jetstream/aggregator_consumer.go

     

       911
       911
       +
       

     

       912
       912
       +
       func (c *AggregatorConsumer) HandleEvent(ctx context.Context, evt *jetstream.Event) error {

     

       913
       913
       +
           switch evt.Collection {

     

       914
       914
       +
           case "social.coves.aggregator.service":

     

       915
       915
       +
               switch evt.Operation {

     

       916
       916
       +
               case "create", "update":

     

       917
       917
       +
                   return c.indexAggregatorService(ctx, evt)

     

       918
       918
       +
               case "delete":

     

       919
       919
       +
                   return c.deleteAggregator(ctx, evt.DID)

     

       920
       920
       +
               }

     

       921
       921
       +
       

     

       922
       922
       +
           case "social.coves.aggregator.authorization":

     

       923
       923
       +
               switch evt.Operation {

     

       924
       924
       +
               case "create", "update":

     

       925
       925
       +
                   return c.indexAuthorization(ctx, evt)

     

       926
       926
       +
               case "delete":

     

       927
       927
       +
                   return c.deleteAuthorization(ctx, evt.URI)

     

       928
       928
       +
               }

     

       929
       929
       +
           }

     

       930
       930
       +
           return nil

     

       931
       931
       +
       }

     

       932
       932
       +
       ```

     

       933
       933
       +
       

     

       934
       934
       +
       #### 1.8 Integration Tests

     

       935
       935
       +
       - [ ] Test aggregator service indexing from Jetstream

     

       936
       936
       +
       - [ ] Test authorization record indexing

     

       937
       937
       +
       - [ ] Test `social.coves.post.create` with aggregator auth

     

       938
       938
       +
       - [ ] Test authorization validation (enabled/disabled)

     

       939
       939
       +
       - [ ] Test rate limiting for aggregators

     

       940
       940
       +
       - [ ] Test config validation against schema

     

       941
       941
       +
       

     

       942
       942
       +
       **Milestone:** Aggregators can authenticate and post to communities with authorization

     

       943
       943
       +
       

     

       944
       944
       +
       ---

     

       945
       945
       +
       

     

       946
       946
       +
       ### Phase 2: Aggregator SDK (Go)

     

       947
       947
       +
       

     

       948
       948
       +
       **Goal:** Provide SDK for building aggregators easily

     

       949
       949
       +
       

     

       950
       950
       +
       #### 2.1 SDK Core

     

       951
       951
       +
       ```go

     

       952
       952
       +
       // github.com/coves-social/aggregator-sdk-go

     

       953
       953
       +
       

     

       954
       954
       +
       package aggregator

     

       955
       955
       +
       

     

       956
       956
       +
       type Aggregator interface {

     

       957
       957
       +
           // Identity

     

       958
       958
       +
           GetDID() string

     

       959
       959
       +
           GetDisplayName() string

     

       960
       960
       +
           GetDescription() string

     

       961
       961
       +
           GetType() string

     

       962
       962
       +
           GetConfigSchema() map[string]interface{}

     

       963
       963
       +
       

     

       964
       964
       +
           // Lifecycle

     

       965
       965
       +
           Start(ctx context.Context) error

     

       966
       966
       +
           Stop() error

     

       967
       967
       +
       

     

       968
       968
       +
           // Posting (provided by SDK)

     

       969
       969
       +
           CreatePost(ctx context.Context, communityDID string, post Post) error

     

       970
       970
       +
           GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error)

     

       971
       971
       +
       }

     

       972
       972
       +
       

     

       973
       973
       +
       type BaseAggregator struct {

     

       974
       974
       +
           DID          string

     

       975
       975
       +
           DisplayName  string

     

       976
       976
       +
           Description  string

     

       977
       977
       +
           Type         string

     

       978
       978
       +
           PrivateKey   crypto.PrivateKey

     

       979
       979
       +
           CovesAPIURL  string

     

       980
       980
       +
       

     

       981
       981
       +
           client *http.Client

     

       982
       982
       +
       }

     

       983
       983
       +
       

     

       984
       984
       +
       type Post struct {

     

       985
       985
       +
           Title          string

     

       986
       986
       +
           Content        string

     

       987
       987
       +
           Embed          interface{}

     

       988
       988
       +
           FederatedFrom  *FederatedSource

     

       989
       989
       +
           ContentLabels  []string

     

       990
       990
       +
       }

     

       991
       991
       +
       

     

       992
       992
       +
       type FederatedSource struct {

     

       993
       993
       +
           Platform           string // "rss", "bluesky", etc.

     

       994
       994
       +
           URI                string

     

       995
       995
       +
           ID                 string

     

       996
       996
       +
           OriginalCreatedAt  time.Time

     

       997
       997
       +
       }

     

       998
       998
       +
       

     

       999
       999
       +
       // Helper methods provided by SDK

     

       1000
       1000
       +
       func (a *BaseAggregator) CreatePost(ctx context.Context, communityDID string, post Post) error {

     

       1001
       1001
       +
           // 1. Sign JWT with aggregator's private key

     

       1002
       1002
       +
           token := a.signJWT()

     

       1003
       1003
       +
       

     

       1004
       1004
       +
           // 2. Call social.coves.post.create via XRPC

     

       1005
       1005
       +
           resp, err := a.client.Post(

     

       1006
       1006
       +
               a.CovesAPIURL + "/xrpc/social.coves.post.create",

     

       1007
       1007
       +
               &CreatePostInput{

     

       1008
       1008
       +
                   Community:     communityDID,

     

       1009
       1009
       +
                   Title:         post.Title,

     

       1010
       1010
       +
                   Content:       post.Content,

     

       1011
       1011
       +
                   Embed:         post.Embed,

     

       1012
       1012
       +
                   FederatedFrom: post.FederatedFrom,

     

       1013
       1013
       +
                   ContentLabels: post.ContentLabels,

     

       1014
       1014
       +
               },

     

       1015
       1015
       +
               &CreatePostOutput{},

     

       1016
       1016
       +
               WithAuth(token),

     

       1017
       1017
       +
           )

     

       1018
       1018
       +
       

     

       1019
       1019
       +
           return err

     

       1020
       1020
       +
       }

     

       1021
       1021
       +
       

     

       1022
       1022
       +
       func (a *BaseAggregator) GetAuthorizedCommunities(ctx context.Context) ([]*CommunityAuth, error) {

     

       1023
       1023
       +
           // Call social.coves.aggregator.getAuthorizations

     

       1024
       1024
       +
           token := a.signJWT()

     

       1025
       1025
       +
       

     

       1026
       1026
       +
           resp, err := a.client.Get(

     

       1027
       1027
       +
               a.CovesAPIURL + "/xrpc/social.coves.aggregator.getAuthorizations",

     

       1028
       1028
       +
               map[string]string{"aggregatorDid": a.DID, "enabledOnly": "true"},

     

       1029
       1029
       +
               &GetAuthorizationsOutput{},

     

       1030
       1030
       +
               WithAuth(token),

     

       1031
       1031
       +
           )

     

       1032
       1032
       +
       

     

       1033
       1033
       +
           return resp.Authorizations, err

     

       1034
       1034
       +
       }

     

       1035
       1035
       +
       ```

     

       1036
       1036
       +
       

     

       1037
       1037
       +
       #### 2.2 SDK Documentation

     

       1038
       1038
       +
       - [ ] README with quickstart guide

     

       1039
       1039
       +
       - [ ] Example aggregators (RSS, Bluesky mirror)

     

       1040
       1040
       +
       - [ ] API reference documentation

     

       1041
       1041
       +
       - [ ] Configuration schema guide

     

       1042
       1042
       +
       

     

       1043
       1043
       +
       **Milestone:** Third parties can build aggregators using SDK

     

       1044
       1044
       +
       

     

       1045
       1045
       +
       ---

     

       1046
       1046
       +
       

     

       1047
       1047
       +
       ### Phase 3: Reference Aggregator (RSS)

     

       1048
       1048
       +
       

     

       1049
       1049
       +
       **Goal:** Build working RSS aggregator as reference implementation

     

       1050
       1050
       +
       

     

       1051
       1051
       +
       #### 3.1 RSS Aggregator Implementation

     

       1052
       1052
       +
       ```go

     

       1053
       1053
       +
       // github.com/coves-social/rss-aggregator

     

       1054
       1054
       +
       

     

       1055
       1055
       +
       package main

     

       1056
       1056
       +
       

     

       1057
       1057
       +
       import "github.com/coves-social/aggregator-sdk-go"

     

       1058
       1058
       +
       

     

       1059
       1059
       +
       type RSSAggregator struct {

     

       1060
       1060
       +
           *aggregator.BaseAggregator

     

       1061
       1061
       +
       

     

       1062
       1062
       +
           // RSS-specific config

     

       1063
       1063
       +
           pollInterval time.Duration

     

       1064
       1064
       +
           llmClient    *openai.Client

     

       1065
       1065
       +
       }

     

       1066
       1066
       +
       

     

       1067
       1067
       +
       func (r *RSSAggregator) Start(ctx context.Context) error {

     

       1068
       1068
       +
           // 1. Get authorized communities

     

       1069
       1069
       +
           communities, err := r.GetAuthorizedCommunities(ctx)

     

       1070
       1070
       +
           if err != nil {

     

       1071
       1071
       +
               return err

     

       1072
       1072
       +
           }

     

       1073
       1073
       +
       

     

       1074
       1074
       +
           // 2. Start polling loop

     

       1075
       1075
       +
           ticker := time.NewTicker(r.pollInterval)

     

       1076
       1076
       +
           defer ticker.Stop()

     

       1077
       1077
       +
       

     

       1078
       1078
       +
           for {

     

       1079
       1079
       +
               select {

     

       1080
       1080
       +
               case <-ticker.C:

     

       1081
       1081
       +
                   r.pollFeeds(ctx, communities)

     

       1082
       1082
       +
               case <-ctx.Done():

     

       1083
       1083
       +
                   return nil

     

       1084
       1084
       +
               }

     

       1085
       1085
       +
           }

     

       1086
       1086
       +
       }

     

       1087
       1087
       +
       

     

       1088
       1088
       +
       func (r *RSSAggregator) pollFeeds(ctx context.Context, communities []*CommunityAuth) {

     

       1089
       1089
       +
           for _, comm := range communities {

     

       1090
       1090
       +
               // Get RSS feeds from community config

     

       1091
       1091
       +
               feeds := comm.Config["feeds"].([]string)

     

       1092
       1092
       +
       

     

       1093
       1093
       +
               for _, feedURL := range feeds {

     

       1094
       1094
       +
                   items, err := r.fetchFeed(feedURL)

     

       1095
       1095
       +
                   if err != nil {

     

       1096
       1096
       +
                       continue

     

       1097
       1097
       +
                   }

     

       1098
       1098
       +
       

     

       1099
       1099
       +
                   // Process new items

     

       1100
       1100
       +
                   for _, item := range items {

     

       1101
       1101
       +
                       // Check if already posted

     

       1102
       1102
       +
                       if r.alreadyPosted(item.GUID) {

     

       1103
       1103
       +
                           continue

     

       1104
       1104
       +
                       }

     

       1105
       1105
       +
       

     

       1106
       1106
       +
                       // LLM deduplication logic

     

       1107
       1107
       +
                       duplicate := r.findDuplicate(item, comm.CommunityDID)

     

       1108
       1108
       +
                       if duplicate != nil {

     

       1109
       1109
       +
                           r.addToMegathread(duplicate, item)

     

       1110
       1110
       +
                           continue

     

       1111
       1111
       +
                       }

     

       1112
       1112
       +
       

     

       1113
       1113
       +
                       // Create new post

     

       1114
       1114
       +
                       post := aggregator.Post{

     

       1115
       1115
       +
                           Title:   item.Title,

     

       1116
       1116
       +
                           Content: r.summarize(item),

     

       1117
       1117
       +
                           FederatedFrom: &aggregator.FederatedSource{

     

       1118
       1118
       +
                               Platform:          "rss",

     

       1119
       1119
       +
                               URI:               item.Link,

     

       1120
       1120
       +
                               OriginalCreatedAt: item.PublishedAt,

     

       1121
       1121
       +
                           },

     

       1122
       1122
       +
                       }

     

       1123
       1123
       +
       

     

       1124
       1124
       +
                       err = r.CreatePost(ctx, comm.CommunityDID, post)

     

       1125
       1125
       +
                       if err != nil {

     

       1126
       1126
       +
                           log.Printf("Failed to create post: %v", err)

     

       1127
       1127
       +
                           continue

     

       1128
       1128
       +
                       }

     

       1129
       1129
       +
       

     

       1130
       1130
       +
                       r.markPosted(item.GUID)

     

       1131
       1131
       +
                   }

     

       1132
       1132
       +
               }

     

       1133
       1133
       +
           }

     

       1134
       1134
       +
       }

     

       1135
       1135
       +
       

     

       1136
       1136
       +
       func (r *RSSAggregator) summarize(item *RSSItem) string {

     

       1137
       1137
       +
           // Use LLM to create unbiased summary

     

       1138
       1138
       +
           prompt := fmt.Sprintf("Summarize this news article in 2-3 sentences: %s", item.Description)

     

       1139
       1139
       +
           summary, _ := r.llmClient.Complete(prompt)

     

       1140
       1140
       +
           return summary

     

       1141
       1141
       +
       }

     

       1142
       1142
       +
       

     

       1143
       1143
       +
       func (r *RSSAggregator) findDuplicate(item *RSSItem, communityDID string) *Post {

     

       1144
       1144
       +
           // Use LLM to detect semantic duplicates

     

       1145
       1145
       +
           // Query recent posts in community

     

       1146
       1146
       +
           // Compare with embeddings/similarity

     

       1147
       1147
       +
           return nil // or duplicate post

     

       1148
       1148
       +
       }

     

       1149
       1149
       +
       ```

     

       1150
       1150
       +
       

     

       1151
       1151
       +
       #### 3.2 Deployment

     

       1152
       1152
       +
       - [ ] Dockerfile for RSS aggregator

     

       1153
       1153
       +
       - [ ] Kubernetes manifests (for Coves-hosted instance)

     

       1154
       1154
       +
       - [ ] Environment configuration guide

     

       1155
       1155
       +
       - [ ] Monitoring and logging setup

     

       1156
       1156
       +
       

     

       1157
       1157
       +
       #### 3.3 Testing

     

       1158
       1158
       +
       - [ ] Unit tests for feed parsing

     

       1159
       1159
       +
       - [ ] Integration tests with mock Coves API

     

       1160
       1160
       +
       - [ ] E2E test with real Coves instance

     

       1161
       1161
       +
       - [ ] LLM deduplication accuracy tests

     

       1162
       1162
       +
       

     

       1163
       1163
       +
       **Milestone:** RSS aggregator running in production for select communities

     

       1164
       1164
       +
       

     

       1165
       1165
       +
       ---

     

       1166
       1166
       +
       

     

       1167
       1167
       +
       ### Phase 4: Additional Aggregators

     

       1168
       1168
       +
       

     

       1169
       1169
       +
       #### 4.1 Bluesky Mirror Aggregator

     

       1170
       1170
       +
       - [ ] Monitor Jetstream for specific users/hashtags

     

       1171
       1171
       +
       - [ ] Preserve `originalAuthor` metadata

     

       1172
       1172
       +
       - [ ] Link back to original Bluesky post

     

       1173
       1173
       +
       - [ ] Rate limiting (don't flood community)

     

       1174
       1174
       +
       

     

       1175
       1175
       +
       #### 4.2 Sports Aggregator

     

       1176
       1176
       +
       - [ ] Integrate with ESPN/TheSportsDB APIs

     

       1177
       1177
       +
       - [ ] Monitor game completions

     

       1178
       1178
       +
       - [ ] Create post-game threads with stats

     

       1179
       1179
       +
       - [ ] Auto-pin major games

     

       1180
       1180
       +
       

     

       1181
       1181
       +
       #### 4.3 Media (TV/Movie) Aggregator

     

       1182
       1182
       +
       - [ ] Integrate with TMDB API

     

       1183
       1183
       +
       - [ ] Track show release schedules

     

       1184
       1184
       +
       - [ ] Create episode discussion threads

     

       1185
       1185
       +
       - [ ] Spoiler protection tags

     

       1186
       1186
       +
       

     

       1187
       1187
       +
       **Milestone:** Multiple official aggregators available for communities

     

       1188
       1188
       +
       

     

       1189
       1189
       +
       ---

     

       1190
       1190
       +
       

     

       1191
       1191
       +
       ## Security Considerations

     

       1192
       1192
       +
       

     

       1193
       1193
       +
       ### Authentication

     

       1194
       1194
       +
       ✅ **DID-based Authentication**

     

       1195
       1195
       +
       - Aggregators sign JWTs with their private keys

     

       1196
       1196
       +
       - Server validates JWT signature against DID document

     

       1197
       1197
       +
       - No shared secrets or API keys

     

       1198
       1198
       +
       

     

       1199
       1199
       +
       ✅ **Scoped Authorization**

     

       1200
       1200
       +
       - Authorization records are per-community

     

       1201
       1201
       +
       - Aggregator can only post to authorized communities

     

       1202
       1202
       +
       - Communities can revoke at any time

     

       1203
       1203
       +
       

     

       1204
       1204
       +
       ### Rate Limiting

     

       1205
       1205
       +
       ✅ **Per-Aggregator Limits**

     

       1206
       1206
       +
       - 10 posts/hour per community (configurable)

     

       1207
       1207
       +
       - Prevents aggregator spam

     

       1208
       1208
       +
       - Separate from user rate limits

     

       1209
       1209
       +
       

     

       1210
       1210
       +
       ✅ **Global Limits**

     

       1211
       1211
       +
       - Total posts across all communities: 100/hour

     

       1212
       1212
       +
       - Prevents runaway aggregators

     

       1213
       1213
       +
       

     

       1214
       1214
       +
       ### Content Validation

     

       1215
       1215
       +
       ✅ **Community Rules**

     

       1216
       1216
       +
       - Aggregator posts validated against community content rules

     

       1217
       1217
       +
       - No special exemptions (same rules as users)

     

       1218
       1218
       +
       - Community can ban specific content patterns

     

       1219
       1219
       +
       

     

       1220
       1220
       +
       ✅ **Config Validation**

     

       1221
       1221
       +
       - Authorization config validated against aggregator's configSchema

     

       1222
       1222
       +
       - Prevents injection attacks via config

     

       1223
       1223
       +
       - JSON schema validation

     

       1224
       1224
       +
       

     

       1225
       1225
       +
       ### Monitoring & Auditing

     

       1226
       1226
       +
       ✅ **Audit Trail**

     

       1227
       1227
       +
       - All aggregator posts logged

     

       1228
       1228
       +
       - `created_by` tracks which moderator authorized

     

       1229
       1229
       +
       - `disabled_by` tracks who revoked access

     

       1230
       1230
       +
       - Full history preserved

     

       1231
       1231
       +
       

     

       1232
       1232
       +
       ✅ **Abuse Detection**

     

       1233
       1233
       +
       - Monitor for spam patterns

     

       1234
       1234
       +
       - Alert if aggregator posts rejected repeatedly

     

       1235
       1235
       +
       - Auto-disable after threshold violations

     

       1236
       1236
       +
       

     

       1237
       1237
       +
       ### Transparency

     

       1238
       1238
       +
       ✅ **Open Source**

     

       1239
       1239
       +
       - Official aggregators open source

     

       1240
       1240
       +
       - Source URL in service declaration

     

       1241
       1241
       +
       - Community can audit behavior

     

       1242
       1242
       +
       

     

       1243
       1243
       +
       ✅ **Attribution**

     

       1244
       1244
       +
       - Posts clearly show aggregator authorship

     

       1245
       1245
       +
       - UI shows "🤖 Posted by [Aggregator Name]"

     

       1246
       1246
       +
       - No attempt to impersonate users

     

       1247
       1247
       +
       

     

       1248
       1248
       +
       ---

     

       1249
       1249
       +
       

     

       1250
       1250
       +
       ## UI/UX Considerations

     

       1251
       1251
       +
       

     

       1252
       1252
       +
       ### Community Settings

     

       1253
       1253
       +
       **Aggregator Management Page:**

     

       1254
       1254
       +
       - List of available aggregators (with descriptions, types)

     

       1255
       1255
       +
       - "Enable" button opens config modal

     

       1256
       1256
       +
       - Config form generated from aggregator's configSchema

     

       1257
       1257
       +
       - Toggle to enable/disable without deleting config

     

       1258
       1258
       +
       - Stats: posts created, last active

     

       1259
       1259
       +
       

     

       1260
       1260
       +
       **Post Display:**

     

       1261
       1261
       +
       - Posts from aggregators have bot badge: "🤖"

     

       1262
       1262
       +
       - Shows aggregator name (e.g., "Posted by RSS News Bot")

     

       1263
       1263
       +
       - `federatedFrom` shows original source

     

       1264
       1264
       +
       - Link to original content (RSS article, Bluesky post, etc.)

     

       1265
       1265
       +
       

     

       1266
       1266
       +
       ### User Preferences

     

       1267
       1267
       +
       - Option to hide all aggregator posts

     

       1268
       1268
       +
       - Option to hide specific aggregators

     

       1269
       1269
       +
       - Filter posts by "user-created only" or "include bots"

     

       1270
       1270
       +
       

     

       1271
       1271
       +
       ---

     

       1272
       1272
       +
       

     

       1273
       1273
       +
       ## Success Metrics

     

       1274
       1274
       +
       

     

       1275
       1275
       +
       ### Pre-Launch Checklist

     

       1276
       1276
       +
       - [ ] Lexicons defined and validated

     

       1277
       1277
       +
       - [ ] Database migrations tested

     

       1278
       1278
       +
       - [ ] Jetstream consumer indexes aggregator records

     

       1279
       1279
       +
       - [ ] Post creation handler validates aggregator auth

     

       1280
       1280
       +
       - [ ] Rate limiting prevents spam

     

       1281
       1281
       +
       - [ ] SDK published and documented

     

       1282
       1282
       +
       - [ ] Reference RSS aggregator working

     

       1283
       1283
       +
       - [ ] E2E tests passing

     

       1284
       1284
       +
       - [ ] Security audit completed

     

       1285
       1285
       +
       

     

       1286
       1286
       +
       ### Alpha Goals

     

       1287
       1287
       +
       - 3+ official aggregators (RSS, Bluesky mirror, sports)

     

       1288
       1288
       +
       - 10+ communities using aggregators

     

       1289
       1289
       +
       - < 0.1% spam posts (false positives)

     

       1290
       1290
       +
       - Aggregator posts appear in feed within 1 minute

     

       1291
       1291
       +
       

     

       1292
       1292
       +
       ### Beta Goals

     

       1293
       1293
       +
       - Third-party aggregators launched

     

       1294
       1294
       +
       - 50+ communities using aggregators

     

       1295
       1295
       +
       - Developer documentation complete

     

       1296
       1296
       +
       - Marketplace/directory for discovery

     

       1297
       1297
       +
       

     

       1298
       1298
       +
       ---

     

       1299
       1299
       +
       

     

       1300
       1300
       +
       ## Out of Scope (Future Versions)

     

       1301
       1301
       +
       

     

       1302
       1302
       +
       ### Aggregator Marketplace

     

       1303
       1303
       +
       - [ ] Community ratings/reviews for aggregators

     

       1304
       1304
       +
       - [ ] Featured aggregators

     

       1305
       1305
       +
       - [ ] Paid aggregators (premium features)

     

       1306
       1306
       +
       - [ ] Aggregator analytics dashboard

     

       1307
       1307
       +
       

     

       1308
       1308
       +
       ### Advanced Features

     

       1309
       1309
       +
       - [ ] Scheduled posts (post at specific time)

     

       1310
       1310
       +
       - [ ] Content moderation integration (auto-label NSFW)

     

       1311
       1311
       +
       - [ ] Multi-community posting (single post to multiple communities)

     

       1312
       1312
       +
       - [ ] Interactive aggregators (respond to comments)

     

       1313
       1313
       +
       - [ ] Aggregator-to-aggregator communication (chains)

     

       1314
       1314
       +
       

     

       1315
       1315
       +
       ### Federation

     

       1316
       1316
       +
       - [ ] Cross-instance aggregator discovery

     

       1317
       1317
       +
       - [ ] Aggregator migration (change hosting provider)

     

       1318
       1318
       +
       - [ ] Federated aggregator authorization (trust other instances' aggregators)

     

       1319
       1319
       +
       

     

       1320
       1320
       +
       ---

     

       1321
       1321
       +
       

     

       1322
       1322
       +
       ## Technical Decisions Log

     

       1323
       1323
       +
       

     

       1324
       1324
       +
       ### 2025-10-19: Reuse `social.coves.post.create` Endpoint

     

       1325
       1325
       +
       

     

       1326
       1326
       +
       **Decision:** Aggregators use existing post creation endpoint, not a separate `social.coves.aggregator.post.create`

     

       1327
       1327
       +
       

     

       1328
       1328
       +
       **Rationale:**

     

       1329
       1329
       +
       - Post record already server-populates `author` field from JWT

     

       1330
       1330
       +
       - Aggregators authenticate as themselves → `author = aggregator DID`

     

       1331
       1331
       +
       - Simpler: one code path for all post creation

     

       1332
       1332
       +
       - Follows atProto principle: actors are actors (users, bots, aggregators)

     

       1333
       1333
       +
       - `federatedFrom` field already handles external content attribution

     

       1334
       1334
       +
       

     

       1335
       1335
       +
       **Implementation:**

     

       1336
       1336
       +
       - Add authorization check to `social.coves.post.create` handler

     

       1337
       1337
       +
       - Check if authenticated DID is aggregator

     

       1338
       1338
       +
       - Validate authorization record exists and enabled

     

       1339
       1339
       +
       - Apply aggregator-specific rate limits

     

       1340
       1340
       +
       - Otherwise same logic as user posts

     

       1341
       1341
       +
       

     

       1342
       1342
       +
       **Trade-offs Accepted:**

     

       1343
       1343
       +
       - Post creation handler has branching logic (user vs aggregator)

     

       1344
       1344
       +
       - But: keeps lexicon simple, reuses existing validation

     

       1345
       1345
       +
       

     

       1346
       1346
       +
       ---

     

       1347
       1347
       +
       

     

       1348
       1348
       +
       ### 2025-10-19: Hybrid Hosting Model

     

       1349
       1349
       +
       

     

       1350
       1350
       +
       **Decision:** Support both Coves-hosted and third-party aggregators

     

       1351
       1351
       +
       

     

       1352
       1352
       +
       **Rationale:**

     

       1353
       1353
       +
       - Coves can provide high-quality official aggregators (RSS, sports, media)

     

       1354
       1354
       +
       - Third parties can build specialized aggregators (niche communities)

     

       1355
       1355
       +
       - SDK makes it easy to build custom aggregators

     

       1356
       1356
       +
       - Follows feed generator model (anyone can run one)

     

       1357
       1357
       +
       - Decentralization-friendly

     

       1358
       1358
       +
       

     

       1359
       1359
       +
       **Requirements:**

     

       1360
       1360
       +
       - SDK must be well-documented and maintained

     

       1361
       1361
       +
       - Authorization system must be DID-agnostic (works for any DID)

     

       1362
       1362
       +
       - Discovery system shows all aggregators (official + third-party)

     

       1363
       1363
       +
       

     

       1364
       1364
       +
       ---

     

       1365
       1365
       +
       

     

       1366
       1366
       +
       ### 2025-10-19: Config as JSON Schema

     

       1367
       1367
       +
       

     

       1368
       1368
       +
       **Decision:** Aggregators declare configSchema in their service record

     

       1369
       1369
       +
       

     

       1370
       1370
       +
       **Rationale:**

     

       1371
       1371
       +
       - Communities need to know what config options are available

     

       1372
       1372
       +
       - JSON Schema is standard, well-supported

     

       1373
       1373
       +
       - Enables UI auto-generation (forms from schema)

     

       1374
       1374
       +
       - Validation at authorization creation time

     

       1375
       1375
       +
       - Flexible: each aggregator can have different config structure

     

       1376
       1376
       +
       

     

       1377
       1377
       +
       **Example:**

     

       1378
       1378
       +
       ```json

     

       1379
       1379
       +
       {

     

       1380
       1380
       +
         "configSchema": {

     

       1381
       1381
       +
           "type": "object",

     

       1382
       1382
       +
           "properties": {

     

       1383
       1383
       +
             "feeds": {

     

       1384
       1384
       +
               "type": "array",

     

       1385
       1385
       +
               "items": { "type": "string", "format": "uri" },

     

       1386
       1386
       +
               "description": "RSS feed URLs to monitor"

     

       1387
       1387
       +
             },

     

       1388
       1388
       +
             "topics": {

     

       1389
       1389
       +
               "type": "array",

     

       1390
       1390
       +
               "items": { "type": "string" },

     

       1391
       1391
       +
               "description": "Topics to filter posts by"

     

       1392
       1392
       +
             }

     

       1393
       1393
       +
           },

     

       1394
       1394
       +
           "required": ["feeds"]

     

       1395
       1395
       +
         }

     

       1396
       1396
       +
       }

     

       1397
       1397
       +
       ```

     

       1398
       1398
       +
       

     

       1399
       1399
       +
       **Trade-offs Accepted:**

     

       1400
       1400
       +
       - More complex than simple key-value config

     

       1401
       1401
       +
       - But: better UX (self-documenting), prevents errors

     

       1402
       1402
       +
       

     

       1403
       1403
       +
       ---

     

       1404
       1404
       +
       

     

       1405
       1405
       +
       ## References

     

       1406
       1406
       +
       

     

       1407
       1407
       +
       - atProto Lexicon Spec: https://atproto.com/specs/lexicon

     

       1408
       1408
       +
       - Feed Generator Starter Kit: https://github.com/bluesky-social/feed-generator

     

       1409
       1409
       +
       - Labeler Implementation: https://github.com/bluesky-social/atproto/tree/main/packages/ozone

     

       1410
       1410
       +
       - JSON Schema Spec: https://json-schema.org/

     

       1411
       1411
       +
       - Coves Communities PRD: [PRD_COMMUNITIES.md](PRD_COMMUNITIES.md)

     

       1412
       1412
       +
       - Coves Posts Implementation: [IMPLEMENTATION_POST_CREATION.md](IMPLEMENTATION_POST_CREATION.md)

+1294

docs/aggregators/PRD_KAGI_NEWS_RSS.md

···

       1
       1
       +
       # Kagi News RSS Aggregator PRD

     

       2
       2
       +
       

     

       3
       3
       +
       **Status:** Planning Phase

     

       4
       4
       +
       **Owner:** Platform Team

     

       5
       5
       +
       **Last Updated:** 2025-10-20

     

       6
       6
       +
       **Parent PRD:** [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       7
       7
       +
       

     

       8
       8
       +
       ## Overview

     

       9
       9
       +
       

     

       10
       10
       +
       The Kagi News RSS Aggregator is a reference implementation of the Coves aggregator system that automatically posts high-quality, multi-source news summaries to communities. It leverages Kagi News's free RSS feeds to provide pre-aggregated, deduped news content with multiple perspectives and source citations.

     

       11
       11
       +
       

     

       12
       12
       +
       **Key Value Propositions:**

     

       13
       13
       +
       - **Multi-source aggregation**: Kagi News already aggregates multiple sources per story

     

       14
       14
       +
       - **Balanced perspectives**: Built-in perspective tracking from different outlets

     

       15
       15
       +
       - **Rich metadata**: Categories, highlights, source links included

     

       16
       16
       +
       - **Legal & free**: CC BY-NC licensed for non-commercial use

     

       17
       17
       +
       - **Low complexity**: No LLM deduplication needed (Kagi does it)

     

       18
       18
       +
       

     

       19
       19
       +
       ## Data Source: Kagi News RSS Feeds

     

       20
       20
       +
       

     

       21
       21
       +
       ### Licensing & Legal

     

       22
       22
       +
       

     

       23
       23
       +
       **License:** CC BY-NC (Creative Commons Attribution-NonCommercial)

     

       24
       24
       +
       

     

       25
       25
       +
       **Terms:**

     

       26
       26
       +
       - ✅ **Free for non-commercial use** (Coves qualifies)

     

       27
       27
       +
       - ✅ **Attribution required** (must credit Kagi News)

     

       28
       28
       +
       - ❌ **Cannot use commercially** (must contact support@kagi.com for commercial license)

     

       29
       29
       +
       - ✅ **Data can be shared** (with same attribution + NC requirements)

     

       30
       30
       +
       

     

       31
       31
       +
       **Source:** https://news.kagi.com/about

     

       32
       32
       +
       

     

       33
       33
       +
       **Quote from Kagi:**

     

       34
       34
       +
       > Note that kite.json and files referenced by it are licensed under CC BY-NC license. This means that this data can be used free of charge (with attribution and for non-commercial use). If you would like to license this data for commercial use let us know through support@kagi.com.

     

       35
       35
       +
       

     

       36
       36
       +
       **Compliance Requirements:**

     

       37
       37
       +
       - Visible attribution to Kagi News on every post

     

       38
       38
       +
       - Link back to original Kagi story page

     

       39
       39
       +
       - Non-commercial operation (met: Coves is non-commercial)

     

       40
       40
       +
       

     

       41
       41
       +
       ---

     

       42
       42
       +
       

     

       43
       43
       +
       ### RSS Feed Structure

     

       44
       44
       +
       

     

       45
       45
       +
       **Base URL Pattern:** `https://news.kagi.com/{category}.xml`

     

       46
       46
       +
       

     

       47
       47
       +
       **Known Categories:**

     

       48
       48
       +
       - `world.xml` - World news

     

       49
       49
       +
       - `tech.xml` - Technology (likely)

     

       50
       50
       +
       - `business.xml` - Business (likely)

     

       51
       51
       +
       - `sports.xml` - Sports (likely)

     

       52
       52
       +
       - Additional categories TBD (need to scrape homepage)

     

       53
       53
       +
       

     

       54
       54
       +
       **Feed Format:** RSS 2.0 (standard XML)

     

       55
       55
       +
       

     

       56
       56
       +
       **Update Frequency:** One daily update (~noon UTC)

     

       57
       57
       +
       

     

       58
       58
       +
       ---

     

       59
       59
       +
       

     

       60
       60
       +
       ### RSS Item Schema

     

       61
       61
       +
       

     

       62
       62
       +
       Each `<item>` in the feed contains:

     

       63
       63
       +
       

     

       64
       64
       +
       ```xml

     

       65
       65
       +
       <item>

     

       66
       66
       +
         <title>Story headline</title>

     

       67
       67
       +
         <link>https://kite.kagi.com/{uuid}/{category}/{id}</link>

     

       68
       68
       +
         <description>Full HTML content (see below)</description>

     

       69
       69
       +
         <guid isPermaLink="true">https://kite.kagi.com/{uuid}/{category}/{id}</guid>

     

       70
       70
       +
         <category>Primary category (e.g., "World")</category>

     

       71
       71
       +
         <category>Subcategory (e.g., "World/Conflict & Security")</category>

     

       72
       72
       +
         <category>Tag (e.g., "Conflict & Security")</category>

     

       73
       73
       +
         <pubDate>Mon, 20 Oct 2025 01:46:31 +0000</pubDate>

     

       74
       74
       +
       </item>

     

       75
       75
       +
       ```

     

       76
       76
       +
       

     

       77
       77
       +
       **Description HTML Structure:**

     

       78
       78
       +
       ```html

     

       79
       79
       +
       <p>Main summary paragraph with inline source citations [source1.com#1][source2.com#1]</p>

     

       80
       80
       +
       

     

       81
       81
       +
       <img src='https://kagiproxy.com/img/...' alt='Image caption' />

     

       82
       82
       +
       

     

       83
       83
       +
       <h3>Highlights:</h3>

     

       84
       84
       +
       <ul>

     

       85
       85
       +
         <li>Key point 1 with [source.com#1] citations</li>

     

       86
       86
       +
         <li>Key point 2...</li>

     

       87
       87
       +
       </ul>

     

       88
       88
       +
       

     

       89
       89
       +
       <blockquote>Notable quote - Person Name</blockquote>

     

       90
       90
       +
       

     

       91
       91
       +
       <h3>Perspectives:</h3>

     

       92
       92
       +
       <ul>

     

       93
       93
       +
         <li>Viewpoint holder: Their perspective. (<a href='...'>Source</a>)</li>

     

       94
       94
       +
       </ul>

     

       95
       95
       +
       

     

       96
       96
       +
       <h3>Sources:</h3>

     

       97
       97
       +
       <ul>

     

       98
       98
       +
         <li><a href='https://...'>Article title</a> - domain.com</li>

     

       99
       99
       +
       </ul>

     

       100
       100
       +
       ```

     

       101
       101
       +
       

     

       102
       102
       +
       **Key Features:**

     

       103
       103
       +
       - Multiple source citations inline

     

       104
       104
       +
       - Balanced perspectives from different actors

     

       105
       105
       +
       - Highlights extract key points

     

       106
       106
       +
       - Direct quotes preserved

     

       107
       107
       +
       - All sources linked with attribution

     

       108
       108
       +
       

     

       109
       109
       +
       ---

     

       110
       110
       +
       

     

       111
       111
       +
       ## Architecture

     

       112
       112
       +
       

     

       113
       113
       +
       ### High-Level Flow

     

       114
       114
       +
       

     

       115
       115
       +
       ```

     

       116
       116
       +
       ┌─────────────────────────────────────────────────────────────┐

     

       117
       117
       +
       │  Kagi News RSS Feeds (External)                             │

     

       118
       118
       +
       │  - https://news.kagi.com/world.xml                          │

     

       119
       119
       +
       │  - https://news.kagi.com/tech.xml                           │

     

       120
       120
       +
       │  - etc.                                                     │

     

       121
       121
       +
       └─────────────────────────────────────────────────────────────┘

     

       122
       122
       +
                                   │

     

       123
       123
       +
                                   │ HTTP GET one job after update

     

       124
       124
       +
                                   ▼

     

       125
       125
       +
       ┌─────────────────────────────────────────────────────────────┐

     

       126
       126
       +
       │  Kagi News Aggregator Service                               │

     

       127
       127
       +
       │  DID: did:web:kagi-news.coves.social                        │

     

       128
       128
       +
       │                                                             │

     

       129
       129
       +
       │  Components:                                                 │

     

       130
       130
       +
       │  1. Feed Poller: Fetches RSS feeds on schedule              │

     

       131
       131
       +
       │  2. Item Parser: Extracts structured data from HTML         │

     

       132
       132
       +
       │  3. Deduplication: Tracks posted GUIDs (no LLM needed)      │

     

       133
       133
       +
       │  4. Category Mapper: Maps Kagi categories to communities    │

     

       134
       134
       +
       │  5. Post Formatter: Converts to Coves post format           │

     

       135
       135
       +
       │  6. Post Publisher: Calls social.coves.post.create          │

     

       136
       136
       +
       └─────────────────────────────────────────────────────────────┘

     

       137
       137
       +
                                   │

     

       138
       138
       +
                                   │ Authenticated XRPC calls

     

       139
       139
       +
                                   ▼

     

       140
       140
       +
       ┌─────────────────────────────────────────────────────────────┐

     

       141
       141
       +
       │  Coves AppView (social.coves.post.create)                   │

     

       142
       142
       +
       │  - Validates aggregator authorization                        │

     

       143
       143
       +
       │  - Creates post with author = did:web:kagi-news.coves.social│

     

       144
       144
       +
       │  - Indexes to community feeds                                │

     

       145
       145
       +
       └─────────────────────────────────────────────────────────────┘

     

       146
       146
       +
       ```

     

       147
       147
       +
       

     

       148
       148
       +
       ---

     

       149
       149
       +
       

     

       150
       150
       +
       ### Aggregator Service Declaration

     

       151
       151
       +
       

     

       152
       152
       +
       ```json

     

       153
       153
       +
       {

     

       154
       154
       +
         "$type": "social.coves.aggregator.service",

     

       155
       155
       +
         "did": "did:web:kagi-news.coves.social",

     

       156
       156
       +
         "displayName": "Kagi News Aggregator",

     

       157
       157
       +
         "description": "Automatically posts breaking news from Kagi News RSS feeds. Kagi News aggregates multiple sources per story with balanced perspectives and comprehensive source citations.",

     

       158
       158
       +
         "aggregatorType": "social.coves.aggregator.types#rss",

     

       159
       159
       +
         "avatar": "<blob reference to Kagi logo>",

     

       160
       160
       +
         "configSchema": {

     

       161
       161
       +
           "type": "object",

     

       162
       162
       +
           "properties": {

     

       163
       163
       +
             "categories": {

     

       164
       164
       +
               "type": "array",

     

       165
       165
       +
               "items": {

     

       166
       166
       +
                 "type": "string",

     

       167
       167
       +
                 "enum": ["world", "tech", "business", "sports", "science"]

     

       168
       168
       +
               },

     

       169
       169
       +
               "description": "Kagi News categories to monitor",

     

       170
       170
       +
               "minItems": 1

     

       171
       171
       +
             },

     

       172
       172
       +
             "subcategoryFilter": {

     

       173
       173
       +
               "type": "array",

     

       174
       174
       +
               "items": { "type": "string" },

     

       175
       175
       +
               "description": "Optional: only post stories with these subcategories (e.g., 'World/Middle East', 'Tech/AI')"

     

       176
       176
       +
             },

     

       177
       177
       +
             "minSources": {

     

       178
       178
       +
               "type": "integer",

     

       179
       179
       +
               "minimum": 1,

     

       180
       180
       +
               "default": 2,

     

       181
       181
       +
               "description": "Minimum number of sources required for a story to be posted"

     

       182
       182
       +
             },

     

       183
       183
       +
             "includeImages": {

     

       184
       184
       +
               "type": "boolean",

     

       185
       185
       +
               "default": true,

     

       186
       186
       +
               "description": "Include images from Kagi proxy in posts"

     

       187
       187
       +
             },

     

       188
       188
       +
             "postFormat": {

     

       189
       189
       +
               "type": "string",

     

       190
       190
       +
               "enum": ["full", "summary", "minimal"],

     

       191
       191
       +
               "default": "full",

     

       192
       192
       +
               "description": "How much content to include: full (all sections), summary (main paragraph + sources), minimal (title + link only)"

     

       193
       193
       +
             }

     

       194
       194
       +
           },

     

       195
       195
       +
           "required": ["categories"]

     

       196
       196
       +
         },

     

       197
       197
       +
         "sourceUrl": "https://github.com/coves-social/kagi-news-aggregator",

     

       198
       198
       +
         "maintainer": "did:plc:coves-platform",

     

       199
       199
       +
         "createdAt": "2025-10-20T12:00:00Z"

     

       200
       200
       +
       }

     

       201
       201
       +
       ```

     

       202
       202
       +
       

     

       203
       203
       +
       ---

     

       204
       204
       +
       

     

       205
       205
       +
       ## Community Configuration Examples

     

       206
       206
       +
       

     

       207
       207
       +
       ### Example 1: World News Community

     

       208
       208
       +
       

     

       209
       209
       +
       ```json

     

       210
       210
       +
       {

     

       211
       211
       +
         "aggregatorDid": "did:web:kagi-news.coves.social",

     

       212
       212
       +
         "enabled": true,

     

       213
       213
       +
         "config": {

     

       214
       214
       +
           "categories": ["world"],

     

       215
       215
       +
           "minSources": 3,

     

       216
       216
       +
           "includeImages": true,

     

       217
       217
       +
           "postFormat": "full"

     

       218
       218
       +
         }

     

       219
       219
       +
       }

     

       220
       220
       +
       ```

     

       221
       221
       +
       

     

       222
       222
       +
       **Result:** Posts all world news stories with 3+ sources, full content including images/highlights/perspectives.

     

       223
       223
       +
       

     

       224
       224
       +
       ---

     

       225
       225
       +
       

     

       226
       226
       +
       ### Example 2: AI/Tech Community (Filtered)

     

       227
       227
       +
       

     

       228
       228
       +
       ```json

     

       229
       229
       +
       {

     

       230
       230
       +
         "aggregatorDid": "did:web:kagi-news.coves.social",

     

       231
       231
       +
         "enabled": true,

     

       232
       232
       +
         "config": {

     

       233
       233
       +
           "categories": ["tech", "business"],

     

       234
       234
       +
           "subcategoryFilter": ["Tech/AI", "Tech/Machine Learning", "Business/Tech Industry"],

     

       235
       235
       +
           "minSources": 2,

     

       236
       236
       +
           "includeImages": true,

     

       237
       237
       +
           "postFormat": "full"

     

       238
       238
       +
         }

     

       239
       239
       +
       }

     

       240
       240
       +
       ```

     

       241
       241
       +
       

     

       242
       242
       +
       **Result:** Only posts tech stories about AI/ML or tech industry business news with 2+ sources.

     

       243
       243
       +
       

     

       244
       244
       +
       ---

     

       245
       245
       +
       

     

       246
       246
       +
       ### Example 3: Breaking News (Minimal)

     

       247
       247
       +
       

     

       248
       248
       +
       ```json

     

       249
       249
       +
       {

     

       250
       250
       +
         "aggregatorDid": "did:web:kagi-news.coves.social",

     

       251
       251
       +
         "enabled": true,

     

       252
       252
       +
         "config": {

     

       253
       253
       +
           "categories": ["world", "business", "tech"],

     

       254
       254
       +
           "minSources": 5,

     

       255
       255
       +
           "includeImages": false,

     

       256
       256
       +
           "postFormat": "minimal"

     

       257
       257
       +
         }

     

       258
       258
       +
       }

     

       259
       259
       +
       ```

     

       260
       260
       +
       

     

       261
       261
       +
       **Result:** Only major stories (5+ sources), minimal format (headline + link), no images.

     

       262
       262
       +
       

     

       263
       263
       +
       ---

     

       264
       264
       +
       

     

       265
       265
       +
       ## Post Format Specification

     

       266
       266
       +
       

     

       267
       267
       +
       ### Post Record Structure

     

       268
       268
       +
       

     

       269
       269
       +
       ```json

     

       270
       270
       +
       {

     

       271
       271
       +
         "$type": "social.coves.post.record",

     

       272
       272
       +
         "author": "did:web:kagi-news.coves.social",

     

       273
       273
       +
         "community": "did:plc:worldnews123",

     

       274
       274
       +
         "title": "{Kagi story title}",

     

       275
       275
       +
         "content": "{formatted content based on postFormat config}",

     

       276
       276
       +
         "embed": {

     

       277
       277
       +
           "$type": "app.bsky.embed.external",

     

       278
       278
       +
           "external": {

     

       279
       279
       +
             "uri": "https://kite.kagi.com/{uuid}/{category}/{id}",

     

       280
       280
       +
             "title": "{story title}",

     

       281
       281
       +
             "description": "{summary excerpt}",

     

       282
       282
       +
             "thumb": "{image blob if includeImages=true}"

     

       283
       283
       +
           }

     

       284
       284
       +
         },

     

       285
       285
       +
         "federatedFrom": {

     

       286
       286
       +
           "platform": "kagi-news-rss",

     

       287
       287
       +
           "uri": "https://kite.kagi.com/{uuid}/{category}/{id}",

     

       288
       288
       +
           "id": "{guid}",

     

       289
       289
       +
           "originalCreatedAt": "{pubDate from RSS}"

     

       290
       290
       +
         },

     

       291
       291
       +
         "contentLabels": [

     

       292
       292
       +
           "{primary category}",

     

       293
       293
       +
           "{subcategories}"

     

       294
       294
       +
         ],

     

       295
       295
       +
         "createdAt": "{current timestamp}"

     

       296
       296
       +
       }

     

       297
       297
       +
       ```

     

       298
       298
       +
       

     

       299
       299
       +
       ---

     

       300
       300
       +
       

     

       301
       301
       +
       ### Content Formatting by `postFormat`

     

       302
       302
       +
       

     

       303
       303
       +
       #### Format: `full` (Default)

     

       304
       304
       +
       

     

       305
       305
       +
       ```markdown

     

       306
       306
       +
       {Main summary paragraph with source citations}

     

       307
       307
       +
       

     

       308
       308
       +
       **Highlights:**

     

       309
       309
       +
       • {Bullet point 1}

     

       310
       310
       +
       • {Bullet point 2}

     

       311
       311
       +
       • ...

     

       312
       312
       +
       

     

       313
       313
       +
       **Perspectives:**

     

       314
       314
       +
       • **{Actor}**: {Their perspective} ([Source]({url}))

     

       315
       315
       +
       • ...

     

       316
       316
       +
       

     

       317
       317
       +
       > {Notable quote} — {Attribution}

     

       318
       318
       +
       

     

       319
       319
       +
       **Sources:**

     

       320
       320
       +
       • [{Title}]({url}) - {domain}

     

       321
       321
       +
       • ...

     

       322
       322
       +
       

     

       323
       323
       +
       ---

     

       324
       324
       +
       📰 Story aggregated by [Kagi News]({kagi_story_url})

     

       325
       325
       +
       ```

     

       326
       326
       +
       

     

       327
       327
       +
       **Rationale:** Preserves Kagi's rich multi-source analysis, provides maximum value.

     

       328
       328
       +
       

     

       329
       329
       +
       ---

     

       330
       330
       +
       

     

       331
       331
       +
       #### Format: `summary`

     

       332
       332
       +
       

     

       333
       333
       +
       ```markdown

     

       334
       334
       +
       {Main summary paragraph with source citations}

     

       335
       335
       +
       

     

       336
       336
       +
       **Sources:**

     

       337
       337
       +
       • [{Title}]({url}) - {domain}

     

       338
       338
       +
       • ...

     

       339
       339
       +
       

     

       340
       340
       +
       ---

     

       341
       341
       +
       📰 Story aggregated by [Kagi News]({kagi_story_url})

     

       342
       342
       +
       ```

     

       343
       343
       +
       

     

       344
       344
       +
       **Rationale:** Clean summary with source links, less overwhelming.

     

       345
       345
       +
       

     

       346
       346
       +
       ---

     

       347
       347
       +
       

     

       348
       348
       +
       #### Format: `minimal`

     

       349
       349
       +
       

     

       350
       350
       +
       ```markdown

     

       351
       351
       +
       {Story title}

     

       352
       352
       +
       

     

       353
       353
       +
       Read more: {kagi_story_url}

     

       354
       354
       +
       

     

       355
       355
       +
       **Sources:** {domain1}, {domain2}, {domain3}...

     

       356
       356
       +
       

     

       357
       357
       +
       ---

     

       358
       358
       +
       📰 Via [Kagi News]({kagi_story_url})

     

       359
       359
       +
       ```

     

       360
       360
       +
       

     

       361
       361
       +
       **Rationale:** Just headlines with link, for high-volume communities or breaking news alerts.

     

       362
       362
       +
       

     

       363
       363
       +
       ---

     

       364
       364
       +
       

     

       365
       365
       +
       ## Implementation Details

     

       366
       366
       +
       

     

       367
       367
       +
       ### Component 1: Feed Poller

     

       368
       368
       +
       

     

       369
       369
       +
       **Responsibility:** Fetch RSS feeds on schedule

     

       370
       370
       +
       

     

       371
       371
       +
       ```go

     

       372
       372
       +
       type FeedPoller struct {

     

       373
       373
       +
           categories   []string

     

       374
       374
       +
           pollInterval time.Duration

     

       375
       375
       +
           httpClient   *http.Client

     

       376
       376
       +
       }

     

       377
       377
       +
       

     

       378
       378
       +
       func (p *FeedPoller) Start(ctx context.Context) error {

     

       379
       379
       +
           ticker := time.NewTicker(p.pollInterval) // 15 minutes

     

       380
       380
       +
           defer ticker.Stop()

     

       381
       381
       +
       

     

       382
       382
       +
           for {

     

       383
       383
       +
               select {

     

       384
       384
       +
               case <-ticker.C:

     

       385
       385
       +
                   for _, category := range p.categories {

     

       386
       386
       +
                       feedURL := fmt.Sprintf("https://news.kagi.com/%s.xml", category)

     

       387
       387
       +
                       feed, err := p.fetchFeed(feedURL)

     

       388
       388
       +
                       if err != nil {

     

       389
       389
       +
                           log.Printf("Failed to fetch %s: %v", feedURL, err)

     

       390
       390
       +
                           continue

     

       391
       391
       +
                       }

     

       392
       392
       +
                       p.handleFeed(ctx, category, feed)

     

       393
       393
       +
                   }

     

       394
       394
       +
               case <-ctx.Done():

     

       395
       395
       +
                   return nil

     

       396
       396
       +
               }

     

       397
       397
       +
           }

     

       398
       398
       +
       }

     

       399
       399
       +
       

     

       400
       400
       +
       func (p *FeedPoller) fetchFeed(url string) (*gofeed.Feed, error) {

     

       401
       401
       +
           parser := gofeed.NewParser()

     

       402
       402
       +
           feed, err := parser.ParseURL(url)

     

       403
       403
       +
           return feed, err

     

       404
       404
       +
       }

     

       405
       405
       +
       ```

     

       406
       406
       +
       

     

       407
       407
       +
       **Libraries:**

     

       408
       408
       +
       - `github.com/mmcdole/gofeed` - RSS/Atom parser

     

       409
       409
       +
       

     

       410
       410
       +
       ---

     

       411
       411
       +
       

     

       412
       412
       +
       ### Component 2: Item Parser

     

       413
       413
       +
       

     

       414
       414
       +
       **Responsibility:** Extract structured data from RSS item HTML

     

       415
       415
       +
       

     

       416
       416
       +
       ```go

     

       417
       417
       +
       type KagiStory struct {

     

       418
       418
       +
           Title             string

     

       419
       419
       +
           Link              string

     

       420
       420
       +
           GUID              string

     

       421
       421
       +
           PubDate           time.Time

     

       422
       422
       +
           Categories        []string

     

       423
       423
       +
       

     

       424
       424
       +
           // Parsed from HTML description

     

       425
       425
       +
           Summary           string

     

       426
       426
       +
           Highlights        []string

     

       427
       427
       +
           Perspectives      []Perspective

     

       428
       428
       +
           Quote             *Quote

     

       429
       429
       +
           Sources           []Source

     

       430
       430
       +
           ImageURL          string

     

       431
       431
       +
           ImageAlt          string

     

       432
       432
       +
       }

     

       433
       433
       +
       

     

       434
       434
       +
       type Perspective struct {

     

       435
       435
       +
           Actor       string

     

       436
       436
       +
           Description string

     

       437
       437
       +
           SourceURL   string

     

       438
       438
       +
       }

     

       439
       439
       +
       

     

       440
       440
       +
       type Quote struct {

     

       441
       441
       +
           Text        string

     

       442
       442
       +
           Attribution string

     

       443
       443
       +
       }

     

       444
       444
       +
       

     

       445
       445
       +
       type Source struct {

     

       446
       446
       +
           Title  string

     

       447
       447
       +
           URL    string

     

       448
       448
       +
           Domain string

     

       449
       449
       +
       }

     

       450
       450
       +
       

     

       451
       451
       +
       func (p *ItemParser) Parse(item *gofeed.Item) (*KagiStory, error) {

     

       452
       452
       +
           doc, err := goquery.NewDocumentFromReader(strings.NewReader(item.Description))

     

       453
       453
       +
           if err != nil {

     

       454
       454
       +
               return nil, err

     

       455
       455
       +
           }

     

       456
       456
       +
       

     

       457
       457
       +
           story := &KagiStory{

     

       458
       458
       +
               Title:      item.Title,

     

       459
       459
       +
               Link:       item.Link,

     

       460
       460
       +
               GUID:       item.GUID,

     

       461
       461
       +
               PubDate:    *item.PublishedParsed,

     

       462
       462
       +
               Categories: item.Categories,

     

       463
       463
       +
           }

     

       464
       464
       +
       

     

       465
       465
       +
           // Extract summary (first <p> tag)

     

       466
       466
       +
           story.Summary = doc.Find("p").First().Text()

     

       467
       467
       +
       

     

       468
       468
       +
           // Extract highlights

     

       469
       469
       +
           doc.Find("h3:contains('Highlights')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       470
       470
       +
               story.Highlights = append(story.Highlights, s.Text())

     

       471
       471
       +
           })

     

       472
       472
       +
       

     

       473
       473
       +
           // Extract perspectives

     

       474
       474
       +
           doc.Find("h3:contains('Perspectives')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       475
       475
       +
               text := s.Text()

     

       476
       476
       +
               link := s.Find("a").First()

     

       477
       477
       +
               sourceURL, _ := link.Attr("href")

     

       478
       478
       +
       

     

       479
       479
       +
               // Parse format: "Actor: Description (Source)"

     

       480
       480
       +
               parts := strings.SplitN(text, ":", 2)

     

       481
       481
       +
               if len(parts) == 2 {

     

       482
       482
       +
                   story.Perspectives = append(story.Perspectives, Perspective{

     

       483
       483
       +
                       Actor:       strings.TrimSpace(parts[0]),

     

       484
       484
       +
                       Description: strings.TrimSpace(parts[1]),

     

       485
       485
       +
                       SourceURL:   sourceURL,

     

       486
       486
       +
                   })

     

       487
       487
       +
               }

     

       488
       488
       +
           })

     

       489
       489
       +
       

     

       490
       490
       +
           // Extract quote

     

       491
       491
       +
           doc.Find("blockquote").Each(func(i int, s *goquery.Selection) {

     

       492
       492
       +
               text := s.Text()

     

       493
       493
       +
               parts := strings.Split(text, " - ")

     

       494
       494
       +
               if len(parts) == 2 {

     

       495
       495
       +
                   story.Quote = &Quote{

     

       496
       496
       +
                       Text:        strings.TrimSpace(parts[0]),

     

       497
       497
       +
                       Attribution: strings.TrimSpace(parts[1]),

     

       498
       498
       +
                   }

     

       499
       499
       +
               }

     

       500
       500
       +
           })

     

       501
       501
       +
       

     

       502
       502
       +
           // Extract sources

     

       503
       503
       +
           doc.Find("h3:contains('Sources')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       504
       504
       +
               link := s.Find("a").First()

     

       505
       505
       +
               url, _ := link.Attr("href")

     

       506
       506
       +
               title := link.Text()

     

       507
       507
       +
               domain := extractDomain(s.Text())

     

       508
       508
       +
       

     

       509
       509
       +
               story.Sources = append(story.Sources, Source{

     

       510
       510
       +
                   Title:  title,

     

       511
       511
       +
                   URL:    url,

     

       512
       512
       +
                   Domain: domain,

     

       513
       513
       +
               })

     

       514
       514
       +
           })

     

       515
       515
       +
       

     

       516
       516
       +
           // Extract image

     

       517
       517
       +
           img := doc.Find("img").First()

     

       518
       518
       +
           if img.Length() > 0 {

     

       519
       519
       +
               story.ImageURL, _ = img.Attr("src")

     

       520
       520
       +
               story.ImageAlt, _ = img.Attr("alt")

     

       521
       521
       +
           }

     

       522
       522
       +
       

     

       523
       523
       +
           return story, nil

     

       524
       524
       +
       }

     

       525
       525
       +
       ```

     

       526
       526
       +
       

     

       527
       527
       +
       **Libraries:**

     

       528
       528
       +
       - `github.com/PuerkitoBio/goquery` - HTML parsing

     

       529
       529
       +
       

     

       530
       530
       +
       ---

     

       531
       531
       +
       

     

       532
       532
       +
       ### Component 3: Deduplication

     

       533
       533
       +
       

     

       534
       534
       +
       **Responsibility:** Track posted stories to prevent duplicates

     

       535
       535
       +
       

     

       536
       536
       +
       ```go

     

       537
       537
       +
       type Deduplicator struct {

     

       538
       538
       +
           db *sql.DB

     

       539
       539
       +
       }

     

       540
       540
       +
       

     

       541
       541
       +
       func (d *Deduplicator) AlreadyPosted(guid string) (bool, error) {

     

       542
       542
       +
           var exists bool

     

       543
       543
       +
           err := d.db.QueryRow(`

     

       544
       544
       +
               SELECT EXISTS(

     

       545
       545
       +
                   SELECT 1 FROM kagi_news_posted_stories

     

       546
       546
       +
                   WHERE guid = $1

     

       547
       547
       +
               )

     

       548
       548
       +
           `, guid).Scan(&exists)

     

       549
       549
       +
           return exists, err

     

       550
       550
       +
       }

     

       551
       551
       +
       

     

       552
       552
       +
       func (d *Deduplicator) MarkPosted(guid, postURI string) error {

     

       553
       553
       +
           _, err := d.db.Exec(`

     

       554
       554
       +
               INSERT INTO kagi_news_posted_stories (guid, post_uri, posted_at)

     

       555
       555
       +
               VALUES ($1, $2, NOW())

     

       556
       556
       +
               ON CONFLICT (guid) DO NOTHING

     

       557
       557
       +
           `, guid, postURI)

     

       558
       558
       +
           return err

     

       559
       559
       +
       }

     

       560
       560
       +
       ```

     

       561
       561
       +
       

     

       562
       562
       +
       **Database Table:**

     

       563
       563
       +
       ```sql

     

       564
       564
       +
       CREATE TABLE kagi_news_posted_stories (

     

       565
       565
       +
           guid TEXT PRIMARY KEY,

     

       566
       566
       +
           post_uri TEXT NOT NULL,

     

       567
       567
       +
           posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW()

     

       568
       568
       +
       );

     

       569
       569
       +
       

     

       570
       570
       +
       CREATE INDEX idx_kagi_posted_at ON kagi_news_posted_stories(posted_at DESC);

     

       571
       571
       +
       ```

     

       572
       572
       +
       

     

       573
       573
       +
       **Cleanup:** Periodic job deletes rows older than 30 days (Kagi unlikely to re-post old stories).

     

       574
       574
       +
       

     

       575
       575
       +
       ---

     

       576
       576
       +
       

     

       577
       577
       +
       ### Component 4: Category Mapper

     

       578
       578
       +
       

     

       579
       579
       +
       **Responsibility:** Map Kagi categories to authorized communities

     

       580
       580
       +
       

     

       581
       581
       +
       ```go

     

       582
       582
       +
       func (m *CategoryMapper) GetTargetCommunities(story *KagiStory) ([]*CommunityAuth, error) {

     

       583
       583
       +
           // Get all communities that have authorized this aggregator

     

       584
       584
       +
           allAuths, err := m.aggregator.GetAuthorizedCommunities(context.Background())

     

       585
       585
       +
           if err != nil {

     

       586
       586
       +
               return nil, err

     

       587
       587
       +
           }

     

       588
       588
       +
       

     

       589
       589
       +
           var targets []*CommunityAuth

     

       590
       590
       +
           for _, auth := range allAuths {

     

       591
       591
       +
               if !auth.Enabled {

     

       592
       592
       +
                   continue

     

       593
       593
       +
               }

     

       594
       594
       +
       

     

       595
       595
       +
               config := auth.Config

     

       596
       596
       +
       

     

       597
       597
       +
               // Check if story's primary category is in config.categories

     

       598
       598
       +
               primaryCategory := story.Categories[0]

     

       599
       599
       +
               if !contains(config["categories"], primaryCategory) {

     

       600
       600
       +
                   continue

     

       601
       601
       +
               }

     

       602
       602
       +
       

     

       603
       603
       +
               // Check subcategory filter (if specified)

     

       604
       604
       +
               if subcatFilter, ok := config["subcategoryFilter"].([]string); ok && len(subcatFilter) > 0 {

     

       605
       605
       +
                   if !hasAnySubcategory(story.Categories, subcatFilter) {

     

       606
       606
       +
                       continue

     

       607
       607
       +
                   }

     

       608
       608
       +
               }

     

       609
       609
       +
       

     

       610
       610
       +
               // Check minimum sources requirement

     

       611
       611
       +
               minSources := config["minSources"].(int)

     

       612
       612
       +
               if len(story.Sources) < minSources {

     

       613
       613
       +
                   continue

     

       614
       614
       +
               }

     

       615
       615
       +
       

     

       616
       616
       +
               targets = append(targets, auth)

     

       617
       617
       +
           }

     

       618
       618
       +
       

     

       619
       619
       +
           return targets, nil

     

       620
       620
       +
       }

     

       621
       621
       +
       ```

     

       622
       622
       +
       

     

       623
       623
       +
       ---

     

       624
       624
       +
       

     

       625
       625
       +
       ### Component 5: Post Formatter

     

       626
       626
       +
       

     

       627
       627
       +
       **Responsibility:** Convert Kagi story to Coves post format

     

       628
       628
       +
       

     

       629
       629
       +
       ```go

     

       630
       630
       +
       func (f *PostFormatter) Format(story *KagiStory, format string) string {

     

       631
       631
       +
           switch format {

     

       632
       632
       +
           case "full":

     

       633
       633
       +
               return f.formatFull(story)

     

       634
       634
       +
           case "summary":

     

       635
       635
       +
               return f.formatSummary(story)

     

       636
       636
       +
           case "minimal":

     

       637
       637
       +
               return f.formatMinimal(story)

     

       638
       638
       +
           default:

     

       639
       639
       +
               return f.formatFull(story)

     

       640
       640
       +
           }

     

       641
       641
       +
       }

     

       642
       642
       +
       

     

       643
       643
       +
       func (f *PostFormatter) formatFull(story *KagiStory) string {

     

       644
       644
       +
           var buf strings.Builder

     

       645
       645
       +
       

     

       646
       646
       +
           // Summary

     

       647
       647
       +
           buf.WriteString(story.Summary)

     

       648
       648
       +
           buf.WriteString("\n\n")

     

       649
       649
       +
       

     

       650
       650
       +
           // Highlights

     

       651
       651
       +
           if len(story.Highlights) > 0 {

     

       652
       652
       +
               buf.WriteString("**Highlights:**\n")

     

       653
       653
       +
               for _, h := range story.Highlights {

     

       654
       654
       +
                   buf.WriteString(fmt.Sprintf("• %s\n", h))

     

       655
       655
       +
               }

     

       656
       656
       +
               buf.WriteString("\n")

     

       657
       657
       +
           }

     

       658
       658
       +
       

     

       659
       659
       +
           // Perspectives

     

       660
       660
       +
           if len(story.Perspectives) > 0 {

     

       661
       661
       +
               buf.WriteString("**Perspectives:**\n")

     

       662
       662
       +
               for _, p := range story.Perspectives {

     

       663
       663
       +
                   buf.WriteString(fmt.Sprintf("• **%s**: %s ([Source](%s))\n", p.Actor, p.Description, p.SourceURL))

     

       664
       664
       +
               }

     

       665
       665
       +
               buf.WriteString("\n")

     

       666
       666
       +
           }

     

       667
       667
       +
       

     

       668
       668
       +
           // Quote

     

       669
       669
       +
           if story.Quote != nil {

     

       670
       670
       +
               buf.WriteString(fmt.Sprintf("> %s — %s\n\n", story.Quote.Text, story.Quote.Attribution))

     

       671
       671
       +
           }

     

       672
       672
       +
       

     

       673
       673
       +
           // Sources

     

       674
       674
       +
           buf.WriteString("**Sources:**\n")

     

       675
       675
       +
           for _, s := range story.Sources {

     

       676
       676
       +
               buf.WriteString(fmt.Sprintf("• [%s](%s) - %s\n", s.Title, s.URL, s.Domain))

     

       677
       677
       +
           }

     

       678
       678
       +
           buf.WriteString("\n")

     

       679
       679
       +
       

     

       680
       680
       +
           // Attribution

     

       681
       681
       +
           buf.WriteString(fmt.Sprintf("---\n📰 Story aggregated by [Kagi News](%s)", story.Link))

     

       682
       682
       +
       

     

       683
       683
       +
           return buf.String()

     

       684
       684
       +
       }

     

       685
       685
       +
       

     

       686
       686
       +
       func (f *PostFormatter) formatSummary(story *KagiStory) string {

     

       687
       687
       +
           var buf strings.Builder

     

       688
       688
       +
       

     

       689
       689
       +
           buf.WriteString(story.Summary)

     

       690
       690
       +
           buf.WriteString("\n\n**Sources:**\n")

     

       691
       691
       +
           for _, s := range story.Sources {

     

       692
       692
       +
               buf.WriteString(fmt.Sprintf("• [%s](%s) - %s\n", s.Title, s.URL, s.Domain))

     

       693
       693
       +
           }

     

       694
       694
       +
           buf.WriteString("\n")

     

       695
       695
       +
           buf.WriteString(fmt.Sprintf("---\n📰 Story aggregated by [Kagi News](%s)", story.Link))

     

       696
       696
       +
       

     

       697
       697
       +
           return buf.String()

     

       698
       698
       +
       }

     

       699
       699
       +
       

     

       700
       700
       +
       func (f *PostFormatter) formatMinimal(story *KagiStory) string {

     

       701
       701
       +
           sourceDomains := make([]string, len(story.Sources))

     

       702
       702
       +
           for i, s := range story.Sources {

     

       703
       703
       +
               sourceDomains[i] = s.Domain

     

       704
       704
       +
           }

     

       705
       705
       +
       

     

       706
       706
       +
           return fmt.Sprintf(

     

       707
       707
       +
               "%s\n\nRead more: %s\n\n**Sources:** %s\n\n---\n📰 Via [Kagi News](%s)",

     

       708
       708
       +
               story.Title,

     

       709
       709
       +
               story.Link,

     

       710
       710
       +
               strings.Join(sourceDomains, ", "),

     

       711
       711
       +
               story.Link,

     

       712
       712
       +
           )

     

       713
       713
       +
       }

     

       714
       714
       +
       ```

     

       715
       715
       +
       

     

       716
       716
       +
       ---

     

       717
       717
       +
       

     

       718
       718
       +
       ### Component 6: Post Publisher

     

       719
       719
       +
       

     

       720
       720
       +
       **Responsibility:** Create posts via Coves API

     

       721
       721
       +
       

     

       722
       722
       +
       ```go

     

       723
       723
       +
       func (p *PostPublisher) PublishStory(ctx context.Context, story *KagiStory, communities []*CommunityAuth) error {

     

       724
       724
       +
           for _, comm := range communities {

     

       725
       725
       +
               config := comm.Config

     

       726
       726
       +
       

     

       727
       727
       +
               // Format content based on config

     

       728
       728
       +
               postFormat := config["postFormat"].(string)

     

       729
       729
       +
               content := p.formatter.Format(story, postFormat)

     

       730
       730
       +
       

     

       731
       731
       +
               // Build embed

     

       732
       732
       +
               var embed *aggregator.Embed

     

       733
       733
       +
               if config["includeImages"].(bool) && story.ImageURL != "" {

     

       734
       734
       +
                   // TODO: Handle image upload/blob creation

     

       735
       735
       +
                   embed = &aggregator.Embed{

     

       736
       736
       +
                       Type: "app.bsky.embed.external",

     

       737
       737
       +
                       External: &aggregator.External{

     

       738
       738
       +
                           URI:         story.Link,

     

       739
       739
       +
                           Title:       story.Title,

     

       740
       740
       +
                           Description: truncate(story.Summary, 300),

     

       741
       741
       +
                           Thumb:       story.ImageURL, // or blob reference

     

       742
       742
       +
                       },

     

       743
       743
       +
                   }

     

       744
       744
       +
               }

     

       745
       745
       +
       

     

       746
       746
       +
               // Create post

     

       747
       747
       +
               post := aggregator.Post{

     

       748
       748
       +
                   Title:   story.Title,

     

       749
       749
       +
                   Content: content,

     

       750
       750
       +
                   Embed:   embed,

     

       751
       751
       +
                   FederatedFrom: &aggregator.FederatedSource{

     

       752
       752
       +
                       Platform:          "kagi-news-rss",

     

       753
       753
       +
                       URI:               story.Link,

     

       754
       754
       +
                       ID:                story.GUID,

     

       755
       755
       +
                       OriginalCreatedAt: story.PubDate,

     

       756
       756
       +
                   },

     

       757
       757
       +
                   ContentLabels: story.Categories,

     

       758
       758
       +
               }

     

       759
       759
       +
       

     

       760
       760
       +
               err := p.aggregator.CreatePost(ctx, comm.CommunityDID, post)

     

       761
       761
       +
               if err != nil {

     

       762
       762
       +
                   log.Printf("Failed to create post in %s: %v", comm.CommunityDID, err)

     

       763
       763
       +
                   continue

     

       764
       764
       +
               }

     

       765
       765
       +
       

     

       766
       766
       +
               // Mark as posted

     

       767
       767
       +
               _ = p.deduplicator.MarkPosted(story.GUID, "post-uri-from-response")

     

       768
       768
       +
           }

     

       769
       769
       +
       

     

       770
       770
       +
           return nil

     

       771
       771
       +
       }

     

       772
       772
       +
       ```

     

       773
       773
       +
       

     

       774
       774
       +
       ---

     

       775
       775
       +
       

     

       776
       776
       +
       ## Image Handling Strategy

     

       777
       777
       +
       

     

       778
       778
       +
       ### Initial Implementation (MVP)

     

       779
       779
       +
       

     

       780
       780
       +
       **Approach:** Use Kagi proxy URLs directly in embeds

     

       781
       781
       +
       

     

       782
       782
       +
       **Rationale:**

     

       783
       783
       +
       - Simplest implementation

     

       784
       784
       +
       - Kagi proxy likely allows hotlinking for non-commercial use

     

       785
       785
       +
       - No storage costs

     

       786
       786
       +
       - Images are already optimized by Kagi

     

       787
       787
       +
       

     

       788
       788
       +
       **Risk Mitigation:**

     

       789
       789
       +
       - Monitor for broken images

     

       790
       790
       +
       - Add fallback: if image fails to load, skip embed

     

       791
       791
       +
       - Prepare migration plan to self-hosting if needed

     

       792
       792
       +
       

     

       793
       793
       +
       **Code:**

     

       794
       794
       +
       ```go

     

       795
       795
       +
       if config["includeImages"].(bool) && story.ImageURL != "" {

     

       796
       796
       +
           // Use Kagi proxy URL directly

     

       797
       797
       +
           embed = &aggregator.Embed{

     

       798
       798
       +
               External: &aggregator.External{

     

       799
       799
       +
                   Thumb: story.ImageURL, // https://kagiproxy.com/img/...

     

       800
       800
       +
               },

     

       801
       801
       +
           }

     

       802
       802
       +
       }

     

       803
       803
       +
       ```

     

       804
       804
       +
       

     

       805
       805
       +
       ---

     

       806
       806
       +
       

     

       807
       807
       +
       ### Future Enhancement (If Issues Arise)

     

       808
       808
       +
       

     

       809
       809
       +
       **Approach:** Download and re-host images

     

       810
       810
       +
       

     

       811
       811
       +
       **Implementation:**

     

       812
       812
       +
       1. Download image from Kagi proxy

     

       813
       813
       +
       2. Upload to Coves blob storage (or S3/CDN)

     

       814
       814
       +
       3. Use blob reference in embed

     

       815
       815
       +
       

     

       816
       816
       +
       **Code:**

     

       817
       817
       +
       ```go

     

       818
       818
       +
       func (p *PostPublisher) uploadImage(imageURL string) (string, error) {

     

       819
       819
       +
           // Download from Kagi proxy

     

       820
       820
       +
           resp, err := http.Get(imageURL)

     

       821
       821
       +
           if err != nil {

     

       822
       822
       +
               return "", err

     

       823
       823
       +
           }

     

       824
       824
       +
           defer resp.Body.Close()

     

       825
       825
       +
       

     

       826
       826
       +
           // Upload to blob storage

     

       827
       827
       +
           blob, err := p.blobStore.Upload(resp.Body, resp.Header.Get("Content-Type"))

     

       828
       828
       +
           if err != nil {

     

       829
       829
       +
               return "", err

     

       830
       830
       +
           }

     

       831
       831
       +
       

     

       832
       832
       +
           return blob.Ref, nil

     

       833
       833
       +
       }

     

       834
       834
       +
       ```

     

       835
       835
       +
       

     

       836
       836
       +
       **Decision Point:** Only implement if:

     

       837
       837
       +
       - Kagi blocks hotlinking

     

       838
       838
       +
       - Kagi proxy becomes unreliable

     

       839
       839
       +
       - Legal clarification needed

     

       840
       840
       +
       

     

       841
       841
       +
       ---

     

       842
       842
       +
       

     

       843
       843
       +
       ## Rate Limiting & Performance

     

       844
       844
       +
       

     

       845
       845
       +
       ### Rate Limits

     

       846
       846
       +
       

     

       847
       847
       +
       **RSS Fetching:**

     

       848
       848
       +
       - Poll each category feed every 15 minutes

     

       849
       849
       +
       - Max 4 categories = 4 requests per 15 min = 16 req/hour

     

       850
       850
       +
       - Well within any reasonable limit

     

       851
       851
       +
       

     

       852
       852
       +
       **Post Creation:**

     

       853
       853
       +
       - Aggregator rate limit: 10 posts/hour per community

     

       854
       854
       +
       - Global limit: 100 posts/hour across all communities

     

       855
       855
       +
       - Kagi News publishes ~5-10 stories per category per day

     

       856
       856
       +
       - = ~20-40 posts/day total across all categories

     

       857
       857
       +
       - = ~2-4 posts/hour average

     

       858
       858
       +
       - Well within limits

     

       859
       859
       +
       

     

       860
       860
       +
       **Performance Targets:**

     

       861
       861
       +
       - Story posted within 15 minutes of appearing in RSS feed

     

       862
       862
       +
       - < 1 second to parse and format a story

     

       863
       863
       +
       - < 500ms to publish a post via API

     

       864
       864
       +
       

     

       865
       865
       +
       ---

     

       866
       866
       +
       

     

       867
       867
       +
       ## Monitoring & Observability

     

       868
       868
       +
       

     

       869
       869
       +
       ### Metrics to Track

     

       870
       870
       +
       

     

       871
       871
       +
       **Feed Polling:**

     

       872
       872
       +
       - `kagi_feed_poll_total` (counter) - Total feed polls by category

     

       873
       873
       +
       - `kagi_feed_poll_errors` (counter) - Failed polls by category/error

     

       874
       874
       +
       - `kagi_feed_items_fetched` (gauge) - Items per poll by category

     

       875
       875
       +
       - `kagi_feed_poll_duration_seconds` (histogram) - Poll latency

     

       876
       876
       +
       

     

       877
       877
       +
       **Story Processing:**

     

       878
       878
       +
       - `kagi_stories_parsed_total` (counter) - Successfully parsed stories

     

       879
       879
       +
       - `kagi_stories_parse_errors` (counter) - Parse failures by error type

     

       880
       880
       +
       - `kagi_stories_filtered` (counter) - Stories filtered out by reason (duplicate, min sources, category)

     

       881
       881
       +
       - `kagi_stories_posted` (counter) - Stories successfully posted by community

     

       882
       882
       +
       

     

       883
       883
       +
       **Post Publishing:**

     

       884
       884
       +
       - `kagi_posts_created_total` (counter) - Total posts created

     

       885
       885
       +
       - `kagi_posts_failed` (counter) - Failed posts by error type

     

       886
       886
       +
       - `kagi_post_publish_duration_seconds` (histogram) - Post creation latency

     

       887
       887
       +
       

     

       888
       888
       +
       **Health:**

     

       889
       889
       +
       - `kagi_aggregator_up` (gauge) - Service health (1 = healthy, 0 = down)

     

       890
       890
       +
       - `kagi_last_successful_poll_timestamp` (gauge) - Last successful poll time by category

     

       891
       891
       +
       

     

       892
       892
       +
       ---

     

       893
       893
       +
       

     

       894
       894
       +
       ### Logging

     

       895
       895
       +
       

     

       896
       896
       +
       **Structured Logging:**

     

       897
       897
       +
       ```go

     

       898
       898
       +
       log.Info("Story posted",

     

       899
       899
       +
           "guid", story.GUID,

     

       900
       900
       +
           "title", story.Title,

     

       901
       901
       +
           "community", comm.CommunityDID,

     

       902
       902
       +
           "post_uri", postURI,

     

       903
       903
       +
           "sources", len(story.Sources),

     

       904
       904
       +
           "format", postFormat,

     

       905
       905
       +
       )

     

       906
       906
       +
       

     

       907
       907
       +
       log.Error("Failed to parse story",

     

       908
       908
       +
           "guid", item.GUID,

     

       909
       909
       +
           "feed", feedURL,

     

       910
       910
       +
           "error", err,

     

       911
       911
       +
       )

     

       912
       912
       +
       ```

     

       913
       913
       +
       

     

       914
       914
       +
       **Log Levels:**

     

       915
       915
       +
       - DEBUG: Feed items, parsing details

     

       916
       916
       +
       - INFO: Stories posted, communities targeted

     

       917
       917
       +
       - WARN: Parse errors, rate limit approaching

     

       918
       918
       +
       - ERROR: Failed posts, feed fetch failures

     

       919
       919
       +
       

     

       920
       920
       +
       ---

     

       921
       921
       +
       

     

       922
       922
       +
       ### Alerts

     

       923
       923
       +
       

     

       924
       924
       +
       **Critical:**

     

       925
       925
       +
       - Feed polling failing for > 1 hour

     

       926
       926
       +
       - Post creation failing for > 10 consecutive attempts

     

       927
       927
       +
       - Aggregator unauthorized (auth record disabled/deleted)

     

       928
       928
       +
       

     

       929
       929
       +
       **Warning:**

     

       930
       930
       +
       - Post creation rate < 50% of expected

     

       931
       931
       +
       - Parse errors > 10% of items

     

       932
       932
       +
       - Approaching rate limits (> 80% of quota)

     

       933
       933
       +
       

     

       934
       934
       +
       ---

     

       935
       935
       +
       

     

       936
       936
       +
       ## Deployment

     

       937
       937
       +
       

     

       938
       938
       +
       ### Infrastructure

     

       939
       939
       +
       

     

       940
       940
       +
       **Service Type:** Long-running daemon

     

       941
       941
       +
       

     

       942
       942
       +
       **Hosting:** Kubernetes (same cluster as Coves AppView)

     

       943
       943
       +
       

     

       944
       944
       +
       **Resources:**

     

       945
       945
       +
       - CPU: 0.5 cores (low CPU usage, mostly I/O)

     

       946
       946
       +
       - Memory: 512 MB (small in-memory cache for recent GUIDs)

     

       947
       947
       +
       - Storage: 1 GB (SQLite for deduplication tracking)

     

       948
       948
       +
       

     

       949
       949
       +
       ---

     

       950
       950
       +
       

     

       951
       951
       +
       ### Configuration

     

       952
       952
       +
       

     

       953
       953
       +
       **Environment Variables:**

     

       954
       954
       +
       ```bash

     

       955
       955
       +
       # Aggregator identity

     

       956
       956
       +
       AGGREGATOR_DID=did:web:kagi-news.coves.social

     

       957
       957
       +
       AGGREGATOR_PRIVATE_KEY_PATH=/secrets/private-key.pem

     

       958
       958
       +
       

     

       959
       959
       +
       # Coves API

     

       960
       960
       +
       COVES_API_URL=https://api.coves.social

     

       961
       961
       +
       

     

       962
       962
       +
       # Feed polling

     

       963
       963
       +
       POLL_INTERVAL=15m

     

       964
       964
       +
       CATEGORIES=world,tech,business,sports

     

       965
       965
       +
       

     

       966
       966
       +
       # Database (for deduplication)

     

       967
       967
       +
       DB_PATH=/data/kagi-news.db

     

       968
       968
       +
       

     

       969
       969
       +
       # Monitoring

     

       970
       970
       +
       METRICS_PORT=9090

     

       971
       971
       +
       LOG_LEVEL=info

     

       972
       972
       +
       ```

     

       973
       973
       +
       

     

       974
       974
       +
       ---

     

       975
       975
       +
       

     

       976
       976
       +
       ### Deployment Manifest

     

       977
       977
       +
       

     

       978
       978
       +
       ```yaml

     

       979
       979
       +
       apiVersion: apps/v1

     

       980
       980
       +
       kind: Deployment

     

       981
       981
       +
       metadata:

     

       982
       982
       +
         name: kagi-news-aggregator

     

       983
       983
       +
         namespace: coves

     

       984
       984
       +
       spec:

     

       985
       985
       +
         replicas: 1

     

       986
       986
       +
         selector:

     

       987
       987
       +
           matchLabels:

     

       988
       988
       +
             app: kagi-news-aggregator

     

       989
       989
       +
         template:

     

       990
       990
       +
           metadata:

     

       991
       991
       +
             labels:

     

       992
       992
       +
               app: kagi-news-aggregator

     

       993
       993
       +
           spec:

     

       994
       994
       +
             containers:

     

       995
       995
       +
             - name: aggregator

     

       996
       996
       +
               image: coves/kagi-news-aggregator:latest

     

       997
       997
       +
               env:

     

       998
       998
       +
               - name: AGGREGATOR_DID

     

       999
       999
       +
                 value: did:web:kagi-news.coves.social

     

       1000
       1000
       +
               - name: COVES_API_URL

     

       1001
       1001
       +
                 value: https://api.coves.social

     

       1002
       1002
       +
               - name: POLL_INTERVAL

     

       1003
       1003
       +
                 value: 15m

     

       1004
       1004
       +
               - name: CATEGORIES

     

       1005
       1005
       +
                 value: world,tech,business,sports

     

       1006
       1006
       +
               - name: DB_PATH

     

       1007
       1007
       +
                 value: /data/kagi-news.db

     

       1008
       1008
       +
               - name: AGGREGATOR_PRIVATE_KEY_PATH

     

       1009
       1009
       +
                 value: /secrets/private-key.pem

     

       1010
       1010
       +
               volumeMounts:

     

       1011
       1011
       +
               - name: data

     

       1012
       1012
       +
                 mountPath: /data

     

       1013
       1013
       +
               - name: secrets

     

       1014
       1014
       +
                 mountPath: /secrets

     

       1015
       1015
       +
                 readOnly: true

     

       1016
       1016
       +
               ports:

     

       1017
       1017
       +
               - name: metrics

     

       1018
       1018
       +
                 containerPort: 9090

     

       1019
       1019
       +
               resources:

     

       1020
       1020
       +
                 requests:

     

       1021
       1021
       +
                   cpu: 250m

     

       1022
       1022
       +
                   memory: 256Mi

     

       1023
       1023
       +
                 limits:

     

       1024
       1024
       +
                   cpu: 500m

     

       1025
       1025
       +
                   memory: 512Mi

     

       1026
       1026
       +
             volumes:

     

       1027
       1027
       +
             - name: data

     

       1028
       1028
       +
               persistentVolumeClaim:

     

       1029
       1029
       +
                 claimName: kagi-news-data

     

       1030
       1030
       +
             - name: secrets

     

       1031
       1031
       +
               secret:

     

       1032
       1032
       +
                 secretName: kagi-news-private-key

     

       1033
       1033
       +
       ```

     

       1034
       1034
       +
       

     

       1035
       1035
       +
       ---

     

       1036
       1036
       +
       

     

       1037
       1037
       +
       ## Testing Strategy

     

       1038
       1038
       +
       

     

       1039
       1039
       +
       ### Unit Tests

     

       1040
       1040
       +
       

     

       1041
       1041
       +
       **Feed Parsing:**

     

       1042
       1042
       +
       ```go

     

       1043
       1043
       +
       func TestParseFeed(t *testing.T) {

     

       1044
       1044
       +
           feed := loadTestFeed("testdata/world.xml")

     

       1045
       1045
       +
           stories, err := parser.Parse(feed)

     

       1046
       1046
       +
           assert.NoError(t, err)

     

       1047
       1047
       +
           assert.Len(t, stories, 10)

     

       1048
       1048
       +
       

     

       1049
       1049
       +
           story := stories[0]

     

       1050
       1050
       +
           assert.NotEmpty(t, story.Title)

     

       1051
       1051
       +
           assert.NotEmpty(t, story.Summary)

     

       1052
       1052
       +
           assert.Greater(t, len(story.Sources), 1)

     

       1053
       1053
       +
       }

     

       1054
       1054
       +
       

     

       1055
       1055
       +
       func TestParseStoryHTML(t *testing.T) {

     

       1056
       1056
       +
           html := `<p>Summary [source.com#1]</p>

     

       1057
       1057
       +
                    <h3>Highlights:</h3>

     

       1058
       1058
       +
                    <ul><li>Point 1</li></ul>

     

       1059
       1059
       +
                    <h3>Sources:</h3>

     

       1060
       1060
       +
                    <ul><li><a href="https://example.com">Title</a> - example.com</li></ul>`

     

       1061
       1061
       +
       

     

       1062
       1062
       +
           story, err := parser.ParseHTML(html)

     

       1063
       1063
       +
           assert.NoError(t, err)

     

       1064
       1064
       +
           assert.Equal(t, "Summary [source.com#1]", story.Summary)

     

       1065
       1065
       +
           assert.Len(t, story.Highlights, 1)

     

       1066
       1066
       +
           assert.Len(t, story.Sources, 1)

     

       1067
       1067
       +
       }

     

       1068
       1068
       +
       ```

     

       1069
       1069
       +
       

     

       1070
       1070
       +
       **Formatting:**

     

       1071
       1071
       +
       ```go

     

       1072
       1072
       +
       func TestFormatFull(t *testing.T) {

     

       1073
       1073
       +
           story := &KagiStory{

     

       1074
       1074
       +
               Summary: "Test summary",

     

       1075
       1075
       +
               Sources: []Source{

     

       1076
       1076
       +
                   {Title: "Article", URL: "https://example.com", Domain: "example.com"},

     

       1077
       1077
       +
               },

     

       1078
       1078
       +
           }

     

       1079
       1079
       +
       

     

       1080
       1080
       +
           content := formatter.Format(story, "full")

     

       1081
       1081
       +
           assert.Contains(t, content, "Test summary")

     

       1082
       1082
       +
           assert.Contains(t, content, "**Sources:**")

     

       1083
       1083
       +
           assert.Contains(t, content, "📰 Story aggregated by")

     

       1084
       1084
       +
       }

     

       1085
       1085
       +
       ```

     

       1086
       1086
       +
       

     

       1087
       1087
       +
       **Deduplication:**

     

       1088
       1088
       +
       ```go

     

       1089
       1089
       +
       func TestDeduplication(t *testing.T) {

     

       1090
       1090
       +
           guid := "test-guid-123"

     

       1091
       1091
       +
       

     

       1092
       1092
       +
           posted, err := deduplicator.AlreadyPosted(guid)

     

       1093
       1093
       +
           assert.NoError(t, err)

     

       1094
       1094
       +
           assert.False(t, posted)

     

       1095
       1095
       +
       

     

       1096
       1096
       +
           err = deduplicator.MarkPosted(guid, "at://...")

     

       1097
       1097
       +
           assert.NoError(t, err)

     

       1098
       1098
       +
       

     

       1099
       1099
       +
           posted, err = deduplicator.AlreadyPosted(guid)

     

       1100
       1100
       +
           assert.NoError(t, err)

     

       1101
       1101
       +
           assert.True(t, posted)

     

       1102
       1102
       +
       }

     

       1103
       1103
       +
       ```

     

       1104
       1104
       +
       

     

       1105
       1105
       +
       ---

     

       1106
       1106
       +
       

     

       1107
       1107
       +
       ### Integration Tests

     

       1108
       1108
       +
       

     

       1109
       1109
       +
       **With Mock Coves API:**

     

       1110
       1110
       +
       ```go

     

       1111
       1111
       +
       func TestPublishStory(t *testing.T) {

     

       1112
       1112
       +
           // Setup mock Coves API

     

       1113
       1113
       +
           mockAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

     

       1114
       1114
       +
               assert.Equal(t, "/xrpc/social.coves.post.create", r.URL.Path)

     

       1115
       1115
       +
       

     

       1116
       1116
       +
               var input CreatePostInput

     

       1117
       1117
       +
               json.NewDecoder(r.Body).Decode(&input)

     

       1118
       1118
       +
       

     

       1119
       1119
       +
               assert.Equal(t, "did:plc:test-community", input.Community)

     

       1120
       1120
       +
               assert.NotEmpty(t, input.Title)

     

       1121
       1121
       +
               assert.Contains(t, input.Content, "📰 Story aggregated by")

     

       1122
       1122
       +
       

     

       1123
       1123
       +
               w.WriteHeader(200)

     

       1124
       1124
       +
               json.NewEncoder(w).Encode(CreatePostOutput{URI: "at://..."})

     

       1125
       1125
       +
           }))

     

       1126
       1126
       +
           defer mockAPI.Close()

     

       1127
       1127
       +
       

     

       1128
       1128
       +
           // Test story publishing

     

       1129
       1129
       +
           publisher := NewPostPublisher(mockAPI.URL)

     

       1130
       1130
       +
           err := publisher.PublishStory(ctx, testStory, []*CommunityAuth{testComm})

     

       1131
       1131
       +
           assert.NoError(t, err)

     

       1132
       1132
       +
       }

     

       1133
       1133
       +
       ```

     

       1134
       1134
       +
       

     

       1135
       1135
       +
       ---

     

       1136
       1136
       +
       

     

       1137
       1137
       +
       ### E2E Tests

     

       1138
       1138
       +
       

     

       1139
       1139
       +
       **With Real RSS Feed:**

     

       1140
       1140
       +
       ```go

     

       1141
       1141
       +
       func TestE2E_FetchAndParse(t *testing.T) {

     

       1142
       1142
       +
           if testing.Short() {

     

       1143
       1143
       +
               t.Skip("Skipping E2E test")

     

       1144
       1144
       +
           }

     

       1145
       1145
       +
       

     

       1146
       1146
       +
           // Fetch real Kagi News feed

     

       1147
       1147
       +
           feed, err := poller.fetchFeed("https://news.kagi.com/world.xml")

     

       1148
       1148
       +
           assert.NoError(t, err)

     

       1149
       1149
       +
           assert.NotEmpty(t, feed.Items)

     

       1150
       1150
       +
       

     

       1151
       1151
       +
           // Parse first item

     

       1152
       1152
       +
           story, err := parser.Parse(feed.Items[0])

     

       1153
       1153
       +
           assert.NoError(t, err)

     

       1154
       1154
       +
           assert.NotEmpty(t, story.Title)

     

       1155
       1155
       +
           assert.NotEmpty(t, story.Summary)

     

       1156
       1156
       +
           assert.Greater(t, len(story.Sources), 0)

     

       1157
       1157
       +
       }

     

       1158
       1158
       +
       ```

     

       1159
       1159
       +
       

     

       1160
       1160
       +
       **With Test Coves Instance:**

     

       1161
       1161
       +
       ```go

     

       1162
       1162
       +
       func TestE2E_CreatePost(t *testing.T) {

     

       1163
       1163
       +
           if testing.Short() {

     

       1164
       1164
       +
               t.Skip("Skipping E2E test")

     

       1165
       1165
       +
           }

     

       1166
       1166
       +
       

     

       1167
       1167
       +
           // Create post in test community

     

       1168
       1168
       +
           post := aggregator.Post{

     

       1169
       1169
       +
               Title:   "Test Kagi News Post",

     

       1170
       1170
       +
               Content: "Test content...",

     

       1171
       1171
       +
           }

     

       1172
       1172
       +
       

     

       1173
       1173
       +
           err := aggregator.CreatePost(ctx, testCommunityDID, post)

     

       1174
       1174
       +
           assert.NoError(t, err)

     

       1175
       1175
       +
       

     

       1176
       1176
       +
           // Verify post appears in feed

     

       1177
       1177
       +
           // (requires test community setup)

     

       1178
       1178
       +
       }

     

       1179
       1179
       +
       ```

     

       1180
       1180
       +
       

     

       1181
       1181
       +
       ---

     

       1182
       1182
       +
       

     

       1183
       1183
       +
       ## Success Metrics

     

       1184
       1184
       +
       

     

       1185
       1185
       +
       ### Pre-Launch Checklist

     

       1186
       1186
       +
       

     

       1187
       1187
       +
       - [ ] Aggregator service declaration published

     

       1188
       1188
       +
       - [ ] DID created and configured (did:web:kagi-news.coves.social)

     

       1189
       1189
       +
       - [ ] RSS feed parser handles all Kagi HTML structures

     

       1190
       1190
       +
       - [ ] Deduplication prevents duplicate posts

     

       1191
       1191
       +
       - [ ] Category mapping works for all configs

     

       1192
       1192
       +
       - [ ] All 3 post formats render correctly

     

       1193
       1193
       +
       - [ ] Attribution to Kagi News visible on all posts

     

       1194
       1194
       +
       - [ ] Rate limiting prevents spam

     

       1195
       1195
       +
       - [ ] Monitoring/alerting configured

     

       1196
       1196
       +
       - [ ] E2E tests passing against test instance

     

       1197
       1197
       +
       

     

       1198
       1198
       +
       ---

     

       1199
       1199
       +
       

     

       1200
       1200
       +
       ### Alpha Goals (First Week)

     

       1201
       1201
       +
       

     

       1202
       1202
       +
       - [ ] 3+ communities using Kagi News aggregator

     

       1203
       1203
       +
       - [ ] 50+ posts created successfully

     

       1204
       1204
       +
       - [ ] Zero duplicate posts

     

       1205
       1205
       +
       - [ ] < 5% parse errors

     

       1206
       1206
       +
       - [ ] < 1% post creation failures

     

       1207
       1207
       +
       - [ ] Stories posted within 15 minutes of RSS publication

     

       1208
       1208
       +
       

     

       1209
       1209
       +
       ---

     

       1210
       1210
       +
       

     

       1211
       1211
       +
       ### Beta Goals (First Month)

     

       1212
       1212
       +
       

     

       1213
       1213
       +
       - [ ] 10+ communities using aggregator

     

       1214
       1214
       +
       - [ ] 500+ posts created

     

       1215
       1215
       +
       - [ ] Community feedback positive (surveys)

     

       1216
       1216
       +
       - [ ] Attribution compliance verified

     

       1217
       1217
       +
       - [ ] No rate limit violations

     

       1218
       1218
       +
       - [ ] < 1% error rate (parsing + posting)

     

       1219
       1219
       +
       

     

       1220
       1220
       +
       ---

     

       1221
       1221
       +
       

     

       1222
       1222
       +
       ## Future Enhancements

     

       1223
       1223
       +
       

     

       1224
       1224
       +
       ### Phase 2 Features

     

       1225
       1225
       +
       

     

       1226
       1226
       +
       **Smart Category Detection:**

     

       1227
       1227
       +
       - Use LLM to suggest additional categories for stories

     

       1228
       1228
       +
       - Map Kagi categories to community tags automatically

     

       1229
       1229
       +
       

     

       1230
       1230
       +
       **Customizable Templates:**

     

       1231
       1231
       +
       - Allow communities to customize post format with templates

     

       1232
       1232
       +
       - Support Markdown/Handlebars templates in config

     

       1233
       1233
       +
       

     

       1234
       1234
       +
       **Story Scoring:**

     

       1235
       1235
       +
       - Prioritize high-impact stories (many sources, breaking news)

     

       1236
       1236
       +
       - Delay low-priority stories to avoid flooding feed

     

       1237
       1237
       +
       

     

       1238
       1238
       +
       **Cross-posting Prevention:**

     

       1239
       1239
       +
       - Detect when multiple communities authorize same category

     

       1240
       1240
       +
       - Intelligently cross-post vs. duplicate

     

       1241
       1241
       +
       

     

       1242
       1242
       +
       ---

     

       1243
       1243
       +
       

     

       1244
       1244
       +
       ### Phase 3 Features

     

       1245
       1245
       +
       

     

       1246
       1246
       +
       **Interactive Features:**

     

       1247
       1247
       +
       - Bot responds to comments with additional sources

     

       1248
       1248
       +
       - Updates megathread with new sources as story develops

     

       1249
       1249
       +
       

     

       1250
       1250
       +
       **Analytics Dashboard:**

     

       1251
       1251
       +
       - Show communities which stories get most engagement

     

       1252
       1252
       +
       - Trending topics from Kagi News

     

       1253
       1253
       +
       - Source diversity metrics

     

       1254
       1254
       +
       

     

       1255
       1255
       +
       **Federation:**

     

       1256
       1256
       +
       - Support other Coves instances using same aggregator

     

       1257
       1257
       +
       - Shared deduplication across instances

     

       1258
       1258
       +
       

     

       1259
       1259
       +
       ---

     

       1260
       1260
       +
       

     

       1261
       1261
       +
       ## Open Questions

     

       1262
       1262
       +
       

     

       1263
       1263
       +
       ### Need to Resolve Before Launch

     

       1264
       1264
       +
       

     

       1265
       1265
       +
       1. **Image Licensing:**

     

       1266
       1266
       +
          - ❓ Are images from Kagi proxy covered by CC BY-NC?

     

       1267
       1267
       +
          - ❓ Do we need to attribute original image sources?

     

       1268
       1268
       +
          - **Action:** Email support@kagi.com for clarification

     

       1269
       1269
       +
       

     

       1270
       1270
       +
       2. **Hotlinking Policy:**

     

       1271
       1271
       +
          - ❓ Is embedding Kagi proxy images acceptable?

     

       1272
       1272
       +
          - ❓ Should we download and re-host?

     

       1273
       1273
       +
          - **Action:** Test in staging, monitor for issues

     

       1274
       1274
       +
       

     

       1275
       1275
       +
       3. **Category Discovery:**

     

       1276
       1276
       +
          - ❓ How to discover all available category feeds?

     

       1277
       1277
       +
          - ❓ Are there categories beyond world/tech/business/sports?

     

       1278
       1278
       +
          - **Action:** Scrape https://news.kagi.com/ for all .xml links

     

       1279
       1279
       +
       

     

       1280
       1280
       +
       4. **Attribution Format:**

     

       1281
       1281
       +
          - ❓ Is "📰 Story aggregated by Kagi News" sufficient?

     

       1282
       1282
       +
          - ❓ Do we need more prominent attribution?

     

       1283
       1283
       +
          - **Action:** Review CC BY-NC best practices

     

       1284
       1284
       +
       

     

       1285
       1285
       +
       ---

     

       1286
       1286
       +
       

     

       1287
       1287
       +
       ## References

     

       1288
       1288
       +
       

     

       1289
       1289
       +
       - Kagi News About Page: https://news.kagi.com/about

     

       1290
       1290
       +
       - Kagi News RSS Example: https://news.kagi.com/world.xml

     

       1291
       1291
       +
       - Kagi Kite Public Repo: https://github.com/kagisearch/kite-public

     

       1292
       1292
       +
       - CC BY-NC License: https://creativecommons.org/licenses/by-nc/4.0/

     

       1293
       1293
       +
       - Parent PRD: [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       1294
       1294
       +
       - Aggregator SDK: [TBD]