···

       
1
       
 
       
# Kagi News RSS Aggregator PRD

     

       
2
       
 
       


     

       
3
       
-
       
**Status:** Planning Phase

     

       
4
       
 
       
**Owner:** Platform Team

     

       
5
       
-
       
**Last Updated:** 2025-10-20

     

       
6
       
 
       
**Parent PRD:** [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
7
       
 
       


     

       
8
       
 
       
## Overview

     

       
9
       
 
       


     
···

       
15
       
 
       
- **Rich metadata**: Categories, highlights, source links included

     

       
16
       
 
       
- **Legal & free**: CC BY-NC licensed for non-commercial use

     

       
17
       
 
       
- **Low complexity**: No LLM deduplication needed (Kagi does it)

     

       
0
       
 
       

     

       
18
       
 
       


     

       
19
       
 
       
## Data Source: Kagi News RSS Feeds

     

       
20
       
 
       


     
···

       
46
       
 
       


     

       
47
       
 
       
**Known Categories:**

     

       
48
       
 
       
- `world.xml` - World news

     

       
49
       
-
       
- `tech.xml` - Technology (likely)

     

       
50
       
-
       
- `business.xml` - Business (likely)

     

       
51
       
 
       
- `sports.xml` - Sports (likely)

     

       
52
       
 
       
- Additional categories TBD (need to scrape homepage)

     

       
53
       
 
       


     
···

       
55
       
 
       


     

       
56
       
 
       
**Update Frequency:** One daily update (~noon UTC)

     

       
57
       
 
       


     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
58
       
 
       
---

     

       
59
       
 
       


     

       
60
       
 
       
### RSS Item Schema

     
···

       
99
       
 
       
</ul>

     

       
100
       
 
       
```

     

       
101
       
 
       


     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
102
       
 
       
**Key Features:**

     

       
103
       
 
       
- Multiple source citations inline

     

       
104
       
 
       
- Balanced perspectives from different actors

     

       
105
       
-
       
- Highlights extract key points

     

       
106
       
-
       
- Direct quotes preserved

     

       
107
       
 
       
- All sources linked with attribution

     

       
0
       
 
       

     

       
108
       
 
       


     

       
109
       
 
       
---

     

       
110
       
 
       


     
···

       
123
       
 
       
                            │ HTTP GET one job after update

     

       
124
       
 
       
                            ▼

     

       
125
       
 
       
┌─────────────────────────────────────────────────────────────┐

     

       
126
       
-
       
│  Kagi News Aggregator Service                               │

     

       
127
       
-
       
│  DID: did:web:kagi-news.coves.social                        │

     

       
0
       
 
       

     

       
128
       
 
       
│                                                             │

     

       
129
       
 
       
│  Components:                                                 │

     

       
130
       
-
       
│  1. Feed Poller: Fetches RSS feeds on schedule              │

     

       
131
       
-
       
│  2. Item Parser: Extracts structured data from HTML         │

     

       
132
       
-
       
│  3. Deduplication: Tracks posted GUIDs (no LLM needed)      │

     

       
133
       
-
       
│  4. Category Mapper: Maps Kagi categories to communities    │

     

       
134
       
 
       
│  5. Post Formatter: Converts to Coves post format           │

     

       
135
       
-
       
│  6. Post Publisher: Calls social.coves.post.create          │

     

       
0
       
 
       

     

       
136
       
 
       
└─────────────────────────────────────────────────────────────┘

     

       
137
       
 
       
                            │

     

       
138
       
 
       
                            │ Authenticated XRPC calls

     
···

       
140
       
 
       
┌─────────────────────────────────────────────────────────────┐

     

       
141
       
 
       
│  Coves AppView (social.coves.post.create)                   │

     

       
142
       
 
       
│  - Validates aggregator authorization                        │

     

       
143
       
-
       
│  - Creates post with author = did:web:kagi-news.coves.social│

     

       
144
       
 
       
│  - Indexes to community feeds                                │

     

       
145
       
 
       
└─────────────────────────────────────────────────────────────┘

     

       
146
       
 
       
```

     
···

       
152
       
 
       
```json

     

       
153
       
 
       
{

     

       
154
       
 
       
  "$type": "social.coves.aggregator.service",

     

       
155
       
-
       
  "did": "did:web:kagi-news.coves.social",

     

       
156
       
 
       
  "displayName": "Kagi News Aggregator",

     

       
157
       
 
       
  "description": "Automatically posts breaking news from Kagi News RSS feeds. Kagi News aggregates multiple sources per story with balanced perspectives and comprehensive source citations.",

     

       
158
       
 
       
  "aggregatorType": "social.coves.aggregator.types#rss",

     
···

       
160
       
 
       
  "configSchema": {

     

       
161
       
 
       
    "type": "object",

     

       
162
       
 
       
    "properties": {

     

       
163
       
-
       
      "categories": {

     

       
164
       
-
       
        "type": "array",

     

       
165
       
-
       
        "items": {

     

       
166
       
-
       
          "type": "string",

     

       
167
       
-
       
          "enum": ["world", "tech", "business", "sports", "science"]

     

       
168
       
-
       
        },

     

       
169
       
-
       
        "description": "Kagi News categories to monitor",

     

       
170
       
-
       
        "minItems": 1

     

       
171
       
-
       
      },

     

       
172
       
-
       
      "subcategoryFilter": {

     

       
173
       
-
       
        "type": "array",

     

       
174
       
-
       
        "items": { "type": "string" },

     

       
175
       
-
       
        "description": "Optional: only post stories with these subcategories (e.g., 'World/Middle East', 'Tech/AI')"

     

       
176
       
-
       
      },

     

       
177
       
-
       
      "minSources": {

     

       
178
       
-
       
        "type": "integer",

     

       
179
       
-
       
        "minimum": 1,

     

       
180
       
-
       
        "default": 2,

     

       
181
       
-
       
        "description": "Minimum number of sources required for a story to be posted"

     

       
182
       
-
       
      },

     

       
183
       
-
       
      "includeImages": {

     

       
184
       
-
       
        "type": "boolean",

     

       
185
       
-
       
        "default": true,

     

       
186
       
-
       
        "description": "Include images from Kagi proxy in posts"

     

       
187
       
-
       
      },

     

       
188
       
-
       
      "postFormat": {

     

       
189
       
 
       
        "type": "string",

     

       
190
       
-
       
        "enum": ["full", "summary", "minimal"],

     

       
191
       
-
       
        "default": "full",

     

       
192
       
-
       
        "description": "How much content to include: full (all sections), summary (main paragraph + sources), minimal (title + link only)"

     

       
193
       
 
       
      }

     

       
194
       
 
       
    },

     

       
195
       
-
       
    "required": ["categories"]

     

       
196
       
 
       
  },

     

       
197
       
 
       
  "sourceUrl": "https://github.com/coves-social/kagi-news-aggregator",

     

       
198
       
 
       
  "maintainer": "did:plc:coves-platform",

     

       
199
       
-
       
  "createdAt": "2025-10-20T12:00:00Z"

     

       
200
       
 
       
}

     

       
201
       
 
       
```

     

       
202
       
 
       


     

       
0
       
 
       

     

       
0
       
 
       

     

       
203
       
 
       
---

     

       
204
       
 
       


     

       
205
       
-
       
## Community Configuration Examples

     

       
206
       
 
       


     

       
207
       
-
       
### Example 1: World News Community

     

       
208
       
 
       


     

       
209
       
-
       
```json

     

       
210
       
-
       
{

     

       
211
       
-
       
  "aggregatorDid": "did:web:kagi-news.coves.social",

     

       
212
       
-
       
  "enabled": true,

     

       
213
       
-
       
  "config": {

     

       
214
       
-
       
    "categories": ["world"],

     

       
215
       
-
       
    "minSources": 3,

     

       
216
       
-
       
    "includeImages": true,

     

       
217
       
-
       
    "postFormat": "full"

     

       
218
       
-
       
  }

     

       
219
       
-
       
}

     

       
220
       
-
       
```

     

       
221
       
 
       


     

       
222
       
-
       
**Result:** Posts all world news stories with 3+ sources, full content including images/highlights/perspectives.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
223
       
 
       


     

       
224
       
-
       
---

     

       
0
       
 
       

     

       
225
       
 
       


     

       
226
       
-
       
### Example 2: AI/Tech Community (Filtered)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
227
       
 
       


     

       
228
       
-
       
```json

     

       
229
       
-
       
{

     

       
230
       
-
       
  "aggregatorDid": "did:web:kagi-news.coves.social",

     

       
231
       
-
       
  "enabled": true,

     

       
232
       
-
       
  "config": {

     

       
233
       
-
       
    "categories": ["tech", "business"],

     

       
234
       
-
       
    "subcategoryFilter": ["Tech/AI", "Tech/Machine Learning", "Business/Tech Industry"],

     

       
235
       
-
       
    "minSources": 2,

     

       
236
       
-
       
    "includeImages": true,

     

       
237
       
-
       
    "postFormat": "full"

     

       
238
       
-
       
  }

     

       
239
       
-
       
}

     

       
240
       
-
       
```

     

       
241
       
 
       


     

       
242
       
-
       
**Result:** Only posts tech stories about AI/ML or tech industry business news with 2+ sources.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
243
       
 
       


     

       
244
       
-
       
---

     

       
245
       
-
       


     

       
246
       
-
       
### Example 3: Breaking News (Minimal)

     

       
247
       
 
       


     

       
248
       
-
       
```json

     

       
249
       
-
       
{

     

       
250
       
-
       
  "aggregatorDid": "did:web:kagi-news.coves.social",

     

       
251
       
-
       
  "enabled": true,

     

       
252
       
-
       
  "config": {

     

       
253
       
-
       
    "categories": ["world", "business", "tech"],

     

       
254
       
-
       
    "minSources": 5,

     

       
255
       
-
       
    "includeImages": false,

     

       
256
       
-
       
    "postFormat": "minimal"

     

       
257
       
-
       
  }

     

       
258
       
-
       
}

     

       
259
       
 
       
```

     

       
260
       
 
       


     

       
261
       
-
       
**Result:** Only major stories (5+ sources), minimal format (headline + link), no images.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
262
       
 
       


     

       
263
       
 
       
---

     

       
264
       
 
       


     
···

       
269
       
 
       
```json

     

       
270
       
 
       
{

     

       
271
       
 
       
  "$type": "social.coves.post.record",

     

       
272
       
-
       
  "author": "did:web:kagi-news.coves.social",

     

       
273
       
-
       
  "community": "did:plc:worldnews123",

     

       
274
       
 
       
  "title": "{Kagi story title}",

     

       
275
       
-
       
  "content": "{formatted content based on postFormat config}",

     

       
276
       
 
       
  "embed": {

     

       
277
       
-
       
    "$type": "app.bsky.embed.external",

     

       
278
       
 
       
    "external": {

     

       
279
       
-
       
      "uri": "https://kite.kagi.com/{uuid}/{category}/{id}",

     

       
280
       
 
       
      "title": "{story title}",

     

       
281
       
-
       
      "description": "{summary excerpt}",

     

       
282
       
-
       
      "thumb": "{image blob if includeImages=true}"

     

       
283
       
 
       
    }

     

       
284
       
 
       
  },

     

       
285
       
 
       
  "federatedFrom": {

     
···

       
296
       
 
       
}

     

       
297
       
 
       
```

     

       
298
       
 
       


     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
299
       
 
       
---

     

       
300
       
 
       


     

       
301
       
-
       
### Content Formatting by `postFormat`

     

       
302
       
 
       


     

       
303
       
-
       
#### Format: `full` (Default)

     

       
304
       
 
       


     

       
305
       
-
       
```markdown

     

       
0
       
 
       

     

       
306
       
 
       
{Main summary paragraph with source citations}

     

       
307
       
 
       


     

       
308
       
-
       
**Highlights:**

     

       
309
       
 
       
• {Bullet point 1}

     

       
310
       
 
       
• {Bullet point 2}

     

       
311
       
 
       
• ...

     

       
312
       
 
       


     

       
313
       
-
       
**Perspectives:**

     

       
314
       
-
       
• **{Actor}**: {Their perspective} ([Source]({url}))

     

       
315
       
 
       
• ...

     

       
316
       
 
       


     

       
317
       
-
       
> {Notable quote} — {Attribution}

     

       
318
       
 
       


     

       
319
       
-
       
**Sources:**

     

       
320
       
-
       
• [{Title}]({url}) - {domain}

     

       
321
       
 
       
• ...

     

       
322
       
 
       


     

       
323
       
 
       
---

     

       
324
       
-
       
📰 Story aggregated by [Kagi News]({kagi_story_url})

     

       
325
       
 
       
```

     

       
326
       
 
       


     

       
327
       
-
       
**Rationale:** Preserves Kagi's rich multi-source analysis, provides maximum value.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
328
       
 
       


     

       
329
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
330
       
 
       


     

       
331
       
-
       
#### Format: `summary`

     

       
332
       
-
       


     

       
333
       
-
       
```markdown

     

       
334
       
-
       
{Main summary paragraph with source citations}

     

       
335
       
-
       


     

       
336
       
-
       
**Sources:**

     

       
337
       
-
       
• [{Title}]({url}) - {domain}

     

       
338
       
-
       
• ...

     

       
339
       
 
       


     

       
340
       
 
       
---

     

       
341
       
-
       
📰 Story aggregated by [Kagi News]({kagi_story_url})

     

       
342
       
-
       
```

     

       
343
       
 
       


     

       
344
       
-
       
**Rationale:** Clean summary with source links, less overwhelming.

     

       
345
       
 
       


     

       
346
       
-
       
---

     

       
347
       
 
       


     

       
348
       
-
       
#### Format: `minimal`

     

       
349
       
 
       


     

       
350
       
-
       
```markdown

     

       
351
       
-
       
{Story title}

     

       
352
       
-
       


     

       
353
       
-
       
Read more: {kagi_story_url}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
354
       
 
       


     

       
355
       
-
       
**Sources:** {domain1}, {domain2}, {domain3}...

     

       
356
       
 
       


     

       
357
       
-
       
---

     

       
358
       
-
       
📰 Via [Kagi News]({kagi_story_url})

     

       
359
       
 
       
```

     

       
360
       
-
       


     

       
361
       
-
       
**Rationale:** Just headlines with link, for high-volume communities or breaking news alerts.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
362
       
 
       


     

       
363
       
 
       
---

     

       
364
       
 
       


     

       
365
       
-
       
## Implementation Details

     

       
366
       
 
       


     

       
367
       
-
       
### Component 1: Feed Poller

     

       
368
       
 
       


     

       
369
       
-
       
**Responsibility:** Fetch RSS feeds on schedule

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
370
       
 
       


     

       
371
       
-
       
```go

     

       
372
       
-
       
type FeedPoller struct {

     

       
373
       
-
       
    categories   []string

     

       
374
       
-
       
    pollInterval time.Duration

     

       
375
       
-
       
    httpClient   *http.Client

     

       
376
       
-
       
}

     

       
377
       
 
       


     

       
378
       
-
       
func (p *FeedPoller) Start(ctx context.Context) error {

     

       
379
       
-
       
    ticker := time.NewTicker(p.pollInterval) // 15 minutes

     

       
380
       
-
       
    defer ticker.Stop()

     

       
381
       
-
       


     

       
382
       
-
       
    for {

     

       
383
       
-
       
        select {

     

       
384
       
-
       
        case <-ticker.C:

     

       
385
       
-
       
            for _, category := range p.categories {

     

       
386
       
-
       
                feedURL := fmt.Sprintf("https://news.kagi.com/%s.xml", category)

     

       
387
       
-
       
                feed, err := p.fetchFeed(feedURL)

     

       
388
       
-
       
                if err != nil {

     

       
389
       
-
       
                    log.Printf("Failed to fetch %s: %v", feedURL, err)

     

       
390
       
-
       
                    continue

     

       
391
       
-
       
                }

     

       
392
       
-
       
                p.handleFeed(ctx, category, feed)

     

       
393
       
-
       
            }

     

       
394
       
-
       
        case <-ctx.Done():

     

       
395
       
-
       
            return nil

     

       
396
       
-
       
        }

     

       
397
       
-
       
    }

     

       
398
       
-
       
}

     

       
399
       
-
       


     

       
400
       
-
       
func (p *FeedPoller) fetchFeed(url string) (*gofeed.Feed, error) {

     

       
401
       
-
       
    parser := gofeed.NewParser()

     

       
402
       
-
       
    feed, err := parser.ParseURL(url)

     

       
403
       
-
       
    return feed, err

     

       
404
       
-
       
}

     

       
405
       
-
       
```

     

       
406
       
-
       


     

       
407
       
-
       
**Libraries:**

     

       
408
       
-
       
- `github.com/mmcdole/gofeed` - RSS/Atom parser

     

       
409
       
 
       


     

       
410
       
 
       
---

     

       
411
       
 
       


     

       
412
       
-
       
### Component 2: Item Parser

     

       
413
       
 
       


     

       
414
       
-
       
**Responsibility:** Extract structured data from RSS item HTML

     

       
415
       
-
       


     

       
416
       
-
       
```go

     

       
417
       
-
       
type KagiStory struct {

     

       
418
       
-
       
    Title             string

     

       
419
       
-
       
    Link              string

     

       
420
       
-
       
    GUID              string

     

       
421
       
-
       
    PubDate           time.Time

     

       
422
       
-
       
    Categories        []string

     

       
423
       
-
       


     

       
424
       
-
       
    // Parsed from HTML description

     

       
425
       
-
       
    Summary           string

     

       
426
       
-
       
    Highlights        []string

     

       
427
       
-
       
    Perspectives      []Perspective

     

       
428
       
-
       
    Quote             *Quote

     

       
429
       
-
       
    Sources           []Source

     

       
430
       
-
       
    ImageURL          string

     

       
431
       
-
       
    ImageAlt          string

     

       
432
       
-
       
}

     

       
433
       
 
       


     

       
434
       
-
       
type Perspective struct {

     

       
435
       
-
       
    Actor       string

     

       
436
       
-
       
    Description string

     

       
437
       
-
       
    SourceURL   string

     

       
438
       
-
       
}

     

       
439
       
 
       


     

       
440
       
-
       
type Quote struct {

     

       
441
       
-
       
    Text        string

     

       
442
       
-
       
    Attribution string

     

       
443
       
-
       
}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
444
       
 
       


     

       
445
       
-
       
type Source struct {

     

       
446
       
-
       
    Title  string

     

       
447
       
-
       
    URL    string

     

       
448
       
-
       
    Domain string

     

       
449
       
-
       
}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
450
       
 
       


     

       
451
       
-
       
func (p *ItemParser) Parse(item *gofeed.Item) (*KagiStory, error) {

     

       
452
       
-
       
    doc, err := goquery.NewDocumentFromReader(strings.NewReader(item.Description))

     

       
453
       
-
       
    if err != nil {

     

       
454
       
-
       
        return nil, err

     

       
455
       
-
       
    }

     

       
456
       
 
       


     

       
457
       
-
       
    story := &KagiStory{

     

       
458
       
-
       
        Title:      item.Title,

     

       
459
       
-
       
        Link:       item.Link,

     

       
460
       
-
       
        GUID:       item.GUID,

     

       
461
       
-
       
        PubDate:    *item.PublishedParsed,

     

       
462
       
-
       
        Categories: item.Categories,

     

       
463
       
-
       
    }

     

       
464
       
 
       


     

       
465
       
-
       
    // Extract summary (first <p> tag)

     

       
466
       
-
       
    story.Summary = doc.Find("p").First().Text()

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
467
       
 
       


     

       
468
       
-
       
    // Extract highlights

     

       
469
       
-
       
    doc.Find("h3:contains('Highlights')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       
470
       
-
       
        story.Highlights = append(story.Highlights, s.Text())

     

       
471
       
-
       
    })

     

       
0
       
 
       

     

       
472
       
 
       


     

       
473
       
-
       
    // Extract perspectives

     

       
474
       
-
       
    doc.Find("h3:contains('Perspectives')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       
475
       
-
       
        text := s.Text()

     

       
476
       
-
       
        link := s.Find("a").First()

     

       
477
       
-
       
        sourceURL, _ := link.Attr("href")

     

       
478
       
 
       


     

       
479
       
-
       
        // Parse format: "Actor: Description (Source)"

     

       
480
       
-
       
        parts := strings.SplitN(text, ":", 2)

     

       
481
       
-
       
        if len(parts) == 2 {

     

       
482
       
-
       
            story.Perspectives = append(story.Perspectives, Perspective{

     

       
483
       
-
       
                Actor:       strings.TrimSpace(parts[0]),

     

       
484
       
-
       
                Description: strings.TrimSpace(parts[1]),

     

       
485
       
-
       
                SourceURL:   sourceURL,

     

       
486
       
-
       
            })

     

       
487
       
-
       
        }

     

       
488
       
-
       
    })

     

       
489
       
 
       


     

       
490
       
-
       
    // Extract quote

     

       
491
       
-
       
    doc.Find("blockquote").Each(func(i int, s *goquery.Selection) {

     

       
492
       
-
       
        text := s.Text()

     

       
493
       
-
       
        parts := strings.Split(text, " - ")

     

       
494
       
-
       
        if len(parts) == 2 {

     

       
495
       
-
       
            story.Quote = &Quote{

     

       
496
       
-
       
                Text:        strings.TrimSpace(parts[0]),

     

       
497
       
-
       
                Attribution: strings.TrimSpace(parts[1]),

     

       
498
       
-
       
            }

     

       
499
       
-
       
        }

     

       
500
       
-
       
    })

     

       
501
       
 
       


     

       
502
       
-
       
    // Extract sources

     

       
503
       
-
       
    doc.Find("h3:contains('Sources')").Next("ul").Find("li").Each(func(i int, s *goquery.Selection) {

     

       
504
       
-
       
        link := s.Find("a").First()

     

       
505
       
-
       
        url, _ := link.Attr("href")

     

       
506
       
-
       
        title := link.Text()

     

       
507
       
-
       
        domain := extractDomain(s.Text())

     

       
508
       
 
       


     

       
509
       
-
       
        story.Sources = append(story.Sources, Source{

     

       
510
       
-
       
            Title:  title,

     

       
511
       
-
       
            URL:    url,

     

       
512
       
-
       
            Domain: domain,

     

       
513
       
-
       
        })

     

       
514
       
-
       
    })

     

       
515
       
 
       


     

       
516
       
-
       
    // Extract image

     

       
517
       
-
       
    img := doc.Find("img").First()

     

       
518
       
-
       
    if img.Length() > 0 {

     

       
519
       
-
       
        story.ImageURL, _ = img.Attr("src")

     

       
520
       
-
       
        story.ImageAlt, _ = img.Attr("alt")

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
521
       
 
       
    }

     

       
522
       
-
       


     

       
523
       
-
       
    return story, nil

     

       
524
       
 
       
}

     

       
525
       
 
       
```

     

       
526
       
 
       


     

       
527
       
-
       
**Libraries:**

     

       
528
       
-
       
- `github.com/PuerkitoBio/goquery` - HTML parsing

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
529
       
 
       


     

       
530
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
531
       
 
       


     

       
532
       
-
       
### Component 3: Deduplication

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
533
       
 
       


     

       
534
       
-
       
**Responsibility:** Track posted stories to prevent duplicates

     

       
535
       
 
       


     

       
536
       
-
       
```go

     

       
537
       
-
       
type Deduplicator struct {

     

       
538
       
-
       
    db *sql.DB

     

       
539
       
-
       
}

     

       
540
       
 
       


     

       
541
       
-
       
func (d *Deduplicator) AlreadyPosted(guid string) (bool, error) {

     

       
542
       
-
       
    var exists bool

     

       
543
       
-
       
    err := d.db.QueryRow(`

     

       
544
       
-
       
        SELECT EXISTS(

     

       
545
       
-
       
            SELECT 1 FROM kagi_news_posted_stories

     

       
546
       
-
       
            WHERE guid = $1

     

       
547
       
-
       
        )

     

       
548
       
-
       
    `, guid).Scan(&exists)

     

       
549
       
-
       
    return exists, err

     

       
550
       
-
       
}

     

       
551
       
 
       


     

       
552
       
-
       
func (d *Deduplicator) MarkPosted(guid, postURI string) error {

     

       
553
       
-
       
    _, err := d.db.Exec(`

     

       
554
       
-
       
        INSERT INTO kagi_news_posted_stories (guid, post_uri, posted_at)

     

       
555
       
-
       
        VALUES ($1, $2, NOW())

     

       
556
       
-
       
        ON CONFLICT (guid) DO NOTHING

     

       
557
       
-
       
    `, guid, postURI)

     

       
558
       
-
       
    return err

     

       
559
       
-
       
}

     

       
560
       
-
       
```

     

       
561
       
 
       


     

       
562
       
-
       
**Database Table:**

     

       
563
       
-
       
```sql

     

       
564
       
-
       
CREATE TABLE kagi_news_posted_stories (

     

       
565
       
-
       
    guid TEXT PRIMARY KEY,

     

       
566
       
-
       
    post_uri TEXT NOT NULL,

     

       
567
       
-
       
    posted_at TIMESTAMPTZ NOT NULL DEFAULT NOW()

     

       
568
       
-
       
);

     

       
569
       
 
       


     

       
570
       
-
       
CREATE INDEX idx_kagi_posted_at ON kagi_news_posted_stories(posted_at DESC);

     

       
571
       
-
       
```

     

       
0
       
 
       

     

       
0
       
 
       

     

       
572
       
 
       


     

       
573
       
-
       
**Cleanup:** Periodic job deletes rows older than 30 days (Kagi unlikely to re-post old stories).

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
574
       
 
       


     

       
575
       
 
       
---

     

       
576
       
 
       


     

       
577
       
-
       
### Component 4: Category Mapper

     

       
578
       
 
       


     

       
579
       
-
       
**Responsibility:** Map Kagi categories to authorized communities

     

       
580
       
 
       


     

       
581
       
-
       
```go

     

       
582
       
-
       
func (m *CategoryMapper) GetTargetCommunities(story *KagiStory) ([]*CommunityAuth, error) {

     

       
583
       
-
       
    // Get all communities that have authorized this aggregator

     

       
584
       
-
       
    allAuths, err := m.aggregator.GetAuthorizedCommunities(context.Background())

     

       
585
       
-
       
    if err != nil {

     

       
586
       
-
       
        return nil, err

     

       
587
       
-
       
    }

     

       
588
       
 
       


     

       
589
       
-
       
    var targets []*CommunityAuth

     

       
590
       
-
       
    for _, auth := range allAuths {

     

       
591
       
-
       
        if !auth.Enabled {

     

       
592
       
-
       
            continue

     

       
593
       
-
       
        }

     

       
594
       
 
       


     

       
595
       
-
       
        config := auth.Config

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
596
       
 
       


     

       
597
       
-
       
        // Check if story's primary category is in config.categories

     

       
598
       
-
       
        primaryCategory := story.Categories[0]

     

       
599
       
-
       
        if !contains(config["categories"], primaryCategory) {

     

       
600
       
-
       
            continue

     

       
601
       
-
       
        }

     

       
602
       
 
       


     

       
603
       
-
       
        // Check subcategory filter (if specified)

     

       
604
       
-
       
        if subcatFilter, ok := config["subcategoryFilter"].([]string); ok && len(subcatFilter) > 0 {

     

       
605
       
-
       
            if !hasAnySubcategory(story.Categories, subcatFilter) {

     

       
606
       
-
       
                continue

     

       
607
       
-
       
            }

     

       
608
       
-
       
        }

     

       
609
       
-
       


     

       
610
       
-
       
        // Check minimum sources requirement

     

       
611
       
-
       
        minSources := config["minSources"].(int)

     

       
612
       
-
       
        if len(story.Sources) < minSources {

     

       
613
       
-
       
            continue

     

       
614
       
-
       
        }

     

       
615
       
-
       


     

       
616
       
-
       
        targets = append(targets, auth)

     

       
617
       
-
       
    }

     

       
618
       
-
       


     

       
619
       
-
       
    return targets, nil

     

       
620
       
-
       
}

     

       
621
       
-
       
```

     

       
622
       
 
       


     

       
623
       
 
       
---

     

       
624
       
 
       


     

       
625
       
-
       
### Component 5: Post Formatter

     

       
626
       
-
       


     

       
627
       
-
       
**Responsibility:** Convert Kagi story to Coves post format

     

       
628
       
-
       


     

       
629
       
-
       
```go

     

       
630
       
-
       
func (f *PostFormatter) Format(story *KagiStory, format string) string {

     

       
631
       
-
       
    switch format {

     

       
632
       
-
       
    case "full":

     

       
633
       
-
       
        return f.formatFull(story)

     

       
634
       
-
       
    case "summary":

     

       
635
       
-
       
        return f.formatSummary(story)

     

       
636
       
-
       
    case "minimal":

     

       
637
       
-
       
        return f.formatMinimal(story)

     

       
638
       
-
       
    default:

     

       
639
       
-
       
        return f.formatFull(story)

     

       
640
       
-
       
    }

     

       
641
       
-
       
}

     

       
642
       
-
       


     

       
643
       
-
       
func (f *PostFormatter) formatFull(story *KagiStory) string {

     

       
644
       
-
       
    var buf strings.Builder

     

       
645
       
-
       


     

       
646
       
-
       
    // Summary

     

       
647
       
-
       
    buf.WriteString(story.Summary)

     

       
648
       
-
       
    buf.WriteString("\n\n")

     

       
649
       
-
       


     

       
650
       
-
       
    // Highlights

     

       
651
       
-
       
    if len(story.Highlights) > 0 {

     

       
652
       
-
       
        buf.WriteString("**Highlights:**\n")

     

       
653
       
-
       
        for _, h := range story.Highlights {

     

       
654
       
-
       
            buf.WriteString(fmt.Sprintf("• %s\n", h))

     

       
655
       
-
       
        }

     

       
656
       
-
       
        buf.WriteString("\n")

     

       
657
       
-
       
    }

     

       
658
       
 
       


     

       
659
       
-
       
    // Perspectives

     

       
660
       
-
       
    if len(story.Perspectives) > 0 {

     

       
661
       
-
       
        buf.WriteString("**Perspectives:**\n")

     

       
662
       
-
       
        for _, p := range story.Perspectives {

     

       
663
       
-
       
            buf.WriteString(fmt.Sprintf("• **%s**: %s ([Source](%s))\n", p.Actor, p.Description, p.SourceURL))

     

       
664
       
-
       
        }

     

       
665
       
-
       
        buf.WriteString("\n")

     

       
666
       
-
       
    }

     

       
667
       
 
       


     

       
668
       
-
       
    // Quote

     

       
669
       
-
       
    if story.Quote != nil {

     

       
670
       
-
       
        buf.WriteString(fmt.Sprintf("> %s — %s\n\n", story.Quote.Text, story.Quote.Attribution))

     

       
671
       
-
       
    }

     

       
0
       
 
       

     

       
0
       
 
       

     

       
672
       
 
       


     

       
673
       
-
       
    // Sources

     

       
674
       
-
       
    buf.WriteString("**Sources:**\n")

     

       
675
       
-
       
    for _, s := range story.Sources {

     

       
676
       
-
       
        buf.WriteString(fmt.Sprintf("• [%s](%s) - %s\n", s.Title, s.URL, s.Domain))

     

       
677
       
-
       
    }

     

       
678
       
-
       
    buf.WriteString("\n")

     

       
679
       
 
       


     

       
680
       
-
       
    // Attribution

     

       
681
       
-
       
    buf.WriteString(fmt.Sprintf("---\n📰 Story aggregated by [Kagi News](%s)", story.Link))

     

       
682
       
 
       


     

       
683
       
-
       
    return buf.String()

     

       
684
       
-
       
}

     

       
685
       
 
       


     

       
686
       
-
       
func (f *PostFormatter) formatSummary(story *KagiStory) string {

     

       
687
       
-
       
    var buf strings.Builder

     

       
688
       
 
       


     

       
689
       
-
       
    buf.WriteString(story.Summary)

     

       
690
       
-
       
    buf.WriteString("\n\n**Sources:**\n")

     

       
691
       
-
       
    for _, s := range story.Sources {

     

       
692
       
-
       
        buf.WriteString(fmt.Sprintf("• [%s](%s) - %s\n", s.Title, s.URL, s.Domain))

     

       
693
       
-
       
    }

     

       
694
       
-
       
    buf.WriteString("\n")

     

       
695
       
-
       
    buf.WriteString(fmt.Sprintf("---\n📰 Story aggregated by [Kagi News](%s)", story.Link))

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
696
       
 
       


     

       
697
       
-
       
    return buf.String()

     

       
698
       
-
       
}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
699
       
 
       


     

       
700
       
-
       
func (f *PostFormatter) formatMinimal(story *KagiStory) string {

     

       
701
       
-
       
    sourceDomains := make([]string, len(story.Sources))

     

       
702
       
-
       
    for i, s := range story.Sources {

     

       
703
       
-
       
        sourceDomains[i] = s.Domain

     

       
704
       
-
       
    }

     

       
705
       
-
       


     

       
706
       
-
       
    return fmt.Sprintf(

     

       
707
       
-
       
        "%s\n\nRead more: %s\n\n**Sources:** %s\n\n---\n📰 Via [Kagi News](%s)",

     

       
708
       
-
       
        story.Title,

     

       
709
       
-
       
        story.Link,

     

       
710
       
-
       
        strings.Join(sourceDomains, ", "),

     

       
711
       
-
       
        story.Link,

     

       
712
       
-
       
    )

     

       
713
       
-
       
}

     

       
714
       
-
       
```

     

       
715
       
 
       


     

       
716
       
 
       
---

     

       
717
       
 
       


     

       
718
       
-
       
### Component 6: Post Publisher

     

       
719
       
-
       


     

       
720
       
-
       
**Responsibility:** Create posts via Coves API

     

       
721
       
 
       


     

       
722
       
-
       
```go

     

       
723
       
-
       
func (p *PostPublisher) PublishStory(ctx context.Context, story *KagiStory, communities []*CommunityAuth) error {

     

       
724
       
-
       
    for _, comm := range communities {

     

       
725
       
-
       
        config := comm.Config

     

       
726
       
 
       


     

       
727
       
-
       
        // Format content based on config

     

       
728
       
-
       
        postFormat := config["postFormat"].(string)

     

       
729
       
-
       
        content := p.formatter.Format(story, postFormat)

     

       
730
       
-
       


     

       
731
       
-
       
        // Build embed

     

       
732
       
-
       
        var embed *aggregator.Embed

     

       
733
       
-
       
        if config["includeImages"].(bool) && story.ImageURL != "" {

     

       
734
       
-
       
            // TODO: Handle image upload/blob creation

     

       
735
       
-
       
            embed = &aggregator.Embed{

     

       
736
       
-
       
                Type: "app.bsky.embed.external",

     

       
737
       
-
       
                External: &aggregator.External{

     

       
738
       
-
       
                    URI:         story.Link,

     

       
739
       
-
       
                    Title:       story.Title,

     

       
740
       
-
       
                    Description: truncate(story.Summary, 300),

     

       
741
       
-
       
                    Thumb:       story.ImageURL, // or blob reference

     

       
742
       
-
       
                },

     

       
743
       
-
       
            }

     

       
744
       
-
       
        }

     

       
745
       
 
       


     

       
746
       
-
       
        // Create post

     

       
747
       
-
       
        post := aggregator.Post{

     

       
748
       
-
       
            Title:   story.Title,

     

       
749
       
-
       
            Content: content,

     

       
750
       
-
       
            Embed:   embed,

     

       
751
       
-
       
            FederatedFrom: &aggregator.FederatedSource{

     

       
752
       
-
       
                Platform:          "kagi-news-rss",

     

       
753
       
-
       
                URI:               story.Link,

     

       
754
       
-
       
                ID:                story.GUID,

     

       
755
       
-
       
                OriginalCreatedAt: story.PubDate,

     

       
756
       
-
       
            },

     

       
757
       
-
       
            ContentLabels: story.Categories,

     

       
758
       
-
       
        }

     

       
759
       
 
       


     

       
760
       
-
       
        err := p.aggregator.CreatePost(ctx, comm.CommunityDID, post)

     

       
761
       
-
       
        if err != nil {

     

       
762
       
-
       
            log.Printf("Failed to create post in %s: %v", comm.CommunityDID, err)

     

       
763
       
-
       
            continue

     

       
764
       
-
       
        }

     

       
765
       
 
       


     

       
766
       
-
       
        // Mark as posted

     

       
767
       
-
       
        _ = p.deduplicator.MarkPosted(story.GUID, "post-uri-from-response")

     

       
768
       
-
       
    }

     

       
769
       
 
       


     

       
770
       
-
       
    return nil

     

       
771
       
-
       
}

     

       
772
       
-
       
```

     

       
0
       
 
       

     

       
773
       
 
       


     

       
774
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
775
       
 
       


     

       
776
       
-
       
## Image Handling Strategy

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
777
       
 
       


     

       
778
       
-
       
### Initial Implementation (MVP)

     

       
779
       
 
       


     

       
780
       
-
       
**Approach:** Use Kagi proxy URLs directly in embeds

     

       
0
       
 
       

     

       
0
       
 
       

     

       
781
       
 
       


     

       
782
       
-
       
**Rationale:**

     

       
783
       
-
       
- Simplest implementation

     

       
784
       
-
       
- Kagi proxy likely allows hotlinking for non-commercial use

     

       
785
       
-
       
- No storage costs

     

       
786
       
-
       
- Images are already optimized by Kagi

     

       
787
       
 
       


     

       
788
       
-
       
**Risk Mitigation:**

     

       
789
       
-
       
- Monitor for broken images

     

       
790
       
-
       
- Add fallback: if image fails to load, skip embed

     

       
791
       
-
       
- Prepare migration plan to self-hosting if needed

     

       
792
       
 
       


     

       
793
       
-
       
**Code:**

     

       
794
       
-
       
```go

     

       
795
       
-
       
if config["includeImages"].(bool) && story.ImageURL != "" {

     

       
796
       
-
       
    // Use Kagi proxy URL directly

     

       
797
       
-
       
    embed = &aggregator.Embed{

     

       
798
       
-
       
        External: &aggregator.External{

     

       
799
       
-
       
            Thumb: story.ImageURL, // https://kagiproxy.com/img/...

     

       
800
       
-
       
        },

     

       
801
       
-
       
    }

     

       
802
       
-
       
}

     

       
803
       
 
       
```

     

       
804
       
 
       


     

       
805
       
 
       
---

     

       
806
       
 
       


     

       
807
       
-
       
### Future Enhancement (If Issues Arise)

     

       
808
       
 
       


     

       
809
       
-
       
**Approach:** Download and re-host images

     

       
0
       
 
       

     

       
810
       
 
       


     

       
811
       
-
       
**Implementation:**

     

       
812
       
-
       
1. Download image from Kagi proxy

     

       
813
       
-
       
2. Upload to Coves blob storage (or S3/CDN)

     

       
814
       
-
       
3. Use blob reference in embed

     

       
0
       
 
       

     

       
815
       
 
       


     

       
816
       
-
       
**Code:**

     

       
817
       
-
       
```go

     

       
818
       
-
       
func (p *PostPublisher) uploadImage(imageURL string) (string, error) {

     

       
819
       
-
       
    // Download from Kagi proxy

     

       
820
       
-
       
    resp, err := http.Get(imageURL)

     

       
821
       
-
       
    if err != nil {

     

       
822
       
-
       
        return "", err

     

       
823
       
-
       
    }

     

       
824
       
-
       
    defer resp.Body.Close()

     

       
825
       
 
       


     

       
826
       
-
       
    // Upload to blob storage

     

       
827
       
-
       
    blob, err := p.blobStore.Upload(resp.Body, resp.Header.Get("Content-Type"))

     

       
828
       
-
       
    if err != nil {

     

       
829
       
-
       
        return "", err

     

       
830
       
-
       
    }

     

       
831
       
 
       


     

       
832
       
-
       
    return blob.Ref, nil

     

       
833
       
-
       
}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
834
       
 
       
```

     

       
835
       
 
       


     

       
836
       
-
       
**Decision Point:** Only implement if:

     

       
837
       
-
       
- Kagi blocks hotlinking

     

       
838
       
-
       
- Kagi proxy becomes unreliable

     

       
839
       
-
       
- Legal clarification needed

     

       
840
       
 
       


     

       
841
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
842
       
 
       


     

       
843
       
-
       
## Rate Limiting & Performance

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
844
       
 
       


     

       
845
       
-
       
### Rate Limits

     

       
0
       
 
       

     

       
0
       
 
       

     

       
846
       
 
       


     

       
847
       
-
       
**RSS Fetching:**

     

       
848
       
-
       
- Poll each category feed every 15 minutes

     

       
849
       
-
       
- Max 4 categories = 4 requests per 15 min = 16 req/hour

     

       
850
       
-
       
- Well within any reasonable limit

     

       
851
       
 
       


     

       
852
       
-
       
**Post Creation:**

     

       
853
       
-
       
- Aggregator rate limit: 10 posts/hour per community

     

       
854
       
-
       
- Global limit: 100 posts/hour across all communities

     

       
855
       
-
       
- Kagi News publishes ~5-10 stories per category per day

     

       
856
       
-
       
- = ~20-40 posts/day total across all categories

     

       
857
       
-
       
- = ~2-4 posts/hour average

     

       
858
       
-
       
- Well within limits

     

       
859
       
-
       


     

       
860
       
-
       
**Performance Targets:**

     

       
861
       
-
       
- Story posted within 15 minutes of appearing in RSS feed

     

       
862
       
-
       
- < 1 second to parse and format a story

     

       
863
       
-
       
- < 500ms to publish a post via API

     

       
864
       
 
       


     

       
865
       
 
       
---

     

       
866
       
 
       


     

       
867
       
-
       
## Monitoring & Observability

     

       
868
       
 
       


     

       
869
       
-
       
### Metrics to Track

     

       
870
       
 
       


     

       
871
       
-
       
**Feed Polling:**

     

       
872
       
-
       
- `kagi_feed_poll_total` (counter) - Total feed polls by category

     

       
873
       
-
       
- `kagi_feed_poll_errors` (counter) - Failed polls by category/error

     

       
874
       
-
       
- `kagi_feed_items_fetched` (gauge) - Items per poll by category

     

       
875
       
-
       
- `kagi_feed_poll_duration_seconds` (histogram) - Poll latency

     

       
876
       
 
       


     

       
877
       
-
       
**Story Processing:**

     

       
878
       
-
       
- `kagi_stories_parsed_total` (counter) - Successfully parsed stories

     

       
879
       
-
       
- `kagi_stories_parse_errors` (counter) - Parse failures by error type

     

       
880
       
-
       
- `kagi_stories_filtered` (counter) - Stories filtered out by reason (duplicate, min sources, category)

     

       
881
       
-
       
- `kagi_stories_posted` (counter) - Stories successfully posted by community

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
882
       
 
       


     

       
883
       
-
       
**Post Publishing:**

     

       
884
       
-
       
- `kagi_posts_created_total` (counter) - Total posts created

     

       
885
       
-
       
- `kagi_posts_failed` (counter) - Failed posts by error type

     

       
886
       
-
       
- `kagi_post_publish_duration_seconds` (histogram) - Post creation latency

     

       
0
       
 
       

     

       
0
       
 
       

     

       
887
       
 
       


     

       
888
       
-
       
**Health:**

     

       
889
       
-
       
- `kagi_aggregator_up` (gauge) - Service health (1 = healthy, 0 = down)

     

       
890
       
-
       
- `kagi_last_successful_poll_timestamp` (gauge) - Last successful poll time by category

     

       
891
       
 
       


     

       
892
       
 
       
---

     

       
893
       
 
       


     

       
894
       
-
       
### Logging

     

       
895
       
 
       


     

       
896
       
-
       
**Structured Logging:**

     

       
897
       
-
       
```go

     

       
898
       
-
       
log.Info("Story posted",

     

       
899
       
-
       
    "guid", story.GUID,

     

       
900
       
-
       
    "title", story.Title,

     

       
901
       
-
       
    "community", comm.CommunityDID,

     

       
902
       
-
       
    "post_uri", postURI,

     

       
903
       
-
       
    "sources", len(story.Sources),

     

       
904
       
-
       
    "format", postFormat,

     

       
905
       
-
       
)

     

       
906
       
 
       


     

       
907
       
-
       
log.Error("Failed to parse story",

     

       
908
       
-
       
    "guid", item.GUID,

     

       
909
       
-
       
    "feed", feedURL,

     

       
910
       
-
       
    "error", err,

     

       
911
       
-
       
)

     

       
912
       
-
       
```

     

       
913
       
 
       


     

       
914
       
-
       
**Log Levels:**

     

       
915
       
-
       
- DEBUG: Feed items, parsing details

     

       
916
       
-
       
- INFO: Stories posted, communities targeted

     

       
917
       
-
       
- WARN: Parse errors, rate limit approaching

     

       
918
       
-
       
- ERROR: Failed posts, feed fetch failures

     

       
919
       
 
       


     

       
920
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
921
       
 
       


     

       
922
       
-
       
### Alerts

     

       
923
       
-
       


     

       
924
       
-
       
**Critical:**

     

       
925
       
-
       
- Feed polling failing for > 1 hour

     

       
926
       
-
       
- Post creation failing for > 10 consecutive attempts

     

       
927
       
-
       
- Aggregator unauthorized (auth record disabled/deleted)

     

       
928
       
-
       


     

       
929
       
-
       
**Warning:**

     

       
930
       
-
       
- Post creation rate < 50% of expected

     

       
931
       
-
       
- Parse errors > 10% of items

     

       
932
       
-
       
- Approaching rate limits (> 80% of quota)

     

       
933
       
 
       


     

       
934
       
 
       
---

     

       
935
       
 
       


     

       
936
       
-
       
## Deployment

     

       
937
       
-
       


     

       
938
       
-
       
### Infrastructure

     

       
939
       
-
       


     

       
940
       
-
       
**Service Type:** Long-running daemon

     

       
941
       
 
       


     

       
942
       
-
       
**Hosting:** Kubernetes (same cluster as Coves AppView)

     

       
943
       
 
       


     

       
944
       
-
       
**Resources:**

     

       
945
       
-
       
- CPU: 0.5 cores (low CPU usage, mostly I/O)

     

       
946
       
-
       
- Memory: 512 MB (small in-memory cache for recent GUIDs)

     

       
947
       
-
       
- Storage: 1 GB (SQLite for deduplication tracking)

     

       
948
       
 
       


     

       
949
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
950
       
 
       


     

       
951
       
-
       
### Configuration

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
952
       
 
       


     

       
953
       
-
       
**Environment Variables:**

     

       
954
       
-
       
```bash

     

       
955
       
-
       
# Aggregator identity

     

       
956
       
-
       
AGGREGATOR_DID=did:web:kagi-news.coves.social

     

       
957
       
-
       
AGGREGATOR_PRIVATE_KEY_PATH=/secrets/private-key.pem

     

       
958
       
 
       


     

       
959
       
-
       
# Coves API

     

       
960
       
-
       
COVES_API_URL=https://api.coves.social

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
961
       
 
       


     

       
962
       
-
       
# Feed polling

     

       
963
       
-
       
POLL_INTERVAL=15m

     

       
964
       
-
       
CATEGORIES=world,tech,business,sports

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
965
       
 
       


     

       
966
       
-
       
# Database (for deduplication)

     

       
967
       
-
       
DB_PATH=/data/kagi-news.db

     

       
0
       
 
       

     

       
0
       
 
       

     

       
968
       
 
       


     

       
969
       
-
       
# Monitoring

     

       
970
       
-
       
METRICS_PORT=9090

     

       
971
       
-
       
LOG_LEVEL=info

     

       
972
       
-
       
```

     

       
973
       
 
       


     

       
974
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
975
       
 
       


     

       
976
       
-
       
### Deployment Manifest

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
977
       
 
       


     

       
978
       
-
       
```yaml

     

       
979
       
-
       
apiVersion: apps/v1

     

       
980
       
-
       
kind: Deployment

     

       
981
       
-
       
metadata:

     

       
982
       
-
       
  name: kagi-news-aggregator

     

       
983
       
-
       
  namespace: coves

     

       
984
       
-
       
spec:

     

       
985
       
-
       
  replicas: 1

     

       
986
       
-
       
  selector:

     

       
987
       
-
       
    matchLabels:

     

       
988
       
-
       
      app: kagi-news-aggregator

     

       
989
       
-
       
  template:

     

       
990
       
-
       
    metadata:

     

       
991
       
-
       
      labels:

     

       
992
       
-
       
        app: kagi-news-aggregator

     

       
993
       
-
       
    spec:

     

       
994
       
-
       
      containers:

     

       
995
       
-
       
      - name: aggregator

     

       
996
       
-
       
        image: coves/kagi-news-aggregator:latest

     

       
997
       
-
       
        env:

     

       
998
       
-
       
        - name: AGGREGATOR_DID

     

       
999
       
-
       
          value: did:web:kagi-news.coves.social

     

       
1000
       
-
       
        - name: COVES_API_URL

     

       
1001
       
-
       
          value: https://api.coves.social

     

       
1002
       
-
       
        - name: POLL_INTERVAL

     

       
1003
       
-
       
          value: 15m

     

       
1004
       
-
       
        - name: CATEGORIES

     

       
1005
       
-
       
          value: world,tech,business,sports

     

       
1006
       
-
       
        - name: DB_PATH

     

       
1007
       
-
       
          value: /data/kagi-news.db

     

       
1008
       
-
       
        - name: AGGREGATOR_PRIVATE_KEY_PATH

     

       
1009
       
-
       
          value: /secrets/private-key.pem

     

       
1010
       
-
       
        volumeMounts:

     

       
1011
       
-
       
        - name: data

     

       
1012
       
-
       
          mountPath: /data

     

       
1013
       
-
       
        - name: secrets

     

       
1014
       
-
       
          mountPath: /secrets

     

       
1015
       
-
       
          readOnly: true

     

       
1016
       
-
       
        ports:

     

       
1017
       
-
       
        - name: metrics

     

       
1018
       
-
       
          containerPort: 9090

     

       
1019
       
-
       
        resources:

     

       
1020
       
-
       
          requests:

     

       
1021
       
-
       
            cpu: 250m

     

       
1022
       
-
       
            memory: 256Mi

     

       
1023
       
-
       
          limits:

     

       
1024
       
-
       
            cpu: 500m

     

       
1025
       
-
       
            memory: 512Mi

     

       
1026
       
-
       
      volumes:

     

       
1027
       
-
       
      - name: data

     

       
1028
       
-
       
        persistentVolumeClaim:

     

       
1029
       
-
       
          claimName: kagi-news-data

     

       
1030
       
-
       
      - name: secrets

     

       
1031
       
-
       
        secret:

     

       
1032
       
-
       
          secretName: kagi-news-private-key

     

       
1033
       
-
       
```

     

       
1034
       
 
       


     

       
1035
       
 
       
---

     

       
1036
       
 
       


     

       
1037
       
-
       
## Testing Strategy

     

       
1038
       
 
       


     

       
1039
       
-
       
### Unit Tests

     

       
1040
       
 
       


     

       
1041
       
-
       
**Feed Parsing:**

     

       
1042
       
-
       
```go

     

       
1043
       
-
       
func TestParseFeed(t *testing.T) {

     

       
1044
       
-
       
    feed := loadTestFeed("testdata/world.xml")

     

       
1045
       
-
       
    stories, err := parser.Parse(feed)

     

       
1046
       
-
       
    assert.NoError(t, err)

     

       
1047
       
-
       
    assert.Len(t, stories, 10)

     

       
1048
       
 
       


     

       
1049
       
-
       
    story := stories[0]

     

       
1050
       
-
       
    assert.NotEmpty(t, story.Title)

     

       
1051
       
-
       
    assert.NotEmpty(t, story.Summary)

     

       
1052
       
-
       
    assert.Greater(t, len(story.Sources), 1)

     

       
1053
       
-
       
}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1054
       
 
       


     

       
1055
       
-
       
func TestParseStoryHTML(t *testing.T) {

     

       
1056
       
-
       
    html := `<p>Summary [source.com#1]</p>

     

       
1057
       
-
       
             <h3>Highlights:</h3>

     

       
1058
       
-
       
             <ul><li>Point 1</li></ul>

     

       
1059
       
-
       
             <h3>Sources:</h3>

     

       
1060
       
-
       
             <ul><li><a href="https://example.com">Title</a> - example.com</li></ul>`

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1061
       
 
       


     

       
1062
       
-
       
    story, err := parser.ParseHTML(html)

     

       
1063
       
-
       
    assert.NoError(t, err)

     

       
1064
       
-
       
    assert.Equal(t, "Summary [source.com#1]", story.Summary)

     

       
1065
       
-
       
    assert.Len(t, story.Highlights, 1)

     

       
1066
       
-
       
    assert.Len(t, story.Sources, 1)

     

       
1067
       
-
       
}

     

       
1068
       
-
       
```

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1069
       
 
       


     

       
1070
       
-
       
**Formatting:**

     

       
1071
       
-
       
```go

     

       
1072
       
-
       
func TestFormatFull(t *testing.T) {

     

       
1073
       
-
       
    story := &KagiStory{

     

       
1074
       
-
       
        Summary: "Test summary",

     

       
1075
       
-
       
        Sources: []Source{

     

       
1076
       
-
       
            {Title: "Article", URL: "https://example.com", Domain: "example.com"},

     

       
1077
       
-
       
        },

     

       
1078
       
-
       
    }

     

       
1079
       
 
       


     

       
1080
       
-
       
    content := formatter.Format(story, "full")

     

       
1081
       
-
       
    assert.Contains(t, content, "Test summary")

     

       
1082
       
-
       
    assert.Contains(t, content, "**Sources:**")

     

       
1083
       
-
       
    assert.Contains(t, content, "📰 Story aggregated by")

     

       
1084
       
-
       
}

     

       
1085
       
-
       
```

     

       
0
       
 
       

     

       
1086
       
 
       


     

       
1087
       
-
       
**Deduplication:**

     

       
1088
       
-
       
```go

     

       
1089
       
-
       
func TestDeduplication(t *testing.T) {

     

       
1090
       
-
       
    guid := "test-guid-123"

     

       
1091
       
 
       


     

       
1092
       
-
       
    posted, err := deduplicator.AlreadyPosted(guid)

     

       
1093
       
-
       
    assert.NoError(t, err)

     

       
1094
       
-
       
    assert.False(t, posted)

     

       
1095
       
-
       


     

       
1096
       
-
       
    err = deduplicator.MarkPosted(guid, "at://...")

     

       
1097
       
-
       
    assert.NoError(t, err)

     

       
1098
       
-
       


     

       
1099
       
-
       
    posted, err = deduplicator.AlreadyPosted(guid)

     

       
1100
       
-
       
    assert.NoError(t, err)

     

       
1101
       
-
       
    assert.True(t, posted)

     

       
1102
       
-
       
}

     

       
1103
       
 
       
```

     

       
1104
       
 
       


     

       
1105
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1106
       
 
       


     

       
1107
       
 
       
### Integration Tests

     

       
1108
       
 
       


     

       
1109
       
-
       
**With Mock Coves API:**

     

       
1110
       
-
       
```go

     

       
1111
       
-
       
func TestPublishStory(t *testing.T) {

     

       
1112
       
-
       
    // Setup mock Coves API

     

       
1113
       
-
       
    mockAPI := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

     

       
1114
       
-
       
        assert.Equal(t, "/xrpc/social.coves.post.create", r.URL.Path)

     

       
0
       
 
       

     

       
1115
       
 
       


     

       
1116
       
-
       
        var input CreatePostInput

     

       
1117
       
-
       
        json.NewDecoder(r.Body).Decode(&input)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1118
       
 
       


     

       
1119
       
-
       
        assert.Equal(t, "did:plc:test-community", input.Community)

     

       
1120
       
-
       
        assert.NotEmpty(t, input.Title)

     

       
1121
       
-
       
        assert.Contains(t, input.Content, "📰 Story aggregated by")

     

       
1122
       
 
       


     

       
1123
       
-
       
        w.WriteHeader(200)

     

       
1124
       
-
       
        json.NewEncoder(w).Encode(CreatePostOutput{URI: "at://..."})

     

       
1125
       
-
       
    }))

     

       
1126
       
-
       
    defer mockAPI.Close()

     

       
1127
       
 
       


     

       
1128
       
-
       
    // Test story publishing

     

       
1129
       
-
       
    publisher := NewPostPublisher(mockAPI.URL)

     

       
1130
       
-
       
    err := publisher.PublishStory(ctx, testStory, []*CommunityAuth{testComm})

     

       
1131
       
-
       
    assert.NoError(t, err)

     

       
1132
       
-
       
}

     

       
1133
       
-
       
```

     

       
1134
       
 
       


     

       
1135
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1136
       
 
       


     

       
1137
       
-
       
### E2E Tests

     

       
1138
       
 
       


     

       
1139
       
-
       
**With Real RSS Feed:**

     

       
1140
       
-
       
```go

     

       
1141
       
-
       
func TestE2E_FetchAndParse(t *testing.T) {

     

       
1142
       
-
       
    if testing.Short() {

     

       
1143
       
-
       
        t.Skip("Skipping E2E test")

     

       
1144
       
-
       
    }

     

       
0
       
 
       

     

       
1145
       
 
       


     

       
1146
       
-
       
    // Fetch real Kagi News feed

     

       
1147
       
-
       
    feed, err := poller.fetchFeed("https://news.kagi.com/world.xml")

     

       
1148
       
-
       
    assert.NoError(t, err)

     

       
1149
       
-
       
    assert.NotEmpty(t, feed.Items)

     

       
1150
       
 
       


     

       
1151
       
-
       
    // Parse first item

     

       
1152
       
-
       
    story, err := parser.Parse(feed.Items[0])

     

       
1153
       
-
       
    assert.NoError(t, err)

     

       
1154
       
-
       
    assert.NotEmpty(t, story.Title)

     

       
1155
       
-
       
    assert.NotEmpty(t, story.Summary)

     

       
1156
       
-
       
    assert.Greater(t, len(story.Sources), 0)

     

       
1157
       
-
       
}

     

       
1158
       
-
       
```

     

       
1159
       
-
       


     

       
1160
       
-
       
**With Test Coves Instance:**

     

       
1161
       
-
       
```go

     

       
1162
       
-
       
func TestE2E_CreatePost(t *testing.T) {

     

       
1163
       
-
       
    if testing.Short() {

     

       
1164
       
-
       
        t.Skip("Skipping E2E test")

     

       
1165
       
-
       
    }

     

       
1166
       
-
       


     

       
1167
       
-
       
    // Create post in test community

     

       
1168
       
-
       
    post := aggregator.Post{

     

       
1169
       
-
       
        Title:   "Test Kagi News Post",

     

       
1170
       
-
       
        Content: "Test content...",

     

       
1171
       
-
       
    }

     

       
1172
       
 
       


     

       
1173
       
-
       
    err := aggregator.CreatePost(ctx, testCommunityDID, post)

     

       
1174
       
-
       
    assert.NoError(t, err)

     

       
1175
       
 
       


     

       
1176
       
-
       
    // Verify post appears in feed

     

       
1177
       
-
       
    // (requires test community setup)

     

       
1178
       
-
       
}

     

       
1179
       
-
       
```

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1180
       
 
       


     

       
1181
       
 
       
---

     

       
1182
       
 
       


     

       
1183
       
-
       
## Success Metrics

     

       
1184
       
 
       


     

       
1185
       
-
       
### Pre-Launch Checklist

     

       
1186
       
 
       


     

       
1187
       
-
       
- [ ] Aggregator service declaration published

     

       
1188
       
-
       
- [ ] DID created and configured (did:web:kagi-news.coves.social)

     

       
1189
       
-
       
- [ ] RSS feed parser handles all Kagi HTML structures

     

       
1190
       
-
       
- [ ] Deduplication prevents duplicate posts

     

       
1191
       
-
       
- [ ] Category mapping works for all configs

     

       
1192
       
-
       
- [ ] All 3 post formats render correctly

     

       
1193
       
-
       
- [ ] Attribution to Kagi News visible on all posts

     

       
1194
       
-
       
- [ ] Rate limiting prevents spam

     

       
1195
       
-
       
- [ ] Monitoring/alerting configured

     

       
1196
       
-
       
- [ ] E2E tests passing against test instance

     

       
1197
       
 
       


     

       
1198
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1199
       
 
       


     

       
1200
       
-
       
### Alpha Goals (First Week)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1201
       
 
       


     

       
1202
       
-
       
- [ ] 3+ communities using Kagi News aggregator

     

       
1203
       
-
       
- [ ] 50+ posts created successfully

     

       
1204
       
-
       
- [ ] Zero duplicate posts

     

       
1205
       
-
       
- [ ] < 5% parse errors

     

       
1206
       
-
       
- [ ] < 1% post creation failures

     

       
1207
       
-
       
- [ ] Stories posted within 15 minutes of RSS publication

     

       
1208
       
 
       


     

       
1209
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1210
       
 
       


     

       
1211
       
-
       
### Beta Goals (First Month)

     

       
1212
       
 
       


     

       
1213
       
-
       
- [ ] 10+ communities using aggregator

     

       
1214
       
-
       
- [ ] 500+ posts created

     

       
1215
       
-
       
- [ ] Community feedback positive (surveys)

     

       
1216
       
-
       
- [ ] Attribution compliance verified

     

       
1217
       
-
       
- [ ] No rate limit violations

     

       
1218
       
-
       
- [ ] < 1% error rate (parsing + posting)

     

       
1219
       
 
       


     

       
1220
       
-
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1221
       
 
       


     

       
1222
       
-
       
## Future Enhancements

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1223
       
 
       


     

       
1224
       
-
       
### Phase 2 Features

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1225
       
 
       


     

       
1226
       
-
       
**Smart Category Detection:**

     

       
1227
       
-
       
- Use LLM to suggest additional categories for stories

     

       
1228
       
-
       
- Map Kagi categories to community tags automatically

     

       
1229
       
 
       


     

       
1230
       
-
       
**Customizable Templates:**

     

       
1231
       
-
       
- Allow communities to customize post format with templates

     

       
1232
       
-
       
- Support Markdown/Handlebars templates in config

     

       
1233
       
 
       


     

       
1234
       
-
       
**Story Scoring:**

     

       
1235
       
-
       
- Prioritize high-impact stories (many sources, breaking news)

     

       
1236
       
-
       
- Delay low-priority stories to avoid flooding feed

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1237
       
 
       


     

       
1238
       
-
       
**Cross-posting Prevention:**

     

       
1239
       
-
       
- Detect when multiple communities authorize same category

     

       
1240
       
-
       
- Intelligently cross-post vs. duplicate

     

       
0
       
 
       

     

       
1241
       
 
       


     

       
1242
       
 
       
---

     

       
1243
       
 
       


     

       
1244
       
-
       
### Phase 3 Features

     

       
1245
       
 
       


     

       
1246
       
-
       
**Interactive Features:**

     

       
1247
       
-
       
- Bot responds to comments with additional sources

     

       
1248
       
-
       
- Updates megathread with new sources as story develops

     

       
1249
       
-
       


     

       
1250
       
-
       
**Analytics Dashboard:**

     

       
1251
       
-
       
- Show communities which stories get most engagement

     

       
1252
       
-
       
- Trending topics from Kagi News

     

       
1253
       
-
       
- Source diversity metrics

     

       
1254
       
-
       


     

       
1255
       
-
       
**Federation:**

     

       
1256
       
-
       
- Support other Coves instances using same aggregator

     

       
1257
       
-
       
- Shared deduplication across instances

     

       
1258
       
 
       


     

       
1259
       
 
       
---

     

       
1260
       
 
       


     

       
1261
       
-
       
## Open Questions

     

       
1262
       
 
       


     

       
1263
       
-
       
### Need to Resolve Before Launch

     

       
1264
       
 
       


     

       
1265
       
-
       
1. **Image Licensing:**

     

       
1266
       
-
       
   - ❓ Are images from Kagi proxy covered by CC BY-NC?

     

       
1267
       
-
       
   - ❓ Do we need to attribute original image sources?

     

       
1268
       
-
       
   - **Action:** Email support@kagi.com for clarification

     

       
1269
       
 
       


     

       
1270
       
-
       
2. **Hotlinking Policy:**

     

       
1271
       
-
       
   - ❓ Is embedding Kagi proxy images acceptable?

     

       
1272
       
-
       
   - ❓ Should we download and re-host?

     

       
1273
       
-
       
   - **Action:** Test in staging, monitor for issues

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1274
       
 
       


     

       
1275
       
-
       
3. **Category Discovery:**

     

       
1276
       
-
       
   - ❓ How to discover all available category feeds?

     

       
1277
       
-
       
   - ❓ Are there categories beyond world/tech/business/sports?

     

       
1278
       
-
       
   - **Action:** Scrape https://news.kagi.com/ for all .xml links

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1279
       
 
       


     

       
1280
       
-
       
4. **Attribution Format:**

     

       
1281
       
-
       
   - ❓ Is "📰 Story aggregated by Kagi News" sufficient?

     

       
1282
       
-
       
   - ❓ Do we need more prominent attribution?

     

       
1283
       
-
       
   - **Action:** Review CC BY-NC best practices

     

       
1284
       
 
       


     

       
1285
       
 
       
---

     

       
1286
       
 
       


     

       
1287
       
-
       
## References

     

       
1288
       
-
       


     

       
1289
       
-
       
- Kagi News About Page: https://news.kagi.com/about

     

       
1290
       
-
       
- Kagi News RSS Example: https://news.kagi.com/world.xml

     

       
1291
       
-
       
- Kagi Kite Public Repo: https://github.com/kagisearch/kite-public

     

       
1292
       
-
       
- CC BY-NC License: https://creativecommons.org/licenses/by-nc/4.0/

     

       
1293
       
-
       
- Parent PRD: [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       
1294
       
-
       
- Aggregator SDK: [TBD]

     

···

       
1
       
 
       
# Kagi News RSS Aggregator PRD

     

       
2
       
 
       


     

       
3
       
+
       
**Status:** ✅ Phase 1 Complete - Ready for Deployment

     

       
4
       
 
       
**Owner:** Platform Team

     

       
5
       
+
       
**Last Updated:** 2025-10-24

     

       
6
       
 
       
**Parent PRD:** [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       
7
       
+
       
**Implementation:** Python + Docker Compose

     

       
8
       
+
       


     

       
9
       
+
       
## 🎉 Implementation Complete

     

       
10
       
+
       


     

       
11
       
+
       
All core components have been implemented and tested:

     

       
12
       
+
       


     

       
13
       
+
       
- ✅ **RSS Fetcher** - Fetches feeds with retry logic and error handling

     

       
14
       
+
       
- ✅ **HTML Parser** - Extracts all structured data (summary, highlights, perspectives, quote, sources)

     

       
15
       
+
       
- ✅ **Rich Text Formatter** - Formats content with proper facets for Coves

     

       
16
       
+
       
- ✅ **State Manager** - Tracks posted stories to prevent duplicates

     

       
17
       
+
       
- ✅ **Config Manager** - Loads and validates YAML configuration

     

       
18
       
+
       
- ✅ **Coves Client** - Handles authentication and post creation

     

       
19
       
+
       
- ✅ **Main Orchestrator** - Coordinates all components

     

       
20
       
+
       
- ✅ **Comprehensive Tests** - 57 tests with 83% code coverage

     

       
21
       
+
       
- ✅ **Documentation** - README with setup and deployment instructions

     

       
22
       
+
       
- ✅ **Example Configs** - config.example.yaml and .env.example

     

       
23
       
+
       


     

       
24
       
+
       
**Test Results:**

     

       
25
       
+
       
```

     

       
26
       
+
       
57 passed, 6 skipped, 1 warning in 8.76s

     

       
27
       
+
       
Coverage: 83%

     

       
28
       
+
       
```

     

       
29
       
+
       


     

       
30
       
+
       
**Ready for:**

     

       
31
       
+
       
- Integration testing with live Coves API

     

       
32
       
+
       
- Aggregator DID creation and authorization

     

       
33
       
+
       
- Production deployment

     

       
34
       
 
       


     

       
35
       
 
       
## Overview

     

       
36
       
 
       


     
···

       
42
       
 
       
- **Rich metadata**: Categories, highlights, source links included

     

       
43
       
 
       
- **Legal & free**: CC BY-NC licensed for non-commercial use

     

       
44
       
 
       
- **Low complexity**: No LLM deduplication needed (Kagi does it)

     

       
45
       
+
       
- **Simple deployment**: Python + Docker Compose, runs alongside Coves on same instance

     

       
46
       
 
       


     

       
47
       
 
       
## Data Source: Kagi News RSS Feeds

     

       
48
       
 
       


     
···

       
74
       
 
       


     

       
75
       
 
       
**Known Categories:**

     

       
76
       
 
       
- `world.xml` - World news

     

       
77
       
+
       
- `tech.xml` - Technology

     

       
78
       
+
       
- `business.xml` - Business

     

       
79
       
 
       
- `sports.xml` - Sports (likely)

     

       
80
       
 
       
- Additional categories TBD (need to scrape homepage)

     

       
81
       
 
       


     
···

       
83
       
 
       


     

       
84
       
 
       
**Update Frequency:** One daily update (~noon UTC)

     

       
85
       
 
       


     

       
86
       
+
       
**Important Note on Domain Migration (October 2025):**

     

       
87
       
+
       
Kagi migrated their RSS feeds from `kite.kagi.com` to `news.kagi.com`. The old domain now redirects (302) to the new domain, but for reliability, always use `news.kagi.com` directly in your feed URLs. Story links within the RSS feed still reference `kite.kagi.com` as permalinks.

     

       
88
       
+
       


     

       
89
       
 
       
---

     

       
90
       
 
       


     

       
91
       
 
       
### RSS Item Schema

     
···

       
130
       
 
       
</ul>

     

       
131
       
 
       
```

     

       
132
       
 
       


     

       
133
       
+
       
**✅ Verified Feed Structure:**

     

       
134
       
+
       
Analysis of live Kagi News feeds confirms the following structure:

     

       
135
       
+
       
- **Only 3 H3 sections:** Highlights, Perspectives, Sources (no other sections like Timeline or Historical Background)

     

       
136
       
+
       
- **Historical context** is woven into the summary paragraph and highlights (not a separate section)

     

       
137
       
+
       
- **Not all stories have all sections** - Quote (blockquote) and image are optional

     

       
138
       
+
       
- **Feed contains everything shown on website** except for Timeline (which is a frontend-only feature)

     

       
139
       
+
       


     

       
140
       
 
       
**Key Features:**

     

       
141
       
 
       
- Multiple source citations inline

     

       
142
       
 
       
- Balanced perspectives from different actors

     

       
143
       
+
       
- Highlights extract key points with historical context

     

       
144
       
+
       
- Direct quotes preserved (when available)

     

       
145
       
 
       
- All sources linked with attribution

     

       
146
       
+
       
- Images from Kagi's proxy CDN

     

       
147
       
 
       


     

       
148
       
 
       
---

     

       
149
       
 
       


     
···

       
162
       
 
       
                            │ HTTP GET one job after update

     

       
163
       
 
       
                            ▼

     

       
164
       
 
       
┌─────────────────────────────────────────────────────────────┐

     

       
165
       
+
       
│  Kagi News Aggregator Service (Python + Docker Compose)    │

     

       
166
       
+
       
│  DID: did:plc:[generated-on-creation]                       │

     

       
167
       
+
       
│  Location: aggregators/kagi-news/                           │

     

       
168
       
 
       
│                                                             │

     

       
169
       
 
       
│  Components:                                                 │

     

       
170
       
+
       
│  1. RSS Fetcher: Fetches RSS feeds on schedule (feedparser) │

     

       
171
       
+
       
│  2. Item Parser: Extracts structured data from HTML (bs4)   │

     

       
172
       
+
       
│  3. Deduplication: Tracks posted items via JSON state file  │

     

       
173
       
+
       
│  4. Feed Mapper: Maps feed URLs to community handles        │

     

       
174
       
 
       
│  5. Post Formatter: Converts to Coves post format           │

     

       
175
       
+
       
│  6. Post Publisher: Calls social.coves.post.create via XRPC │

     

       
176
       
+
       
│  7. Blob Uploader: Handles image upload to ATProto          │

     

       
177
       
 
       
└─────────────────────────────────────────────────────────────┘

     

       
178
       
 
       
                            │

     

       
179
       
 
       
                            │ Authenticated XRPC calls

     
···

       
181
       
 
       
┌─────────────────────────────────────────────────────────────┐

     

       
182
       
 
       
│  Coves AppView (social.coves.post.create)                   │

     

       
183
       
 
       
│  - Validates aggregator authorization                        │

     

       
184
       
+
       
│  - Creates post with author = did:plc:[aggregator-did]      │

     

       
185
       
 
       
│  - Indexes to community feeds                                │

     

       
186
       
 
       
└─────────────────────────────────────────────────────────────┘

     

       
187
       
 
       
```

     
···

       
193
       
 
       
```json

     

       
194
       
 
       
{

     

       
195
       
 
       
  "$type": "social.coves.aggregator.service",

     

       
196
       
+
       
  "did": "did:plc:[generated-on-creation]",

     

       
197
       
 
       
  "displayName": "Kagi News Aggregator",

     

       
198
       
 
       
  "description": "Automatically posts breaking news from Kagi News RSS feeds. Kagi News aggregates multiple sources per story with balanced perspectives and comprehensive source citations.",

     

       
199
       
 
       
  "aggregatorType": "social.coves.aggregator.types#rss",

     
···

       
201
       
 
       
  "configSchema": {

     

       
202
       
 
       
    "type": "object",

     

       
203
       
 
       
    "properties": {

     

       
204
       
+
       
      "feedUrl": {

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
205
       
 
       
        "type": "string",

     

       
206
       
+
       
        "format": "uri",

     

       
207
       
+
       
        "description": "Kagi News RSS feed URL (e.g., https://news.kagi.com/world.xml)"

     

       
0
       
 
       

     

       
208
       
 
       
      }

     

       
209
       
 
       
    },

     

       
210
       
+
       
    "required": ["feedUrl"]

     

       
211
       
 
       
  },

     

       
212
       
 
       
  "sourceUrl": "https://github.com/coves-social/kagi-news-aggregator",

     

       
213
       
 
       
  "maintainer": "did:plc:coves-platform",

     

       
214
       
+
       
  "createdAt": "2025-10-23T00:00:00Z"

     

       
215
       
 
       
}

     

       
216
       
 
       
```

     

       
217
       
 
       


     

       
218
       
+
       
**Note:** The MVP implementation uses a simpler configuration model. Feed-to-community mappings are defined in the aggregator's own config file rather than per-community configuration. This allows one aggregator instance to post to multiple communities.

     

       
219
       
+
       


     

       
220
       
 
       
---

     

       
221
       
 
       


     

       
222
       
+
       
## Aggregator Configuration (MVP)

     

       
223
       
 
       


     

       
224
       
+
       
The MVP uses a simplified configuration model where the aggregator service defines feed-to-community mappings in its own config file.

     

       
225
       
 
       


     

       
226
       
+
       
### Configuration File: `config.yaml`

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
227
       
 
       


     

       
228
       
+
       
```yaml

     

       
229
       
+
       
# Aggregator credentials (from environment variables)

     

       
230
       
+
       
# AGGREGATOR_DID=did:plc:xyz...

     

       
231
       
+
       
# AGGREGATOR_PRIVATE_KEY=base64-encoded-key...

     

       
232
       
 
       


     

       
233
       
+
       
# Coves API endpoint

     

       
234
       
+
       
coves_api_url: "https://api.coves.social"

     

       
235
       
 
       


     

       
236
       
+
       
# Feed-to-community mappings

     

       
237
       
+
       
feeds:

     

       
238
       
+
       
  - name: "World News"

     

       
239
       
+
       
    url: "https://news.kagi.com/world.xml"

     

       
240
       
+
       
    community_handle: "world-news.coves.social"

     

       
241
       
+
       
    enabled: true

     

       
242
       
 
       


     

       
243
       
+
       
  - name: "Tech News"

     

       
244
       
+
       
    url: "https://news.kagi.com/tech.xml"

     

       
245
       
+
       
    community_handle: "tech.coves.social"

     

       
246
       
+
       
    enabled: true

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
247
       
 
       


     

       
248
       
+
       
  - name: "Science News"

     

       
249
       
+
       
    url: "https://news.kagi.com/science.xml"

     

       
250
       
+
       
    community_handle: "science.coves.social"

     

       
251
       
+
       
    enabled: false  # Can be disabled without removing

     

       
252
       
 
       


     

       
253
       
+
       
# Scheduling

     

       
254
       
+
       
check_interval: "24h"  # Run once daily

     

       
0
       
 
       

     

       
255
       
 
       


     

       
256
       
+
       
# Logging

     

       
257
       
+
       
log_level: "info"

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
258
       
 
       
```

     

       
259
       
 
       


     

       
260
       
+
       
**Key Decisions:**

     

       
261
       
+
       
- Uses **community handles** (not DIDs) for easier configuration - resolved at runtime

     

       
262
       
+
       
- One aggregator can post to multiple communities

     

       
263
       
+
       
- Feed mappings managed in aggregator config (not per-community config)

     

       
264
       
+
       
- No complex filtering logic in MVP - one feed = one community

     

       
265
       
 
       


     

       
266
       
 
       
---

     

       
267
       
 
       


     
···

       
272
       
 
       
```json

     

       
273
       
 
       
{

     

       
274
       
 
       
  "$type": "social.coves.post.record",

     

       
275
       
+
       
  "author": "did:plc:[aggregator-did]",

     

       
276
       
+
       
  "community": "world-news.coves.social",

     

       
277
       
 
       
  "title": "{Kagi story title}",

     

       
278
       
+
       
  "content": "{formatted content - full format for MVP}",

     

       
279
       
 
       
  "embed": {

     

       
280
       
+
       
    "$type": "social.coves.embed.external",

     

       
281
       
 
       
    "external": {

     

       
282
       
+
       
      "uri": "{Kagi story URL}",

     

       
283
       
 
       
      "title": "{story title}",

     

       
284
       
+
       
      "description": "{summary excerpt - first 200 chars}",

     

       
285
       
+
       
      "thumb": "{Kagi proxy image URL from HTML}"

     

       
286
       
 
       
    }

     

       
287
       
 
       
  },

     

       
288
       
 
       
  "federatedFrom": {

     
···

       
299
       
 
       
}

     

       
300
       
 
       
```

     

       
301
       
 
       


     

       
302
       
+
       
**MVP Notes:**

     

       
303
       
+
       
- Uses `social.coves.embed.external` for hot-linked images (no blob upload)

     

       
304
       
+
       
- Community specified as handle (resolved to DID by post creation endpoint)

     

       
305
       
+
       
- Images referenced via original Kagi proxy URLs

     

       
306
       
+
       
- "Full" format only for MVP (no format variations)

     

       
307
       
+
       
- Content uses Coves rich text with facets (not markdown)

     

       
308
       
+
       


     

       
309
       
 
       
---

     

       
310
       
 
       


     

       
311
       
+
       
### Content Formatting (MVP: "Full" Format Only)

     

       
312
       
 
       


     

       
313
       
+
       
The MVP implements a single "full" format using Coves rich text with facets:

     

       
314
       
 
       


     

       
315
       
+
       
**Plain Text Structure:**

     

       
316
       
+
       
```

     

       
317
       
 
       
{Main summary paragraph with source citations}

     

       
318
       
 
       


     

       
319
       
+
       
Highlights:

     

       
320
       
 
       
• {Bullet point 1}

     

       
321
       
 
       
• {Bullet point 2}

     

       
322
       
 
       
• ...

     

       
323
       
 
       


     

       
324
       
+
       
Perspectives:

     

       
325
       
+
       
• {Actor}: {Their perspective} (Source)

     

       
326
       
 
       
• ...

     

       
327
       
 
       


     

       
328
       
+
       
"{Notable quote}" — {Attribution}

     

       
329
       
 
       


     

       
330
       
+
       
Sources:

     

       
331
       
+
       
• {Title} - {domain}

     

       
332
       
 
       
• ...

     

       
333
       
 
       


     

       
334
       
 
       
---

     

       
335
       
+
       
📰 Story aggregated by Kagi News

     

       
336
       
 
       
```

     

       
337
       
 
       


     

       
338
       
+
       
**Rich Text Facets Applied:**

     

       
339
       
+
       
- **Bold** (`social.coves.richtext.facet#bold`) on section headers: "Highlights:", "Perspectives:", "Sources:"

     

       
340
       
+
       
- **Bold** on perspective actors

     

       
341
       
+
       
- **Italic** (`social.coves.richtext.facet#italic`) on quotes

     

       
342
       
+
       
- **Link** (`social.coves.richtext.facet#link`) on all URLs (source links, Kagi story link, perspective sources)

     

       
343
       
+
       
- Byte ranges calculated using UTF-8 byte positions

     

       
344
       
 
       


     

       
345
       
+
       
**Example with Facets:**

     

       
346
       
+
       
```json

     

       
347
       
+
       
{

     

       
348
       
+
       
  "content": "Main summary [source.com#1]\n\nHighlights:\n• Key point 1...",

     

       
349
       
+
       
  "facets": [

     

       
350
       
+
       
    {

     

       
351
       
+
       
      "index": {"byteStart": 35, "byteEnd": 46},

     

       
352
       
+
       
      "features": [{"$type": "social.coves.richtext.facet#bold"}]

     

       
353
       
+
       
    },

     

       
354
       
+
       
    {

     

       
355
       
+
       
      "index": {"byteStart": 15, "byteEnd": 26},

     

       
356
       
+
       
      "features": [{"$type": "social.coves.richtext.facet#link", "uri": "https://source.com"}]

     

       
357
       
+
       
    }

     

       
358
       
+
       
  ]

     

       
359
       
+
       
}

     

       
360
       
+
       
```

     

       
361
       
 
       


     

       
362
       
+
       
**Rationale:**

     

       
363
       
+
       
- Uses native Coves rich text format (not markdown)

     

       
364
       
+
       
- Preserves Kagi's rich multi-source analysis

     

       
365
       
+
       
- Provides maximum value to communities

     

       
366
       
+
       
- Meets CC BY-NC attribution requirements

     

       
367
       
+
       
- Additional formats ("summary", "minimal") can be added post-MVP

     

       
0
       
 
       

     

       
0
       
 
       

     

       
368
       
 
       


     

       
369
       
 
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
370
       
 
       


     

       
371
       
+
       
## Implementation Details (Python MVP)

     

       
372
       
 
       


     

       
373
       
+
       
### Technology Stack

     

       
374
       
 
       


     

       
375
       
+
       
**Language:** Python 3.11+

     

       
376
       
 
       


     

       
377
       
+
       
**Key Libraries:**

     

       
378
       
+
       
- `feedparser` - RSS/Atom parsing

     

       
379
       
+
       
- `beautifulsoup4` - HTML parsing for RSS item descriptions

     

       
380
       
+
       
- `requests` - HTTP client for fetching feeds

     

       
381
       
+
       
- `atproto` - Official ATProto Python SDK for authentication

     

       
382
       
+
       
- `pyyaml` - Configuration file parsing

     

       
383
       
+
       
- `pytest` - Testing framework

     

       
384
       
 
       


     

       
385
       
+
       
### Project Structure

     

       
386
       
 
       


     

       
0
       
 
       

     

       
0
       
 
       

     

       
387
       
 
       
```

     

       
388
       
+
       
aggregators/kagi-news/

     

       
389
       
+
       
├── Dockerfile

     

       
390
       
+
       
├── docker-compose.yml

     

       
391
       
+
       
├── requirements.txt

     

       
392
       
+
       
├── config.example.yaml

     

       
393
       
+
       
├── crontab                  # CRON schedule configuration

     

       
394
       
+
       
├── .env.example             # Environment variables template

     

       
395
       
+
       
├── scripts/

     

       
396
       
+
       
│   └── generate_did.py      # Helper to generate aggregator DID

     

       
397
       
+
       
├── src/

     

       
398
       
+
       
│   ├── main.py              # Entry point (single run, called by CRON)

     

       
399
       
+
       
│   ├── config.py            # Configuration loading and validation

     

       
400
       
+
       
│   ├── rss_fetcher.py       # RSS feed fetching with retry logic

     

       
401
       
+
       
│   ├── html_parser.py       # Parse Kagi HTML to structured data

     

       
402
       
+
       
│   ├── richtext_formatter.py # Format content with rich text facets

     

       
403
       
+
       
│   ├── atproto_client.py    # ATProto authentication and operations

     

       
404
       
+
       
│   ├── state_manager.py     # Deduplication state tracking (JSON)

     

       
405
       
+
       
│   └── models.py            # Data models (KagiStory, etc.)

     

       
406
       
+
       
├── tests/

     

       
407
       
+
       
│   ├── test_parser.py

     

       
408
       
+
       
│   ├── test_richtext_formatter.py

     

       
409
       
+
       
│   ├── test_state_manager.py

     

       
410
       
+
       
│   └── fixtures/            # Sample RSS feeds for testing

     

       
411
       
+
       
└── README.md

     

       
412
       
+
       
```

     

       
413
       
 
       


     

       
414
       
 
       
---

     

       
415
       
 
       


     

       
416
       
+
       
### Component 1: RSS Fetcher (`rss_fetcher.py`) ✅ COMPLETE

     

       
417
       
 
       


     

       
418
       
+
       
**Responsibility:** Fetch RSS feeds with retry logic and error handling

     

       
419
       
 
       


     

       
420
       
+
       
**Key Functions:**

     

       
421
       
+
       
- `fetch_feed(url: str) -> feedparser.FeedParserDict`

     

       
422
       
+
       
  - Uses `requests` with timeout (30s)

     

       
423
       
+
       
  - Retry logic: 3 attempts with exponential backoff

     

       
424
       
+
       
  - Returns parsed RSS feed or raises exception

     

       
425
       
 
       


     

       
426
       
+
       
**Error Handling:**

     

       
427
       
+
       
- Network timeouts

     

       
428
       
+
       
- Invalid XML

     

       
429
       
+
       
- HTTP errors (404, 500, etc.)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
430
       
 
       


     

       
431
       
+
       
**Implementation Status:**

     

       
432
       
+
       
- ✅ Implemented with comprehensive error handling

     

       
433
       
+
       
- ✅ Tests passing (5 tests)

     

       
434
       
+
       
- ✅ Handles retries with exponential backoff

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
435
       
 
       


     

       
436
       
 
       
---

     

       
437
       
 
       


     

       
438
       
+
       
### Component 2: HTML Parser (`html_parser.py`) ✅ COMPLETE

     

       
439
       
 
       


     

       
440
       
+
       
**Responsibility:** Extract structured data from Kagi's HTML description field

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
441
       
 
       


     

       
442
       
+
       
**Key Class:** `KagiHTMLParser`

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
443
       
 
       


     

       
444
       
+
       
**Data Model (`models.py`):**

     

       
445
       
+
       
```python

     

       
446
       
+
       
@dataclass

     

       
447
       
+
       
class KagiStory:

     

       
448
       
+
       
    title: str

     

       
449
       
+
       
    link: str

     

       
450
       
+
       
    guid: str

     

       
451
       
+
       
    pub_date: datetime

     

       
452
       
+
       
    categories: List[str]

     

       
453
       
 
       


     

       
454
       
+
       
    # Parsed from HTML

     

       
455
       
+
       
    summary: str

     

       
456
       
+
       
    highlights: List[str]

     

       
457
       
+
       
    perspectives: List[Perspective]

     

       
458
       
+
       
    quote: Optional[Quote]

     

       
459
       
+
       
    sources: List[Source]

     

       
460
       
+
       
    image_url: Optional[str]

     

       
461
       
+
       
    image_alt: Optional[str]

     

       
462
       
 
       


     

       
463
       
+
       
@dataclass

     

       
464
       
+
       
class Perspective:

     

       
465
       
+
       
    actor: str

     

       
466
       
+
       
    description: str

     

       
467
       
+
       
    source_url: str

     

       
468
       
 
       


     

       
469
       
+
       
@dataclass

     

       
470
       
+
       
class Quote:

     

       
471
       
+
       
    text: str

     

       
472
       
+
       
    attribution: str

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
473
       
 
       


     

       
474
       
+
       
@dataclass

     

       
475
       
+
       
class Source:

     

       
476
       
+
       
    title: str

     

       
477
       
+
       
    url: str

     

       
478
       
+
       
    domain: str

     

       
479
       
+
       
```

     

       
480
       
 
       


     

       
481
       
+
       
**Parsing Strategy:**

     

       
482
       
+
       
- Use BeautifulSoup to parse HTML description

     

       
483
       
+
       
- Extract sections by finding `<h3>` tags (Highlights, Perspectives, Sources)

     

       
484
       
+
       
- Handle missing sections gracefully (not all stories have all sections)

     

       
485
       
+
       
- Clean and normalize text

     

       
486
       
 
       


     

       
487
       
+
       
**Implementation Status:**

     

       
488
       
+
       
- ✅ Extracts all 3 H3 sections (Highlights, Perspectives, Sources)

     

       
489
       
+
       
- ✅ Handles optional elements (quote, image)

     

       
490
       
+
       
- ✅ Tests passing (8 tests)

     

       
491
       
+
       
- ✅ Validates against real feed data

     

       
492
       
 
       


     

       
493
       
+
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
494
       
 
       


     

       
495
       
+
       
### Component 3: State Manager (`state_manager.py`) ✅ COMPLETE

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
496
       
 
       


     

       
497
       
+
       
**Responsibility:** Track processed stories to prevent duplicates

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
498
       
 
       


     

       
499
       
+
       
**Implementation:** Simple JSON file persistence

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
500
       
 
       


     

       
501
       
+
       
**State File Format:**

     

       
502
       
+
       
```json

     

       
503
       
+
       
{

     

       
504
       
+
       
  "feeds": {

     

       
505
       
+
       
    "https://news.kagi.com/world.xml": {

     

       
506
       
+
       
      "last_successful_run": "2025-10-23T12:00:00Z",

     

       
507
       
+
       
      "posted_guids": [

     

       
508
       
+
       
        "https://kite.kagi.com/uuid1/world/123",

     

       
509
       
+
       
        "https://kite.kagi.com/uuid2/world/124"

     

       
510
       
+
       
      ]

     

       
511
       
 
       
    }

     

       
512
       
+
       
  }

     

       
0
       
 
       

     

       
513
       
 
       
}

     

       
514
       
 
       
```

     

       
515
       
 
       


     

       
516
       
+
       
**Key Functions:**

     

       
517
       
+
       
- `is_posted(feed_url: str, guid: str) -> bool`

     

       
518
       
+
       
- `mark_posted(feed_url: str, guid: str, post_uri: str)`

     

       
519
       
+
       
- `get_last_run(feed_url: str) -> Optional[datetime]`

     

       
520
       
+
       
- `update_last_run(feed_url: str, timestamp: datetime)`

     

       
521
       
 
       


     

       
522
       
+
       
**Deduplication Strategy:**

     

       
523
       
+
       
- Keep last 100 GUIDs per feed (rolling window)

     

       
524
       
+
       
- Stories older than 30 days are automatically removed

     

       
525
       
+
       
- Simple, no database needed

     

       
526
       
 
       


     

       
527
       
+
       
**Implementation Status:**

     

       
528
       
+
       
- ✅ JSON-based persistence with atomic writes

     

       
529
       
+
       
- ✅ GUID tracking with rolling window

     

       
530
       
+
       
- ✅ Tests passing (12 tests)

     

       
531
       
+
       
- ✅ Thread-safe operations

     

       
532
       
 
       


     

       
533
       
+
       
---

     

       
534
       
 
       


     

       
535
       
+
       
### Component 4: Rich Text Formatter (`richtext_formatter.py`) ✅ COMPLETE

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
536
       
 
       


     

       
537
       
+
       
**Responsibility:** Format parsed Kagi stories into Coves rich text with facets

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
538
       
 
       


     

       
539
       
+
       
**Key Function:**

     

       
540
       
+
       
- `format_full(story: KagiStory) -> dict`

     

       
541
       
+
       
  - Returns: `{"content": str, "facets": List[dict]}`

     

       
542
       
+
       
  - Builds plain text content with all sections

     

       
543
       
+
       
  - Calculates UTF-8 byte positions for facets

     

       
544
       
+
       
  - Applies bold, italic, and link facets

     

       
545
       
+
       
  - Includes all sections: summary, highlights, perspectives, quote, sources

     

       
546
       
+
       
  - Adds Kagi News attribution footer with link

     

       
0
       
 
       

     

       
547
       
 
       


     

       
548
       
+
       
**Facet Types Applied:**

     

       
549
       
+
       
- `social.coves.richtext.facet#bold` - Section headers, perspective actors

     

       
550
       
+
       
- `social.coves.richtext.facet#italic` - Quotes

     

       
551
       
+
       
- `social.coves.richtext.facet#link` - All URLs (sources, Kagi story link)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
552
       
 
       


     

       
553
       
+
       
**Key Challenge:** UTF-8 byte position calculation

     

       
554
       
+
       
- Must handle multi-byte characters correctly (emoji, non-ASCII)

     

       
555
       
+
       
- Use `str.encode('utf-8')` to get byte positions

     

       
556
       
+
       
- Test with complex characters

     

       
557
       
 
       


     

       
558
       
+
       
**Implementation Status:**

     

       
559
       
+
       
- ✅ Full rich text formatting with facets

     

       
560
       
+
       
- ✅ UTF-8 byte position calculation working correctly

     

       
561
       
+
       
- ✅ Tests passing (10 tests)

     

       
562
       
+
       
- ✅ Handles all sections: summary, highlights, perspectives, quote, sources

     

       
563
       
 
       


     

       
564
       
 
       
---

     

       
565
       
 
       


     

       
566
       
+
       
### Component 5: Coves Client (`coves_client.py`) ✅ COMPLETE

     

       
567
       
 
       


     

       
568
       
+
       
**Responsibility:** Handle authentication and post creation via Coves API

     

       
569
       
 
       


     

       
570
       
+
       
**Implementation Note:** Uses direct HTTP client instead of ATProto SDK for simplicity in MVP.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
571
       
 
       


     

       
572
       
+
       
**Key Functions:**

     

       
573
       
+
       
- `authenticate() -> dict`

     

       
574
       
+
       
  - Authenticates aggregator using credentials

     

       
575
       
+
       
  - Returns auth token for subsequent API calls

     

       
0
       
 
       

     

       
576
       
 
       


     

       
577
       
+
       
- `create_post(community_handle: str, title: str, content: str, facets: List[dict], ...) -> dict`

     

       
578
       
+
       
  - Calls Coves post creation endpoint

     

       
579
       
+
       
  - Includes aggregator authentication

     

       
580
       
+
       
  - Returns post URI and metadata

     

       
581
       
 
       


     

       
582
       
+
       
**Authentication Flow:**

     

       
583
       
+
       
- Load aggregator credentials from environment

     

       
584
       
+
       
- Authenticate with Coves API

     

       
585
       
+
       
- Store and use auth token for requests

     

       
586
       
+
       
- Handle token refresh if needed

     

       
587
       
 
       


     

       
588
       
+
       
**Implementation Status:**

     

       
589
       
+
       
- ✅ HTTP-based client implementation

     

       
590
       
+
       
- ✅ Authentication and token management

     

       
591
       
+
       
- ✅ Post creation with all required fields

     

       
592
       
+
       
- ✅ Error handling and retries

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
593
       
 
       


     

       
594
       
 
       
---

     

       
595
       
 
       


     

       
596
       
+
       
### Component 6: Config Manager (`config.py`) ✅ COMPLETE

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
597
       
 
       


     

       
598
       
+
       
**Responsibility:** Load and validate configuration from YAML and environment

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
599
       
 
       


     

       
600
       
+
       
**Key Functions:**

     

       
601
       
+
       
- `load_config(config_path: str) -> AggregatorConfig`

     

       
602
       
+
       
  - Loads YAML configuration

     

       
603
       
+
       
  - Validates structure and required fields

     

       
604
       
+
       
  - Merges with environment variables

     

       
605
       
+
       
  - Returns validated config object

     

       
606
       
 
       


     

       
607
       
+
       
**Implementation Status:**

     

       
608
       
+
       
- ✅ YAML parsing with validation

     

       
609
       
+
       
- ✅ Environment variable support

     

       
610
       
+
       
- ✅ Tests passing (3 tests)

     

       
611
       
+
       
- ✅ Clear error messages for config issues

     

       
0
       
 
       

     

       
612
       
 
       


     

       
613
       
+
       
---

     

       
0
       
 
       

     

       
614
       
 
       


     

       
615
       
+
       
### Main Orchestration (`main.py`) ✅ COMPLETE

     

       
0
       
 
       

     

       
616
       
 
       


     

       
617
       
+
       
**Responsibility:** Coordinate all components in a single execution (called by CRON)

     

       
0
       
 
       

     

       
618
       
 
       


     

       
619
       
+
       
**Flow (Single Run):**

     

       
620
       
+
       
1. Load configuration from `config.yaml`

     

       
621
       
+
       
2. Load environment variables (AGGREGATOR_DID, AGGREGATOR_PRIVATE_KEY)

     

       
622
       
+
       
3. Initialize all components (fetcher, parser, formatter, client, state)

     

       
623
       
+
       
4. For each enabled feed in config:

     

       
624
       
+
       
   a. Fetch RSS feed

     

       
625
       
+
       
   b. Parse all items

     

       
626
       
+
       
   c. Filter out already-posted items (check state)

     

       
627
       
+
       
   d. For each new item:

     

       
628
       
+
       
      - Parse HTML to structured KagiStory

     

       
629
       
+
       
      - Format post content with rich text facets

     

       
630
       
+
       
      - Build post record (with hot-linked image if present)

     

       
631
       
+
       
      - Create post via XRPC

     

       
632
       
+
       
      - Mark as posted in state

     

       
633
       
+
       
   e. Update last run timestamp

     

       
634
       
+
       
5. Save state to disk

     

       
635
       
+
       
6. Log summary (posts created, errors encountered)

     

       
636
       
+
       
7. Exit (CRON will call again on schedule)

     

       
637
       
 
       


     

       
638
       
+
       
**Error Isolation:**

     

       
639
       
+
       
- Feed-level: One feed failing doesn't stop others

     

       
640
       
+
       
- Item-level: One item failing doesn't stop feed processing

     

       
641
       
+
       
- Continue on non-fatal errors, log all failures

     

       
642
       
+
       
- Exit code 0 even with partial failures (CRON won't alert)

     

       
643
       
+
       
- Exit code 1 only on catastrophic failure (config missing, auth failure)

     

       
644
       
 
       


     

       
645
       
+
       
**Implementation Status:**

     

       
646
       
+
       
- ✅ Complete orchestration logic implemented

     

       
647
       
+
       
- ✅ Feed-level and item-level error isolation

     

       
648
       
+
       
- ✅ Structured logging throughout

     

       
649
       
+
       
- ✅ Tests passing (9 tests covering various scenarios)

     

       
650
       
+
       
- ✅ Dry-run mode for testing

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
651
       
 
       


     

       
652
       
 
       
---

     

       
653
       
 
       


     

       
654
       
+
       
## Deployment (Docker Compose with CRON)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
655
       
 
       


     

       
656
       
+
       
### Dockerfile

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
657
       
 
       


     

       
658
       
+
       
```dockerfile

     

       
659
       
+
       
FROM python:3.11-slim

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
660
       
 
       


     

       
661
       
+
       
WORKDIR /app

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
662
       
 
       


     

       
663
       
+
       
# Install cron

     

       
664
       
+
       
RUN apt-get update && apt-get install -y cron && rm -rf /var/lib/apt/lists/*

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
665
       
 
       


     

       
666
       
+
       
# Install dependencies

     

       
667
       
+
       
COPY requirements.txt .

     

       
668
       
+
       
RUN pip install --no-cache-dir -r requirements.txt

     

       
669
       
 
       


     

       
670
       
+
       
# Copy source code and scripts

     

       
671
       
+
       
COPY src/ ./src/

     

       
672
       
+
       
COPY scripts/ ./scripts/

     

       
673
       
+
       
COPY crontab /etc/cron.d/kagi-news-cron

     

       
674
       
 
       


     

       
675
       
+
       
# Set up cron

     

       
676
       
+
       
RUN chmod 0644 /etc/cron.d/kagi-news-cron && \

     

       
677
       
+
       
    crontab /etc/cron.d/kagi-news-cron && \

     

       
678
       
+
       
    touch /var/log/cron.log

     

       
679
       
 
       


     

       
680
       
+
       
# Create non-root user for security

     

       
681
       
+
       
RUN useradd --create-home appuser && \

     

       
682
       
+
       
    chown -R appuser:appuser /app && \

     

       
683
       
+
       
    chown appuser:appuser /var/log/cron.log

     

       
684
       
 
       


     

       
685
       
+
       
USER appuser

     

       
686
       
 
       


     

       
687
       
+
       
# Run cron in foreground

     

       
688
       
+
       
CMD ["cron", "-f"]

     

       
689
       
+
       
```

     

       
690
       
 
       


     

       
691
       
+
       
### Crontab Configuration (`crontab`)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
692
       
 
       


     

       
693
       
+
       
```bash

     

       
694
       
+
       
# Run Kagi News aggregator daily at 1 PM UTC (after Kagi updates around noon)

     

       
695
       
+
       
0 13 * * * cd /app && /usr/local/bin/python -m src.main >> /var/log/cron.log 2>&1

     

       
0
       
 
       

     

       
696
       
 
       


     

       
697
       
+
       
# Blank line required at end of crontab

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
698
       
 
       
```

     

       
699
       
 
       


     

       
700
       
 
       
---

     

       
701
       
 
       


     

       
702
       
+
       
### docker-compose.yml

     

       
703
       
 
       


     

       
704
       
+
       
```yaml

     

       
705
       
+
       
version: '3.8'

     

       
706
       
 
       


     

       
707
       
+
       
services:

     

       
708
       
+
       
  kagi-news-aggregator:

     

       
709
       
+
       
    build: .

     

       
710
       
+
       
    container_name: kagi-news-aggregator

     

       
711
       
+
       
    restart: unless-stopped

     

       
712
       
 
       


     

       
713
       
+
       
    environment:

     

       
714
       
+
       
      # Aggregator identity (from aggregator creation)

     

       
715
       
+
       
      - AGGREGATOR_DID=${AGGREGATOR_DID}

     

       
716
       
+
       
      - AGGREGATOR_PRIVATE_KEY=${AGGREGATOR_PRIVATE_KEY}

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
717
       
 
       


     

       
718
       
+
       
    volumes:

     

       
719
       
+
       
      # Config file (read-only)

     

       
720
       
+
       
      - ./config.yaml:/app/config.yaml:ro

     

       
721
       
+
       
      # State file (read-write for deduplication)

     

       
722
       
+
       
      - ./data/state.json:/app/data/state.json

     

       
723
       
 
       


     

       
724
       
+
       
    logging:

     

       
725
       
+
       
      driver: "json-file"

     

       
726
       
+
       
      options:

     

       
727
       
+
       
        max-size: "10m"

     

       
728
       
+
       
        max-file: "3"

     

       
729
       
 
       
```

     

       
730
       
 
       


     

       
731
       
+
       
**Environment Variables:**

     

       
732
       
+
       
- `AGGREGATOR_DID`: PLC DID created for this aggregator instance

     

       
733
       
+
       
- `AGGREGATOR_PRIVATE_KEY`: Base64-encoded private key for signing

     

       
0
       
 
       

     

       
734
       
 
       


     

       
735
       
+
       
**Volumes:**

     

       
736
       
+
       
- `config.yaml`: Feed-to-community mappings (user-editable)

     

       
737
       
+
       
- `data/state.json`: Deduplication state (managed by aggregator)

     

       
738
       
 
       


     

       
739
       
+
       
**Deployment:**

     

       
740
       
+
       
```bash

     

       
741
       
+
       
# On same host as Coves

     

       
742
       
+
       
cd aggregators/kagi-news

     

       
743
       
+
       
cp config.example.yaml config.yaml

     

       
744
       
+
       
# Edit config.yaml with your feed mappings

     

       
745
       
 
       


     

       
746
       
+
       
# Set environment variables

     

       
747
       
+
       
export AGGREGATOR_DID="did:plc:xyz..."

     

       
748
       
+
       
export AGGREGATOR_PRIVATE_KEY="base64-key..."

     

       
749
       
 
       


     

       
750
       
+
       
# Start aggregator

     

       
751
       
+
       
docker-compose up -d

     

       
0
       
 
       

     

       
0
       
 
       

     

       
752
       
 
       


     

       
753
       
+
       
# View logs

     

       
754
       
+
       
docker-compose logs -f

     

       
755
       
+
       
```

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
756
       
 
       


     

       
757
       
 
       
---

     

       
758
       
 
       


     

       
759
       
+
       
## Image Handling Strategy (MVP)

     

       
760
       
 
       


     

       
761
       
+
       
### Approach: Hot-Linked Images via External Embed

     

       
762
       
 
       


     

       
763
       
+
       
The MVP uses hot-linked images from Kagi's proxy:

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
764
       
 
       


     

       
765
       
+
       
**Flow:**

     

       
766
       
+
       
1. Extract image URL from HTML description (`https://kagiproxy.com/img/...`)

     

       
767
       
+
       
2. Include in post using `social.coves.embed.external`:

     

       
768
       
+
       
   ```json

     

       
769
       
+
       
   {

     

       
770
       
+
       
     "$type": "social.coves.embed.external",

     

       
771
       
+
       
     "external": {

     

       
772
       
+
       
       "uri": "{Kagi story URL}",

     

       
773
       
+
       
       "title": "{Story title}",

     

       
774
       
+
       
       "description": "{Summary excerpt}",

     

       
775
       
+
       
       "thumb": "{Kagi proxy image URL}"

     

       
776
       
+
       
     }

     

       
777
       
+
       
   }

     

       
778
       
+
       
   ```

     

       
779
       
+
       
3. Frontend renders image from Kagi proxy URL

     

       
780
       
 
       


     

       
781
       
+
       
**Rationale:**

     

       
782
       
+
       
- Simpler MVP implementation (no blob upload complexity)

     

       
783
       
+
       
- No storage requirements on our end

     

       
784
       
+
       
- Kagi proxy is reliable and CDN-backed

     

       
785
       
+
       
- Faster posting (no download/upload step)

     

       
786
       
+
       
- Images already properly sized and optimized

     

       
787
       
 
       


     

       
788
       
+
       
**Future Consideration:** If Kagi proxy becomes unreliable, migrate to blob storage in Phase 2.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
789
       
 
       


     

       
790
       
 
       
---

     

       
791
       
 
       


     

       
792
       
+
       
## Rate Limiting & Performance (MVP)

     

       
793
       
 
       


     

       
794
       
+
       
### Simplified Rate Strategy

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
795
       
 
       


     

       
796
       
+
       
**RSS Fetching:**

     

       
797
       
+
       
- Poll each feed once per day (~noon UTC after Kagi updates)

     

       
798
       
+
       
- No aggressive polling needed (Kagi updates daily)

     

       
799
       
+
       
- ~3-5 feeds = minimal load

     

       
0
       
 
       

     

       
0
       
 
       

     

       
800
       
 
       


     

       
801
       
+
       
**Post Creation:**

     

       
802
       
+
       
- One run per day = 5-15 posts per feed

     

       
803
       
+
       
- Total: ~15-75 posts/day across all communities

     

       
804
       
+
       
- Well within any reasonable rate limits

     

       
0
       
 
       

     

       
805
       
 
       


     

       
806
       
+
       
**Performance:**

     

       
807
       
+
       
- RSS fetch + parse: < 5 seconds per feed

     

       
808
       
+
       
- Image download + upload: < 3 seconds per image

     

       
809
       
+
       
- Post creation: < 1 second per post

     

       
810
       
+
       
- Total runtime per day: < 5 minutes

     

       
811
       
 
       


     

       
812
       
+
       
No complex rate limiting needed for MVP.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
813
       
 
       


     

       
814
       
 
       
---

     

       
815
       
 
       


     

       
816
       
+
       
## Logging & Observability (MVP)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
817
       
 
       


     

       
818
       
+
       
### Structured Logging

     

       
819
       
 
       


     

       
820
       
+
       
**Python logging module** with JSON formatter:

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
821
       
 
       


     

       
822
       
+
       
```python

     

       
823
       
+
       
import logging

     

       
824
       
+
       
import json

     

       
825
       
 
       


     

       
826
       
+
       
logging.basicConfig(

     

       
827
       
+
       
    level=logging.INFO,

     

       
828
       
+
       
    format='%(message)s'

     

       
829
       
+
       
)

     

       
830
       
 
       


     

       
831
       
+
       
logger = logging.getLogger(__name__)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
832
       
 
       


     

       
833
       
+
       
# Example structured log

     

       
834
       
+
       
logger.info(json.dumps({

     

       
835
       
+
       
    "event": "post_created",

     

       
836
       
+
       
    "feed": "world.xml",

     

       
837
       
+
       
    "story_title": "Breaking News...",

     

       
838
       
+
       
    "community": "world-news.coves.social",

     

       
839
       
+
       
    "post_uri": "at://...",

     

       
840
       
+
       
    "timestamp": "2025-10-23T12:00:00Z"

     

       
841
       
+
       
}))

     

       
842
       
+
       
```

     

       
843
       
 
       


     

       
844
       
+
       
**Key Events to Log:**

     

       
845
       
+
       
- `feed_fetched`: RSS feed successfully fetched

     

       
846
       
+
       
- `story_parsed`: Story successfully parsed from HTML

     

       
847
       
+
       
- `post_created`: Post successfully created

     

       
848
       
+
       
- `error`: Any failures (with context)

     

       
849
       
+
       
- `run_completed`: Summary of entire run

     

       
850
       
 
       


     

       
851
       
+
       
**Log Levels:**

     

       
852
       
+
       
- INFO: Successful operations

     

       
853
       
+
       
- WARNING: Retryable errors, skipped items

     

       
854
       
+
       
- ERROR: Fatal errors, failed posts

     

       
855
       
 
       


     

       
856
       
+
       
### Simple Monitoring

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
857
       
 
       


     

       
858
       
+
       
**Health Check:** Check last successful run timestamp

     

       
859
       
+
       
- If > 48 hours: alert (should run daily)

     

       
860
       
+
       
- If errors > 50% of items: investigate

     

       
861
       
 
       


     

       
862
       
+
       
**Metrics to Track (manually via logs):**

     

       
863
       
+
       
- Posts created per run

     

       
864
       
+
       
- Parse failures per run

     

       
865
       
+
       
- Post creation failures per run

     

       
866
       
+
       
- Total runtime

     

       
867
       
 
       


     

       
868
       
+
       
No complex metrics infrastructure needed for MVP - Docker logs are sufficient.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
869
       
 
       


     

       
870
       
 
       
---

     

       
871
       
 
       


     

       
872
       
+
       
## Testing Strategy ✅ COMPLETE

     

       
873
       
 
       


     

       
874
       
+
       
### Unit Tests - 57 Tests Passing (83% Coverage)

     

       
875
       
 
       


     

       
876
       
+
       
**Test Coverage by Component:**

     

       
877
       
+
       
- ✅ **RSS Fetcher** (5 tests)

     

       
878
       
+
       
  - Successful feed fetch

     

       
879
       
+
       
  - Timeout handling

     

       
880
       
+
       
  - Retry logic with exponential backoff

     

       
881
       
+
       
  - Invalid XML handling

     

       
882
       
+
       
  - Empty URL validation

     

       
883
       
 
       


     

       
884
       
+
       
- ✅ **HTML Parser** (8 tests)

     

       
885
       
+
       
  - Summary extraction

     

       
886
       
+
       
  - Image URL and alt text extraction

     

       
887
       
+
       
  - Highlights list parsing

     

       
888
       
+
       
  - Quote extraction with attribution

     

       
889
       
+
       
  - Perspectives parsing with actors and sources

     

       
890
       
+
       
  - Sources list extraction

     

       
891
       
+
       
  - Missing sections handling

     

       
892
       
+
       
  - Full story object creation

     

       
893
       
 
       


     

       
894
       
+
       
- ✅ **Rich Text Formatter** (10 tests)

     

       
895
       
+
       
  - Full format generation

     

       
896
       
+
       
  - Bold facets on headers and actors

     

       
897
       
+
       
  - Italic facets on quotes

     

       
898
       
+
       
  - Link facets on URLs

     

       
899
       
+
       
  - UTF-8 byte position calculation

     

       
900
       
+
       
  - Multi-byte character handling (emoji, special chars)

     

       
901
       
+
       
  - All sections formatted correctly

     

       
902
       
 
       


     

       
903
       
+
       
- ✅ **State Manager** (12 tests)

     

       
904
       
+
       
  - GUID tracking

     

       
905
       
+
       
  - Duplicate detection

     

       
906
       
+
       
  - Rolling window (100 GUID limit)

     

       
907
       
+
       
  - Age-based cleanup (30 days)

     

       
908
       
+
       
  - Last run timestamp tracking

     

       
909
       
+
       
  - JSON persistence

     

       
910
       
+
       
  - Atomic file writes

     

       
911
       
+
       
  - Concurrent access safety

     

       
912
       
 
       


     

       
913
       
+
       
- ✅ **Config Manager** (3 tests)

     

       
914
       
+
       
  - YAML loading and validation

     

       
915
       
+
       
  - Environment variable merging

     

       
916
       
+
       
  - Error handling for missing/invalid config

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
917
       
 
       


     

       
918
       
+
       
- ✅ **Main Orchestrator** (9 tests)

     

       
919
       
+
       
  - End-to-end flow

     

       
920
       
+
       
  - Feed-level error isolation

     

       
921
       
+
       
  - Item-level error isolation

     

       
922
       
+
       
  - Dry-run mode

     

       
923
       
+
       
  - State persistence across runs

     

       
924
       
+
       
  - Multiple feed handling

     

       
925
       
 
       


     

       
926
       
+
       
- ✅ **E2E Tests** (6 skipped - require live API)

     

       
927
       
+
       
  - Integration with Coves API (manual testing required)

     

       
928
       
+
       
  - Authentication flow

     

       
929
       
+
       
  - Post creation

     

       
930
       
 
       


     

       
931
       
+
       
**Test Results:**

     

       
932
       
+
       
```

     

       
933
       
+
       
57 passed, 6 skipped, 1 warning in 8.76s

     

       
934
       
+
       
Coverage: 83%

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
935
       
 
       
```

     

       
936
       
 
       


     

       
937
       
+
       
**Test Fixtures:**

     

       
938
       
+
       
- Real Kagi News RSS item with all sections

     

       
939
       
+
       
- Sample HTML descriptions

     

       
940
       
+
       
- Mock HTTP responses

     

       
941
       
 
       


     

       
942
       
 
       
### Integration Tests

     

       
943
       
 
       


     

       
944
       
+
       
**Manual Integration Testing Required:**

     

       
945
       
+
       
- [ ] Can authenticate with live Coves API

     

       
946
       
+
       
- [ ] Can create post via Coves API

     

       
947
       
+
       
- [ ] Can fetch real Kagi RSS feed

     

       
948
       
+
       
- [ ] Images display correctly from Kagi proxy

     

       
949
       
+
       
- [ ] State persistence works in production

     

       
950
       
+
       
- [ ] CRON scheduling works correctly

     

       
951
       
 
       


     

       
952
       
+
       
**Pre-deployment Checklist:**

     

       
953
       
+
       
- [x] All unit tests passing

     

       
954
       
+
       
- [x] Can parse real Kagi HTML

     

       
955
       
+
       
- [x] State persistence works

     

       
956
       
+
       
- [x] Config validation works

     

       
957
       
+
       
- [x] Error handling comprehensive

     

       
958
       
+
       
- [ ] Aggregator DID created

     

       
959
       
+
       
- [ ] Can authenticate with Coves API

     

       
960
       
+
       
- [ ] Docker container builds and runs

     

       
961
       
 
       


     

       
962
       
+
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
963
       
 
       


     

       
964
       
+
       
## Success Metrics

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
965
       
 
       


     

       
966
       
+
       
### ✅ Phase 1: Implementation - COMPLETE

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
967
       
 
       


     

       
968
       
+
       
- [x] All core components implemented

     

       
969
       
+
       
- [x] 57 tests passing with 83% coverage

     

       
970
       
+
       
- [x] RSS fetching and parsing working

     

       
971
       
+
       
- [x] Rich text formatting with facets

     

       
972
       
+
       
- [x] State management and deduplication

     

       
973
       
+
       
- [x] Configuration management

     

       
974
       
+
       
- [x] Comprehensive error handling

     

       
975
       
+
       
- [x] Documentation complete

     

       
976
       
 
       


     

       
977
       
+
       
### 🔄 Phase 2: Integration Testing - IN PROGRESS

     

       
978
       
 
       


     

       
979
       
+
       
- [ ] Aggregator DID created (PLC)

     

       
980
       
+
       
- [ ] Aggregator authorized in 1+ test communities

     

       
981
       
+
       
- [ ] Can authenticate with Coves API

     

       
982
       
+
       
- [ ] First post created end-to-end

     

       
983
       
+
       
- [ ] Attribution visible ("Via Kagi News")

     

       
984
       
+
       
- [ ] No duplicate posts on repeated runs

     

       
985
       
+
       
- [ ] Images display correctly

     

       
986
       
 
       


     

       
987
       
+
       
### 📋 Phase 3: Alpha Deployment (First Week)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
988
       
 
       


     

       
989
       
+
       
- [ ] Docker Compose runs successfully in production

     

       
990
       
+
       
- [ ] 2-3 communities receiving posts

     

       
991
       
+
       
- [ ] 20+ posts created successfully

     

       
992
       
+
       
- [ ] Zero duplicates

     

       
993
       
+
       
- [ ] < 10% errors (parse or post creation)

     

       
994
       
+
       
- [ ] CRON scheduling reliable

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
995
       
 
       


     

       
996
       
+
       
### 🎯 Phase 4: Beta (First Month)

     

       
0
       
 
       

     

       
997
       
 
       


     

       
998
       
+
       
- [ ] 5+ communities using aggregator

     

       
999
       
+
       
- [ ] 200+ posts created

     

       
1000
       
+
       
- [ ] Positive community feedback

     

       
1001
       
+
       
- [ ] No rate limit issues

     

       
1002
       
+
       
- [ ] < 5% error rate

     

       
1003
       
+
       
- [ ] Performance metrics tracked

     

       
1004
       
 
       


     

       
1005
       
 
       
---

     

       
1006
       
 
       


     

       
1007
       
+
       
## What's Next: Integration & Deployment

     

       
1008
       
 
       


     

       
1009
       
+
       
### Immediate Next Steps

     

       
1010
       
 
       


     

       
1011
       
+
       
1. **Create Aggregator Identity**

     

       
1012
       
+
       
   - Generate DID for aggregator

     

       
1013
       
+
       
   - Store credentials securely

     

       
1014
       
+
       
   - Test authentication with Coves API

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1015
       
 
       


     

       
1016
       
+
       
2. **Integration Testing**

     

       
1017
       
+
       
   - Test with live Coves API

     

       
1018
       
+
       
   - Verify post creation works

     

       
1019
       
+
       
   - Validate rich text rendering

     

       
1020
       
+
       
   - Check image display from Kagi proxy

     

       
1021
       
 
       


     

       
1022
       
+
       
3. **Docker Deployment**

     

       
1023
       
+
       
   - Build Docker image

     

       
1024
       
+
       
   - Test docker-compose setup

     

       
1025
       
+
       
   - Verify CRON scheduling

     

       
1026
       
+
       
   - Set up monitoring/logging

     

       
1027
       
 
       


     

       
1028
       
+
       
4. **Community Authorization**

     

       
1029
       
+
       
   - Get aggregator authorized in test community

     

       
1030
       
+
       
   - Verify authorization flow works

     

       
1031
       
+
       
   - Test posting to real community

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1032
       
 
       


     

       
1033
       
+
       
5. **Production Deployment**

     

       
1034
       
+
       
   - Deploy to production server

     

       
1035
       
+
       
   - Configure feeds for real communities

     

       
1036
       
+
       
   - Monitor first batch of posts

     

       
1037
       
+
       
   - Gather community feedback

     

       
1038
       
 
       


     

       
1039
       
+
       
### Open Questions to Resolve

     

       
1040
       
 
       


     

       
1041
       
+
       
1. **Aggregator DID Creation:**

     

       
1042
       
+
       
   - Need helper script or manual process?

     

       
1043
       
+
       
   - Where to store credentials securely?

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1044
       
 
       


     

       
1045
       
+
       
2. **Authorization Flow:**

     

       
1046
       
+
       
   - How does community admin authorize aggregator?

     

       
1047
       
+
       
   - UI flow or XRPC endpoint?

     

       
1048
       
 
       


     

       
1049
       
+
       
3. **Image Strategy:**

     

       
1050
       
+
       
   - Confirm Kagi proxy images work reliably

     

       
1051
       
+
       
   - Fallback plan if proxy becomes unreliable?

     

       
1052
       
 
       


     

       
1053
       
+
       
4. **Monitoring:**

     

       
1054
       
+
       
   - What metrics to track initially?

     

       
1055
       
+
       
   - Alerting strategy for failures?

     

       
1056
       
 
       


     

       
1057
       
+
       
---

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1058
       
 
       


     

       
1059
       
+
       
## Future Enhancements (Post-MVP)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1060
       
 
       


     

       
1061
       
+
       
### Phase 2

     

       
1062
       
+
       
- Multiple post formats (summary, minimal)

     

       
1063
       
+
       
- Per-community filtering (subcategories, min sources)

     

       
1064
       
+
       
- More sophisticated deduplication

     

       
1065
       
+
       
- Metrics dashboard

     

       
1066
       
 
       


     

       
1067
       
+
       
### Phase 3

     

       
1068
       
+
       
- Interactive features (bot responds to comments)

     

       
1069
       
+
       
- Cross-posting prevention

     

       
1070
       
+
       
- Federation support

     

       
1071
       
 
       


     

       
1072
       
 
       
---

     

       
1073
       
 
       


     

       
1074
       
+
       
## References

     

       
1075
       
 
       


     

       
1076
       
+
       
- Kagi News About: https://news.kagi.com/about

     

       
1077
       
+
       
- Kagi News RSS: https://news.kagi.com/world.xml

     

       
1078
       
+
       
- CC BY-NC License: https://creativecommons.org/licenses/by-nc/4.0/

     

       
1079
       
+
       
- Parent PRD: [PRD_AGGREGATORS.md](PRD_AGGREGATORS.md)

     

       
1080
       
+
       
- ATProto Python SDK: https://github.com/MarshalX/atproto

     

       
1081
       
+
       
- Implementation: [aggregators/kagi-news/](/aggregators/kagi-news/)

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1082
       
 
       


     

       
1083
       
 
       
---

     

       
1084
       
 
       


     

       
1085
       
+
       
## Implementation Summary

     

       
1086
       
 
       


     

       
1087
       
+
       
**Phase 1 Status:** ✅ **COMPLETE**

     

       
1088
       
 
       


     

       
1089
       
+
       
The Kagi News RSS Aggregator implementation is complete and ready for integration testing and deployment. All 7 core components have been implemented with comprehensive test coverage (57 tests, 83% coverage).

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1090
       
 
       


     

       
1091
       
+
       
**What Was Built:**

     

       
1092
       
+
       
- Complete RSS feed fetching and parsing pipeline

     

       
1093
       
+
       
- HTML parser that extracts all structured data from Kagi News feeds (summary, highlights, perspectives, quote, sources)

     

       
1094
       
+
       
- Rich text formatter with proper facets for Coves

     

       
1095
       
+
       
- State management system for deduplication

     

       
1096
       
+
       
- Configuration management with YAML and environment variables

     

       
1097
       
+
       
- HTTP client for Coves API authentication and post creation

     

       
1098
       
+
       
- Main orchestrator with robust error handling

     

       
1099
       
+
       
- Comprehensive test suite with real feed fixtures

     

       
1100
       
+
       
- Documentation and example configurations

     

       
1101
       
 
       


     

       
1102
       
+
       
**Key Findings:**

     

       
1103
       
+
       
- Kagi News RSS feeds contain only 3 structured sections (Highlights, Perspectives, Sources)

     

       
1104
       
+
       
- Historical context is woven into the summary and highlights, not a separate section

     

       
1105
       
+
       
- Timeline feature visible on Kagi website is not in the RSS feed

     

       
1106
       
+
       
- All essential data for rich posts is available in the feed

     

       
1107
       
+
       
- Feed structure is stable and well-formed

     

       
1108
       
 
       


     

       
1109
       
+
       
**Next Phase:**

     

       
1110
       
+
       
Integration testing with live Coves API, followed by alpha deployment to test communities.

     

       
0
       
 
       

     

       
0
       
 
       

     

       
1111
       
 
       


     

       
1112
       
 
       
---

     

       
1113
       
 
       


     

       
1114
       
+
       
**End of PRD - Phase 1 Implementation Complete**

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0
       
 
       

     

       
0