Manage Atom feeds in a persistent git repository
anil.recoil.org
1# Thicket
2
3A modern CLI tool for persisting Atom/RSS feeds in Git repositories, designed to enable distributed webblog comment structures.
4
5## Features
6
7- **Feed Auto-Discovery**: Automatically extracts user metadata from Atom/RSS feeds
8- **Git Storage**: Stores feed entries in a Git repository with full history
9- **Duplicate Management**: Manual curation of duplicate entries across feeds
10- **Modern CLI**: Built with Typer and Rich for beautiful terminal output
11- **Comprehensive Parsing**: Supports RSS 0.9x, RSS 1.0, RSS 2.0, and Atom feeds
12- **Cron-Friendly**: Designed for scheduled execution
13
14## Installation
15
16```bash
17# Install from source
18pip install -e .
19
20# Or install with dev dependencies
21pip install -e .[dev]
22```
23
24## Quick Start
25
261. **Initialize a new thicket repository:**
27```bash
28thicket init ./my-feeds
29```
30
312. **Add a user with their feed:**
32```bash
33thicket add user "alice" --feed "https://alice.example.com/feed.xml"
34```
35
363. **Sync feeds to download entries:**
37```bash
38thicket sync --all
39```
40
414. **List users and feeds:**
42```bash
43thicket list users
44thicket list feeds
45thicket list entries
46```
47
48## Commands
49
50### Initialize
51```bash
52thicket init <git-store-path> [--cache-dir <path>] [--config <config-file>]
53```
54
55### Add Users and Feeds
56```bash
57# Add user with auto-discovery
58thicket add user "username" --feed "https://example.com/feed.xml"
59
60# Add user with manual metadata
61thicket add user "username" \
62 --feed "https://example.com/feed.xml" \
63 --email "user@example.com" \
64 --homepage "https://example.com" \
65 --display-name "User Name"
66
67# Add additional feed to existing user
68thicket add feed "username" "https://example.com/other-feed.xml"
69```
70
71### Sync Feeds
72```bash
73# Sync all users
74thicket sync --all
75
76# Sync specific user
77thicket sync --user "username"
78
79# Dry run (preview changes)
80thicket sync --all --dry-run
81```
82
83### List Information
84```bash
85# List all users
86thicket list users
87
88# List all feeds
89thicket list feeds
90
91# List feeds for specific user
92thicket list feeds --user "username"
93
94# List recent entries
95thicket list entries --limit 20
96
97# List entries for specific user
98thicket list entries --user "username"
99```
100
101### Manage Duplicates
102```bash
103# List duplicate mappings
104thicket duplicates list
105
106# Mark entries as duplicates
107thicket duplicates add "https://example.com/dup" "https://example.com/canonical"
108
109# Remove duplicate mapping
110thicket duplicates remove "https://example.com/dup"
111```
112
113## Configuration
114
115Thicket uses a YAML configuration file (default: `thicket.yaml`):
116
117```yaml
118git_store: ./feeds-repo
119cache_dir: ~/.cache/thicket
120users:
121 - username: alice
122 feeds:
123 - https://alice.example.com/feed.xml
124 email: alice@example.com
125 homepage: https://alice.example.com
126 display_name: Alice
127```
128
129## Git Repository Structure
130
131```
132feeds-repo/
133├── index.json # User directory index
134├── duplicates.json # Duplicate entry mappings
135├── alice/
136│ ├── metadata.json # User metadata
137│ ├── entry_id_1.json # Feed entries
138│ └── entry_id_2.json
139└── bob/
140 └── ...
141```
142
143## Development
144
145### Setup
146```bash
147# Install in development mode
148pip install -e .[dev]
149
150# Run tests
151pytest
152
153# Run linting
154ruff check src/
155black --check src/
156
157# Run type checking
158mypy src/
159```
160
161### Architecture
162
163- **CLI**: Modern interface with Typer and Rich
164- **Feed Processing**: Universal parsing with feedparser
165- **Git Storage**: Structured storage with GitPython
166- **Data Models**: Pydantic for validation and serialization
167- **Async HTTP**: httpx for efficient feed fetching
168
169## Use Cases
170
171- **Blog Aggregation**: Collect and archive blog posts from multiple sources
172- **Comment Networks**: Enable distributed commenting systems
173- **Feed Archival**: Preserve feed history beyond typical feed depth limits
174- **Content Curation**: Manage and deduplicate content across feeds
175
176## License
177
178MIT License - see LICENSE file for details.