A community based topic aggregation platform built on atproto

feat(kagi): add automated registration script and docs

Add Kagi-specific automated registration script and update README.

Changes:
- Move setup-kagi-aggregator.sh to kagi-news/scripts/setup.sh
- Add comprehensive Registration section to README
- Document automated vs manual setup options
- Explain registration workflow and requirements
- Update project structure to reflect new scripts

The setup script automates all 4 registration steps:
1. PDS account creation
2. .well-known file generation
3. Coves registration via XRPC
4. Service declaration creation

This makes the Kagi aggregator self-contained and ready to be
split into its own repository.

๐Ÿค– Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

Changed files
+383 -1
aggregators
kagi-news
+188 -1
aggregators/kagi-news/README.md
···
โ”‚ โ”œโ”€โ”€ sample_rss_item.xml
โ”‚ โ””โ”€โ”€ world.xml
โ”œโ”€โ”€ scripts/
-
โ”‚ โ””โ”€โ”€ generate_did.py # Helper to generate aggregator DID (TODO)
+
โ”‚ โ””โ”€โ”€ setup.sh # Automated Coves registration script
+
โ”œโ”€โ”€ Dockerfile # Docker image definition
+
โ”œโ”€โ”€ docker-compose.yml # Docker Compose configuration
+
โ”œโ”€โ”€ docker-entrypoint.sh # Container entrypoint script
+
โ”œโ”€โ”€ .dockerignore # Docker build exclusions
โ”œโ”€โ”€ requirements.txt # Python dependencies
โ”œโ”€โ”€ config.example.yaml # Example configuration
โ”œโ”€โ”€ .env.example # Environment variables template
···
โ””โ”€โ”€ README.md
```
+
## Registration with Coves
+
+
Before running the aggregator, you must register it with a Coves instance. This creates a DID for your aggregator and registers it with Coves.
+
+
### Quick Setup (Automated)
+
+
The automated setup script handles the entire registration process:
+
+
```bash
+
cd scripts
+
chmod +x setup.sh
+
./setup.sh
+
```
+
+
This will:
+
1. **Create a PDS account** for your aggregator (generates a DID)
+
2. **Generate `.well-known/atproto-did`** file for domain verification
+
3. **Pause for manual upload** - you'll upload the file to your web server
+
4. **Register with Coves** instance via XRPC
+
5. **Create service declaration** record (indexed by Jetstream)
+
+
**Manual step required:** During the process, you'll need to upload the `.well-known/atproto-did` file to your domain so it's accessible at `https://yourdomain.com/.well-known/atproto-did`.
+
+
After completion, you'll have a `kagi-aggregator-config.env` file with:
+
- Aggregator DID and credentials
+
- Access/refresh JWTs
+
- Service declaration URI
+
+
**Keep this file secure!** It contains your aggregator's credentials.
+
+
### Manual Setup (Step-by-step)
+
+
Alternatively, use the generic setup scripts from the main Coves repo for more control:
+
+
```bash
+
# From the Coves project root
+
cd scripts/aggregator-setup
+
+
# Follow the 4-step process
+
./1-create-pds-account.sh
+
./2-setup-wellknown.sh
+
./3-register-with-coves.sh
+
./4-create-service-declaration.sh
+
```
+
+
See [scripts/aggregator-setup/README.md](../../scripts/aggregator-setup/README.md) for detailed documentation on each step.
+
+
### What Happens During Registration?
+
+
1. **PDS Account Creation**: Your aggregator gets a `did:plc:...` identifier
+
2. **Domain Verification**: Proves you control your aggregator's domain
+
3. **Coves Registration**: Inserts your DID into the Coves instance's `users` table
+
4. **Service Declaration**: Creates a record that gets indexed into the `aggregators` table
+
5. **Ready for Authorization**: Community moderators can now authorize your aggregator
+
+
Once registered and authorized by a community, your aggregator can post content.
+
## Setup
### Prerequisites
- Python 3.11+
- python3-venv package (`apt install python3.12-venv`)
+
- **Completed registration** (see above)
### Installation
···
# Run with coverage
pytest --cov=src --cov-report=html
```
+
+
## Deployment
+
+
### Docker Deployment (Recommended for Production)
+
+
The easiest way to deploy the Kagi aggregator is using Docker. The cron job runs inside the container automatically.
+
+
#### Prerequisites
+
+
- Docker and Docker Compose installed
+
- Completed registration (you have `.env` with credentials)
+
- `config.yaml` configured with your feed mappings
+
+
#### Quick Start
+
+
1. **Configure your environment:**
+
```bash
+
# Copy and edit configuration
+
cp config.example.yaml config.yaml
+
cp .env.example .env
+
+
# Edit .env with your aggregator credentials
+
nano .env
+
```
+
+
2. **Start the aggregator:**
+
```bash
+
docker compose up -d
+
```
+
+
3. **View logs:**
+
```bash
+
docker compose logs -f
+
```
+
+
4. **Stop the aggregator:**
+
```bash
+
docker compose down
+
```
+
+
#### Configuration
+
+
The `docker-compose.yml` file supports these environment variables:
+
+
- **`AGGREGATOR_HANDLE`** (required): Your aggregator's handle
+
- **`AGGREGATOR_PASSWORD`** (required): Your aggregator's password
+
- **`COVES_API_URL`** (optional): Override Coves API endpoint (defaults to `https://api.coves.social`)
+
- **`RUN_ON_STARTUP`** (optional): Set to `true` to run immediately on container start (useful for testing)
+
+
#### Testing the Setup
+
+
Run the aggregator immediately without waiting for cron:
+
+
```bash
+
# Run once and exit
+
docker compose run --rm kagi-aggregator python -m src.main
+
+
# Or set RUN_ON_STARTUP=true in .env and restart
+
docker compose restart
+
```
+
+
#### Production Deployment
+
+
For production, consider:
+
+
1. **Using Docker Secrets** for credentials:
+
```yaml
+
secrets:
+
aggregator_credentials:
+
file: ./secrets/aggregator.env
+
```
+
+
2. **Setting up log rotation** (already configured in docker-compose.yml):
+
- Max size: 10MB per file
+
- Max files: 3
+
+
3. **Monitoring health checks:**
+
```bash
+
docker inspect --format='{{.State.Health.Status}}' kagi-news-aggregator
+
```
+
+
4. **Auto-restart on failure** (already enabled with `restart: unless-stopped`)
+
+
#### Viewing Cron Logs
+
+
```bash
+
# Follow cron execution logs
+
docker compose logs -f kagi-aggregator
+
+
# View last 100 lines
+
docker compose logs --tail=100 kagi-aggregator
+
```
+
+
#### Updating the Aggregator
+
+
```bash
+
# Pull latest code
+
git pull
+
+
# Rebuild and restart
+
docker compose up -d --build
+
```
+
+
### Manual Deployment (Alternative)
+
+
If you prefer running without Docker, use the traditional approach:
+
+
1. **Install dependencies:**
+
```bash
+
python3 -m venv venv
+
source venv/bin/activate
+
pip install -r requirements.txt
+
```
+
+
2. **Configure crontab:**
+
```bash
+
# Edit the crontab file with your paths
+
# Then install it:
+
crontab crontab
+
```
+
+
3. **Verify cron is running:**
+
```bash
+
crontab -l
+
```
## Development Status
+195
aggregators/kagi-news/scripts/setup.sh
···
+
#!/bin/bash
+
+
# Script: setup-kagi-aggregator.sh
+
# Purpose: Complete setup script for Kagi News RSS aggregator
+
#
+
# This is a reference implementation showing automated setup for a specific aggregator.
+
# Other aggregator developers can use this as a template.
+
+
set -e
+
+
echo "================================================"
+
echo "Kagi News RSS Aggregator - Automated Setup"
+
echo "================================================"
+
echo ""
+
+
# Configuration for Kagi aggregator
+
AGGREGATOR_NAME="kagi-news-bot"
+
DISPLAY_NAME="Kagi News RSS"
+
DESCRIPTION="Aggregates tech news from Kagi RSS feeds and posts to relevant communities"
+
SOURCE_URL="https://github.com/coves-social/kagi-aggregator"
+
+
# Check if config already exists
+
if [ -f "kagi-aggregator-config.env" ]; then
+
echo "Configuration file already exists. Loading existing configuration..."
+
source kagi-aggregator-config.env
+
SKIP_ACCOUNT_CREATION=true
+
else
+
SKIP_ACCOUNT_CREATION=false
+
fi
+
+
# Get runtime configuration
+
if [ "$SKIP_ACCOUNT_CREATION" = false ]; then
+
read -p "Enter PDS URL (default: https://bsky.social): " PDS_URL
+
PDS_URL=${PDS_URL:-https://bsky.social}
+
+
read -p "Enter email for bot account: " EMAIL
+
read -sp "Enter password for bot account: " PASSWORD
+
echo ""
+
+
# Generate handle
+
TIMESTAMP=$(date +%s)
+
HANDLE="$AGGREGATOR_NAME-$TIMESTAMP.bsky.social"
+
+
echo ""
+
echo "Creating PDS account..."
+
echo "Handle: $HANDLE"
+
+
# Create account
+
RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.server.createAccount" \
+
-H "Content-Type: application/json" \
+
-d "{
+
\"handle\": \"$HANDLE\",
+
\"email\": \"$EMAIL\",
+
\"password\": \"$PASSWORD\"
+
}")
+
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
+
echo "โœ— Error creating account:"
+
echo "$RESPONSE" | jq '.'
+
exit 1
+
fi
+
+
DID=$(echo "$RESPONSE" | jq -r '.did')
+
ACCESS_JWT=$(echo "$RESPONSE" | jq -r '.accessJwt')
+
REFRESH_JWT=$(echo "$RESPONSE" | jq -r '.refreshJwt')
+
+
echo "โœ“ Account created: $DID"
+
+
# Save configuration
+
cat > kagi-aggregator-config.env <<EOF
+
# Kagi Aggregator Configuration
+
AGGREGATOR_DID="$DID"
+
AGGREGATOR_HANDLE="$HANDLE"
+
AGGREGATOR_PDS_URL="$PDS_URL"
+
AGGREGATOR_EMAIL="$EMAIL"
+
AGGREGATOR_PASSWORD="$PASSWORD"
+
AGGREGATOR_ACCESS_JWT="$ACCESS_JWT"
+
AGGREGATOR_REFRESH_JWT="$REFRESH_JWT"
+
EOF
+
+
echo "โœ“ Configuration saved to kagi-aggregator-config.env"
+
fi
+
+
# Get domain and Coves instance
+
read -p "Enter aggregator domain (e.g., kagi-news.example.com): " DOMAIN
+
read -p "Enter Coves instance URL (default: https://api.coves.social): " COVES_URL
+
COVES_URL=${COVES_URL:-https://api.coves.social}
+
+
# Setup .well-known
+
echo ""
+
echo "Setting up .well-known/atproto-did..."
+
mkdir -p .well-known
+
echo "$DID" > .well-known/atproto-did
+
echo "โœ“ Created .well-known/atproto-did"
+
+
echo ""
+
echo "================================================"
+
echo "IMPORTANT: Manual Step Required"
+
echo "================================================"
+
echo ""
+
echo "Upload the .well-known directory to your web server at:"
+
echo " https://$DOMAIN/.well-known/atproto-did"
+
echo ""
+
read -p "Press Enter when the file is uploaded and accessible..."
+
+
# Verify .well-known
+
echo ""
+
echo "Verifying .well-known/atproto-did..."
+
WELLKNOWN_CONTENT=$(curl -s "https://$DOMAIN/.well-known/atproto-did" || echo "ERROR")
+
+
if [ "$WELLKNOWN_CONTENT" != "$DID" ]; then
+
echo "โœ— Error: .well-known/atproto-did not accessible or contains wrong DID"
+
echo " Expected: $DID"
+
echo " Got: $WELLKNOWN_CONTENT"
+
exit 1
+
fi
+
+
echo "โœ“ .well-known/atproto-did verified"
+
+
# Register with Coves
+
echo ""
+
echo "Registering with Coves instance..."
+
RESPONSE=$(curl -s -X POST "$COVES_URL/xrpc/social.coves.aggregator.register" \
+
-H "Content-Type: application/json" \
+
-d "{
+
\"did\": \"$DID\",
+
\"domain\": \"$DOMAIN\"
+
}")
+
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
+
echo "โœ— Registration failed:"
+
echo "$RESPONSE" | jq '.'
+
exit 1
+
fi
+
+
echo "โœ“ Registered with Coves"
+
+
# Create service declaration
+
echo ""
+
echo "Creating service declaration..."
+
SERVICE_RECORD=$(cat <<EOF
+
{
+
"\$type": "social.coves.aggregator.service",
+
"did": "$DID",
+
"displayName": "$DISPLAY_NAME",
+
"description": "$DESCRIPTION",
+
"sourceUrl": "$SOURCE_URL",
+
"createdAt": "$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
+
}
+
EOF
+
)
+
+
RESPONSE=$(curl -s -X POST "$PDS_URL/xrpc/com.atproto.repo.createRecord" \
+
-H "Authorization: Bearer $ACCESS_JWT" \
+
-H "Content-Type: application/json" \
+
-d "{
+
\"repo\": \"$DID\",
+
\"collection\": \"social.coves.aggregator.service\",
+
\"rkey\": \"self\",
+
\"record\": $SERVICE_RECORD
+
}")
+
+
if echo "$RESPONSE" | jq -e '.error' > /dev/null 2>&1; then
+
echo "โœ— Failed to create service declaration:"
+
echo "$RESPONSE" | jq '.'
+
exit 1
+
fi
+
+
RECORD_URI=$(echo "$RESPONSE" | jq -r '.uri')
+
echo "โœ“ Service declaration created: $RECORD_URI"
+
+
# Save final configuration
+
cat >> kagi-aggregator-config.env <<EOF
+
+
# Setup completed on $(date)
+
AGGREGATOR_DOMAIN="$DOMAIN"
+
COVES_INSTANCE_URL="$COVES_URL"
+
SERVICE_DECLARATION_URI="$RECORD_URI"
+
EOF
+
+
echo ""
+
echo "================================================"
+
echo "โœ“ Kagi Aggregator Setup Complete!"
+
echo "================================================"
+
echo ""
+
echo "Configuration saved to: kagi-aggregator-config.env"
+
echo ""
+
echo "Your aggregator is now registered and ready to use."
+
echo ""
+
echo "Next steps:"
+
echo "1. Start your aggregator bot: npm start (or appropriate command)"
+
echo "2. Community moderators can authorize your aggregator"
+
echo "3. Once authorized, your bot can start posting"
+
echo ""
+
echo "See docs/aggregators/SETUP_GUIDE.md for more information"