A community based topic aggregation platform built on atproto

Splitting CLAUDE.md into ATPROTO_GUIDE.md and PROJECT_STRUCTURE.md

+339
ATPROTO_GUIDE.md
···
+
# AT Protocol Implementation Guide
+
+
This guide provides comprehensive information about implementing AT Protocol (atproto) in the Coves platform.
+
+
## Table of Contents
+
- [Core Concepts](#core-concepts)
+
- [Architecture Overview](#architecture-overview)
+
- [Lexicons](#lexicons)
+
- [XRPC](#xrpc)
+
- [Data Storage](#data-storage)
+
- [Identity & Authentication](#identity--authentication)
+
- [Firehose & Sync](#firehose--sync)
+
- [Go Implementation Patterns](#go-implementation-patterns)
+
- [Best Practices](#best-practices)
+
+
## Core Concepts
+
+
### What is AT Protocol?
+
AT Protocol is a federated social networking protocol that enables:
+
- **Decentralized identity** - Users own their identity (DID) and can move between providers
+
- **Data portability** - Users can export and migrate their full social graph and content
+
- **Interoperability** - Different apps can interact with the same underlying data
+
+
### Key Components
+
1. **DIDs (Decentralized Identifiers)** - Persistent user identifiers (e.g., `did:plc:xyz123`)
+
2. **Handles** - Human-readable names that resolve to DIDs (e.g., `alice.bsky.social`)
+
3. **Repositories** - User data stored as signed Merkle trees in CAR files
+
4. **Lexicons** - Schema definitions for data types and API methods
+
5. **XRPC** - The RPC protocol for client-server communication
+
6. **Firehose** - Real-time event stream of repository changes
+
+
## Architecture Overview
+
+
### Two-Database Pattern
+
AT Protocol requires two distinct data stores:
+
+
#### 1. Repository Database (Source of Truth)
+
- **Purpose**: Stores user-generated content as immutable, signed records
+
- **Storage**: CAR files containing Merkle trees + PostgreSQL metadata
+
- **Access**: Through XRPC procedures that modify repositories
+
- **Properties**:
+
- Append-only (soft deletes via tombstones)
+
- Cryptographically verifiable
+
- User-controlled and portable
+
+
#### 2. AppView Database (Query Layer)
+
- **Purpose**: Denormalized, indexed data optimized for queries
+
- **Storage**: PostgreSQL with application-specific schema
+
- **Access**: Through XRPC queries (read-only)
+
- **Properties**:
+
- Eventually consistent with repositories
+
- Can be rebuilt from repository data
+
- Application-specific aggregations
+
+
### Data Flow
+
+
```
+
Write Path:
+
Client → XRPC Procedure → Service → Write Repo → CAR Store
+
+
Firehose Event
+
+
AppView Indexer
+
+
AppView Database
+
+
Read Path:
+
Client → XRPC Query → Service → Read Repo → AppView Database
+
```
+
+
## Lexicons
+
+
### What are Lexicons?
+
Lexicons are JSON schema files that define:
+
- Data types (records stored in repositories)
+
- API methods (queries and procedures)
+
- Input/output schemas for API calls
+
+
### Lexicon Structure
+
```json
+
{
+
"lexicon": 1,
+
"id": "social.coves.community.profile",
+
"defs": {
+
"main": {
+
"type": "record",
+
"key": "self",
+
"record": {
+
"type": "object",
+
"required": ["name", "createdAt"],
+
"properties": {
+
"name": {"type": "string", "maxLength": 64},
+
"description": {"type": "string", "maxLength": 256},
+
"rules": {"type": "array", "items": {"type": "string"}},
+
"createdAt": {"type": "string", "format": "datetime"}
+
}
+
}
+
}
+
}
+
}
+
```
+
+
### Lexicon Types
+
+
#### 1. Record Types
+
Define data structures stored in user repositories:
+
```json
+
{
+
"type": "record",
+
"key": "tid|rkey|literal",
+
"record": { /* schema */ }
+
}
+
```
+
+
#### 2. Query Types (Read-only)
+
Define read operations that don't modify state:
+
```json
+
{
+
"type": "query",
+
"parameters": { /* input schema */ },
+
"output": { /* response schema */ }
+
}
+
```
+
+
#### 3. Procedure Types (Write)
+
Define operations that modify repositories:
+
```json
+
{
+
"type": "procedure",
+
"input": { /* request body schema */ },
+
"output": { /* response schema */ }
+
}
+
```
+
+
### Naming Conventions
+
- Use reverse-DNS format: `social.coves.community.profile`
+
- Queries often start with `get`, `list`, or `search`
+
- Procedures often start with `create`, `update`, or `delete`
+
- Keep names descriptive but concise
+
+
## XRPC
+
+
### What is XRPC?
+
XRPC (Cross-Protocol RPC) is AT Protocol's HTTP-based RPC system:
+
- All methods live under `/xrpc/` path
+
- Method names map directly to Lexicon IDs
+
- Supports both JSON and binary data
+
+
### Request Format
+
```
+
# Query (GET)
+
GET /xrpc/social.coves.community.getCommunity?id=123
+
+
# Procedure (POST)
+
POST /xrpc/social.coves.community.createPost
+
Content-Type: application/json
+
Authorization: Bearer <token>
+
+
{"text": "Hello, Coves!"}
+
```
+
+
### Authentication
+
- Uses Bearer tokens in Authorization header
+
- Tokens are JWTs signed by the user's signing key
+
- Service auth for server-to-server calls
+
+
## Data Storage
+
+
### CAR Files
+
Content Addressable archive files store repository data:
+
- Contains IPLD blocks forming a Merkle tree
+
- Each block identified by CID (Content IDentifier)
+
- Enables cryptographic verification and efficient sync
+
+
### Record Keys (rkeys)
+
- Unique identifiers for records within a collection
+
- Can be TIDs (timestamp-based) or custom strings
+
- Must match pattern: `[a-zA-Z0-9._~-]{1,512}`
+
+
### Repository Structure
+
```
+
Repository (did:plc:user123)
+
├── social.coves.post
+
│ ├── 3kkreaz3amd27 (TID)
+
│ └── 3kkreaz3amd28 (TID)
+
├── social.coves.community.member
+
│ ├── community123
+
│ └── community456
+
└── app.bsky.actor.profile
+
└── self
+
```
+
+
## Identity & Authentication
+
+
### DIDs (Decentralized Identifiers)
+
- Permanent, unique identifiers for users
+
- Two types supported:
+
- `did:plc:*` - Hosted by PLC Directory
+
- `did:web:*` - Self-hosted
+
+
### Handle Resolution
+
Handles resolve to DIDs via:
+
1. DNS TXT record: `_atproto.alice.com → did:plc:xyz`
+
2. HTTPS well-known: `https://alice.com/.well-known/atproto-did`
+
+
### Authentication Flow
+
1. Client creates session with identifier/password
+
2. Server returns access/refresh tokens
+
3. Client uses access token for API requests
+
4. Refresh when access token expires
+
+
## Firehose & Sync
+
+
### Firehose Events
+
Real-time stream of repository changes:
+
- Commit events (creates, updates, deletes)
+
- Identity events (handle changes)
+
- Account events (status changes)
+
+
### Subscribing to Firehose
+
Connect via WebSocket to `com.atproto.sync.subscribeRepos`:
+
```
+
wss://bsky.network/xrpc/com.atproto.sync.subscribeRepos
+
```
+
+
### Processing Events
+
- Events include full record data and operation type
+
- Process events to update AppView database
+
- Handle out-of-order events with sequence numbers
+
+
## Go Implementation Patterns
+
+
### Using Indigo Library
+
Bluesky's official Go implementation provides:
+
- Lexicon code generation
+
- CAR file handling
+
- XRPC client/server
+
- Firehose subscription
+
+
### Code Generation
+
Generate Go types from Lexicons:
+
```bash
+
go run github.com/bluesky-social/indigo/cmd/lexgen \
+
--package coves \
+
--prefix social.coves \
+
--outdir api/coves \
+
lexicons/social/coves/*.json
+
```
+
+
### Repository Operations
+
```go
+
// Write to repository
+
rkey := models.GenerateTID()
+
err := repoStore.CreateRecord(ctx, userDID, "social.coves.post", rkey, &Post{
+
Text: "Hello",
+
CreatedAt: time.Now().Format(time.RFC3339),
+
})
+
+
// Read from repository
+
records, err := repoStore.ListRecords(ctx, userDID, "social.coves.post", limit, cursor)
+
```
+
+
### XRPC Handler Pattern
+
```go
+
func (s *Server) HandleGetCommunity(ctx context.Context) error {
+
// 1. Parse and validate input
+
id := xrpc.QueryParam(ctx, "id")
+
+
// 2. Call service layer
+
community, err := s.communityService.GetByID(ctx, id)
+
if err != nil {
+
return err
+
}
+
+
// 3. Return response
+
return xrpc.WriteJSONResponse(ctx, community)
+
}
+
```
+
+
## Best Practices
+
+
### 1. Lexicon Design
+
- Keep schemas focused and single-purpose
+
- Use references (`$ref`) for shared types
+
- Version carefully - Lexicons are contracts
+
- Document thoroughly with descriptions
+
+
### 2. Data Modeling
+
- Store minimal data in repositories
+
- Denormalize extensively in AppView
+
- Use record keys that are meaningful
+
- Plan for data portability
+
+
### 3. Performance
+
- Batch firehose processing
+
- Use database transactions wisely
+
- Index AppView tables appropriately
+
- Cache frequently accessed data
+
+
### 4. Error Handling
+
- Use standard XRPC error codes
+
- Provide meaningful error messages
+
- Handle network failures gracefully
+
- Implement proper retry logic
+
+
### 5. Security
+
- Validate all inputs against Lexicons
+
- Verify signatures on repository data
+
- Rate limit API endpoints
+
- Sanitize user-generated content
+
+
### 6. Federation
+
- Design for multi-instance deployment
+
- Handle remote user identities
+
- Respect instance-specific policies
+
- Plan for cross-instance data sync
+
+
## Common Patterns
+
+
### Handling User Content
+
- Always validate against Lexicon schemas
+
- Store in user's repository via CAR files
+
- Index in AppView for efficient queries
+
- Emit firehose events for subscribers
+
+
## Resources
+
+
### Official Documentation
+
- [ATProto Specifications](https://atproto.com/specs)
+
- [Lexicon Documentation](https://atproto.com/specs/lexicon)
+
- [XRPC Specification](https://atproto.com/specs/xrpc)
+
+
### Reference Implementations
+
- [Indigo (Go)](https://github.com/bluesky-social/indigo)
+
- [ATProto SDK (TypeScript)](https://github.com/bluesky-social/atproto)
+
+
### Tools
+
- [Lexicon CLI](https://github.com/bluesky-social/atproto/tree/main/packages/lex-cli)
+
- [goat CLI](https://github.com/bluesky-social/indigo/tree/main/cmd/goat)
+51 -75
CLAUDE.md
···
Project:
-
You are a distinguished developer helping build Coves, a forum like atProto social media platform (think reddit).
+
You are a distinguished developer helping build Coves, a forum like atProto social media platform (think reddit / lemmy).
Human & LLM Readability Guidelines:
- Clear Module Boundaries: Each feature is a self-contained module with explicit interfaces
···
- DB: PostgreSQL
- atProto for federation & user identities
-
## atProto Guidelines
-
- Attempt to utilize bsky built indigo packages before building atProto layer functions from scratch
+
## atProto Guidelines
+
+
For comprehensive AT Protocol implementation details, see [ATPROTO_GUIDE.md](./ATPROTO_GUIDE.md).
+
+
Key principles:
+
- Utilize Bluesky's Indigo packages before building custom atProto functionality
+
- Everything is XRPC - no separate REST API layer needed
+
- Follow the two-database pattern: Repository (CAR files) and AppView (PostgreSQL)
+
- Design for federation and data portability from the start
# Architecture Guidelines
## Required Layered Architecture
Follow this strict separation of concerns:
```
-
Handler (HTTP) → Service (Business Logic) → Repository (Data Access) → Database
+
Handler (XRPC) → Service (Business Logic) → Repository (Data Access) → Database
```
+
- Handlers: XRPC request/response only
+
- Services: Business logic, uses both write/read repos
+
- Write Repos: CAR store operations
+
- Read Repos: AppView queries
+
## Directory Structure
-
```
-
internal/
-
├── api/
-
│ ├── handlers/ # HTTP request/response handling ONLY
-
│ └── routes/ # Route definitions
-
├── core/
-
│ └── [domain]/ # Business logic, domain models, service interfaces
-
│ ├── service.go # Business logic implementation
-
│ ├── repository.go # Data access interface
-
│ └── [domain].go # Domain models
-
└── db/
-
└── postgres/ # Database implementation details
-
└── [domain]_repo.go # Repository implementations
-
```
+
+
For a detailed project structure with file-level details and implementation status, see [PROJECT_STRUCTURE.md](./PROJECT_STRUCTURE.md).
+
+
The project follows a layered architecture with clear separation between:
+
- **XRPC handlers** - atProto API layer
+
- Only handle XRPC concerns: parsing requests, formatting responses
+
- Delegate all business logic to services
+
- No direct database access
+
- **Core business logic** - Domain services and models
+
- Contains all business logic
+
- Orchestrates between write and read repositories
+
- Manages transactions and complex operations
+
- **Data repositories** - Split between CAR store writes and AppView reads
+
- **Write Repositories** (`internal/atproto/carstore/*_write_repo.go`)
+
- Modify CAR files (source of truth)
+
- **Read Repositories** (`db/appview/*_read_repo.go`)
+
- Query denormalized PostgreSQL tables
+
- Optimized for performance
## Strict Prohibitions
- **NEVER** put SQL queries in handlers
- **NEVER** import database packages in handlers
- **NEVER** pass *sql.DB directly to handlers
-
- **NEVER** mix business logic with HTTP concerns
+
- **NEVER** mix business logic with XRPC concerns
- **NEVER** bypass the service layer
-
## Required Patterns
-
-
### Handlers (HTTP Layer)
-
- Only handle HTTP concerns: parsing requests, formatting responses
-
- Delegate all business logic to services
-
- No direct database access
-
-
Example:
-
```go
-
func (h *UserHandler) CreateUser(w http.ResponseWriter, r *http.Request) {
-
var req CreateUserRequest
-
json.NewDecoder(r.Body).Decode(&req)
-
-
user, err := h.userService.CreateUser(req) // Delegate to service
-
// Handle response formatting only
-
}
-
```
-
-
### Services (Business Layer)
-
- Contain all business logic and validation
-
- Use repository interfaces, never concrete implementations
-
- Handle transactions and complex operations
-
-
Example:
-
```go
-
type UserService struct {
-
userRepo UserRepository // Interface, not concrete type
-
}
-
```
-
-
### Repositories (Data Layer)
-
- Define interfaces in core/[domain]/
-
- Implement in db/postgres/
-
- Handle all SQL queries and database operations
-
-
Example:
-
```go
-
// Interface in core/users/repository.go
-
type UserRepository interface {
-
Create(user User) (*User, error)
-
GetByID(id int) (*User, error)
-
}
-
-
// Implementation in db/postgres/user_repo.go
-
type PostgresUserRepo struct {
-
db *sql.DB
-
}
-
```
-
## Testing Requirements
- Services must be easily mockable (use interfaces)
- Integration tests should test the full stack
···
2. Generate test file with failing tests
3. Generate implementation to pass tests
4. Generate handler with tests
-
5. Update routes in api/routes/
+
5. Update routes in xrpc/routes/
### Refactoring Checklist
Before considering a feature complete:
···
- Never modify existing migrations
- Always provide rollback migrations
-
-
## Dependency Injection
- Use constructor functions for all components
- Pass interfaces, not concrete types
···
Example dependency wiring:
```go
// main.go
-
userRepo := postgres.NewUserRepository(db)
-
userService := users.NewUserService(userRepo)
-
userHandler := handlers.NewUserHandler(userService)
+
userWriteRepo := carstore.NewUserWriteRepository(carStore)
+
userReadRepo := appview.NewUserReadRepository(db)
+
userService := users.NewUserService(userWriteRepo, userReadRepo)
+
userHandler := xrpc.NewUserHandler(userService)
```
## Error Handling
···
- Never expose internal error details in API responses
### Context7 Usage Guidelines:
-
- Always check Context7 for best practices before implementing external integrations
+
- Always check Context7 for best practices before implementing external integrations and packages
- Use Context7 to understand proper error handling patterns for specific libraries
- Reference Context7 for testing patterns with external dependencies
- Consult Context7 for proper configuration patterns
+
+
## XRPC Implementation
+
+
For detailed XRPC patterns and Lexicon examples, see [ATPROTO_GUIDE.md](./ATPROTO_GUIDE.md#xrpc).
+
+
### Key Points
+
- All client interactions go through XRPC endpoints
+
- Handlers validate against Lexicon schemas automatically
+
- Queries are read-only, procedures modify repositories
+
- Every endpoint must have a corresponding Lexicon definition
+76
PROJECT_STRUCTURE.md
···
+
# Coves Project Structure
+
+
This document provides an overview of the Coves project directory structure, following atProto architecture patterns.
+
+
**Legend:**
+
- † = Planned but not yet implemented
+
- 🔒 = Security-sensitive files
+
+
```
+
Coves/
+
├── CLAUDE.md # Project guidelines and architecture decisions
+
├── ATPROTO_GUIDE.md # Comprehensive AT Protocol implementation guide
+
├── PROJECT_STRUCTURE.md # This file - project structure overview
+
├── LICENSE # Project license
+
├── README.md # Project overview and setup instructions
+
├── go.mod # Go module definition
+
├── go.sum # Go module checksums
+
+
├── cmd/ # Application entrypoints
+
├── internal/ # Private application code
+
│ ├── xrpc/ † # XRPC handlers (atProto API layer)
+
│ ├── api/ # Traditional HTTP endpoints (minimal)
+
│ ├── core/ # Business logic and domain models
+
│ ├── atproto/ # atProto-specific implementations
+
│ └── config/ † # Configuration management
+
+
├── db/ # Database layer
+
│ ├── appview/ † # AppView PostgreSQL queries
+
│ ├── postgres/ # Legacy/non-atProto database operations
+
│ ├── migrations/ # Database migrations
+
│ ├── local_dev_db_compose/ # Local development database
+
│ └── test_db_compose/ # Test database setup
+
+
├── pkg/ # Public packages (can be imported by external projects)
+
├── data/ # Runtime data storage
+
│ └── carstore/ 🔒 # CAR file storage directory
+
+
├── scripts/ # Development and deployment scripts
+
├── tests/ # Integration and e2e tests
+
├── docs/ † # Additional documentation
+
├── local_dev_data/ # Local development data
+
├── test_db_data/ # Test database seed data
+
└── build/ † # Build artifacts
+
```
+
+
## Implementation Status
+
+
### Completed ✓
+
- Basic repository structure
+
- User domain models
+
- CAR store foundation
+
- Lexicon schemas
+
- Database migrations
+
+
### In Progress 🚧
+
- Repository service implementation
+
- User service
+
- Basic authentication
+
+
### Planned 📋
+
- XRPC handlers
+
- AppView indexer
+
- Firehose implementation
+
- Community features
+
- Moderation system
+
- Feed algorithms
+
+
## Development Guidelines
+
+
For detailed implementation guidelines, see [CLAUDE.md](./CLAUDE.md) and [ATPROTO_GUIDE.md](./ATPROTO_GUIDE.md).
+
+
1. **Start with Lexicons**: Define data schemas first
+
2. **Implement Core Domain**: Create models and interfaces
+
3. **Build Services**: Implement business logic
+
4. **Add Repositories**: Create data access layers
+
5. **Wire XRPC**: Connect handlers last