code
Clone this repository
https://tangled.org/bretton.dev/coves
git@knot.bretton.dev:bretton.dev/coves
For self-hosted knots, clone URLs may differ based on your setup.
Update external embed lexicon to use proper nested structure with dedicated
external object, aligning with atproto conventions and enabling better validation.
**Schema Changes:**
1. Main object now requires "external" property (was flat structure)
2. Add dedicated "external" definition with link metadata
3. Update embedType known values:
- OLD: ["article", "image", "video-stream"]
- NEW: ["article", "image", "video", "website"]
- Removed "video-stream" (use "video" instead)
- Added "website" for generic pages
**Before (flat structure):**
```json
{
"$type": "social.coves.embed.external",
"uri": "https://example.com",
"title": "Example",
"thumb": {...}
}
```
**After (nested structure):**
```json
{
"$type": "social.coves.embed.external",
"external": {
"uri": "https://example.com",
"title": "Example",
"thumb": {...}
}
}
```
**Rationale:**
- Follows atproto pattern (app.bsky.embed.external uses same structure)
- Enables future extensibility (can add external-level metadata)
- Clearer separation between embed type and embedded content
- Better validation with required "external" property
**embedType Values:**
- "article": Blog posts, news articles (rich text content)
- "image": Image galleries, photos (visual content)
- "video": Video embeds from Streamable, YouTube, etc.
- "website": Generic web pages without specific type
This aligns our lexicon with atproto best practices and prepares for
potential federation with other atproto implementations.
Breaking change: Clients must update to use nested structure.
Transform blob references to direct PDS URLs in feed responses, enabling
clients to fetch thumbnails without complex blob resolution logic.
**Blob Transform Module:**
- TransformBlobRefsToURLs: Convert blob refs → PDS URLs in-place
- transformThumbToURL: Extract CID and build getBlob URL
- Handles external embeds only (social.coves.embed.external)
- Graceful handling of missing/malformed data
**Transform Logic:**
```go
// Before (blob ref in database):
"thumb": {
"$type": "blob",
"ref": {"$link": "bafyrei..."},
"mimeType": "image/jpeg",
"size": 52813
}
// After (URL string in API response):
"thumb": "http://pds.example.com/xrpc/com.atproto.sync.getBlob?did=did:plc:community&cid=bafyrei..."
```
**Repository Updates:**
- Add community_pds_url to all feed queries (feed_repo_base.go)
- Include PDSURL in PostView.Community for transformation
- Apply to: GetCommunityFeed, GetTimeline, GetDiscover
**Handler Updates:**
- Call TransformBlobRefsToURLs before returning posts
- Applies to: social.coves.feed.getCommunityFeed
- Applies to: social.coves.feed.getTimeline
- Applies to: social.coves.feed.getDiscover
**Comprehensive Tests** (13 test cases):
- Valid blob ref → URL transformation
- Missing thumb (no-op)
- Already-transformed URL (no-op)
- Nil post/community (no-op)
- Missing/empty PDS URL (no-op)
- Malformed blob refs (graceful)
- Non-external embed types (ignored)
**Why This Matters:**
Clients receive ready-to-use image URLs instead of blob references,
simplifying rendering and eliminating need for CID resolution logic.
Works seamlessly with federated communities (each has own PDS URL).
Fixes client-side rendering for external embeds with thumbnails.
Wire unfurl and blob services into the post creation pipeline, enabling
automatic enhancement of external embeds with rich metadata and thumbnails.
**Post Service Integration:**
- Add optional BlobService and UnfurlService dependencies
- Update constructor to accept blob/unfurl services (nil-safe)
- Add ThumbnailURL field to CreatePostRequest for client-provided URLs
- Add PDSURL to CommunityRef for blob URL transformation (internal only)
**Server Main Changes:**
- Initialize unfurl repository with PostgreSQL
- Initialize blob service with default PDS URL
- Initialize unfurl service with:
- 10s timeout for HTTP fetches
- 24h cache TTL
- CovesBot/1.0 user agent
- Pass blob and unfurl services to post service constructor
**Flow:**
```
Client POST → CreateHandler
↓
PostService.Create() [external embed detected]
↓ (if no thumb provided)
UnfurlService.UnfurlURL() [fetch oEmbed/OpenGraph]
↓ (cache miss)
HTTP fetch → oEmbed provider / HTML parser
↓ (thumbnail URL found)
BlobService.UploadBlobFromURL() [download & upload to PDS]
↓
com.atproto.repo.uploadBlob → PDS
↓ (returns BlobRef with CID)
Embed enriched with thumb blob → Write to PDS
```
**Interface Documentation:**
- Added comments explaining optional blob/unfurl service injection
- Unfurl service auto-enriches external embeds when provided
- Blob service uploads thumbnails from unfurled URLs
This is the core integration that enables the full unfurling feature.
The actual unfurl logic in posts/service.go will be implemented separately.
Implement blob upload service to fetch images from URLs and upload them to
PDS as atproto blobs, enabling proper thumbnail storage for external embeds.
**Service Features:**
- UploadBlobFromURL: Fetch image from URL → validate → upload to PDS
- UploadBlob: Upload raw binary data to PDS with authentication
- Size limit: 1MB per image (atproto recommendation)
- Supported MIME types: image/jpeg, image/png, image/webp
- MIME type normalization (image/jpg → image/jpeg)
- Timeout handling (10s for fetch, 30s for upload)
**Security & Validation:**
- Input validation (empty checks, nil guards)
- Size validation before network calls
- MIME type validation before reading data
- HTTP status code checking with sanitized error logs
- Proper error wrapping for debugging
**Federated Support:**
- Uses community's PDS URL when available
- Fallback to service default PDS
- Community authentication via PDSAccessToken
**Flow:**
1. Client posts external embed with URI (no thumb)
2. Unfurl service fetches metadata from oEmbed/OpenGraph
3. Blob service downloads thumbnail from metadata.thumbnailURL
4. Upload to community's PDS via com.atproto.repo.uploadBlob
5. Return BlobRef with CID for inclusion in post record
**BlobRef Type:**
```go
type BlobRef struct {
Type string `json:"$type"` // "blob"
Ref map[string]string `json:"ref"` // {"$link": "bafyrei..."}
MimeType string `json:"mimeType"` // "image/jpeg"
Size int `json:"size"` // bytes
}
```
This enables automatic thumbnail upload when users post links to
Streamable, YouTube, Reddit, Kagi Kite, or any URL with OpenGraph metadata.
Add PostgreSQL-backed cache for oEmbed and OpenGraph unfurl results to reduce
external API calls and improve performance.
**Database Layer:**
- Migration 017: Create unfurl_cache table with JSONB metadata storage
- Index on expires_at for efficient TTL-based cleanup
- Store provider, metadata, and thumbnail_url with expiration
**Repository Layer:**
- Repository interface with Get/Set operations
- PostgreSQL implementation with JSON marshaling
- Automatic TTL handling via PostgreSQL intervals
- Returns nil on cache miss (not an error)
**Error Types:**
- ErrNotFound: Cache miss or expired entry
- ErrInvalidURL: Invalid URL format
- ErrInvalidTTL: Non-positive TTL value
Design decisions:
- JSONB metadata column for flexible schema evolution
- Separate thumbnail_url for potential query optimization
- ON CONFLICT for upsert behavior (update on re-fetch)
- TTL-based expiration (default: 24 hours)
Part of URL unfurling feature to auto-populate external embeds with rich
metadata from supported providers (Streamable, YouTube, Reddit, Kagi, etc.).
Related: Circuit breaker pattern prevents cascading failures when providers
go down (already implemented in previous commits).
Update integration tests to reflect new validation order and circuit
breaker integration in unfurl service.
Changes in post_creation_test.go:
- Fix content length validation test expectations
- Update validation order: basic input before DID authentication
- Adjust test assertions to match new error flow
Changes in post_unfurl_test.go:
- Update Kagi provider test to expect circuit breaker wrapper
- Fix provider name expectations in unfurl tests
- Ensure tests align with circuit breaker integration
These changes ensure all integration tests pass with the new validation
flow and circuit breaker implementation.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>
Restore full aggregator authorization checks while maintaining the
special case for Kagi aggregator's thumbnail URL handling.
Changes:
- Restore aggregator DID validation in post creation flow
- Add distinction between Kagi (trusted) and other aggregators
- Map aggregator authorization errors to 403 Forbidden
- Maintain validation order: basic input -> DID auth -> aggregator check
- Keep Kagi special case for thumbnail URL transformation
Security improvements:
- All aggregator posts now require valid aggregator DID registration
- Kagi aggregator identified via KAGI_AGGREGATOR_DID environment variable
- Non-Kagi aggregators must follow standard thumbnail validation rules
- Unauthorized aggregator attempts return 403 with clear error message
This ensures only authorized aggregators can create posts while allowing
Kagi's existing thumbnail URL workflow to continue working.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <noreply@anthropic.com>