commits
- Created 7 shared packages (@wisp/*)
- lexicons: 15k+ lines deduplicated
- observability: 670 lines deduplicated
- atproto-utils, fs-utils, database, constants, safe-fetch
Prevent accidentally uploading unwanted files during site deployment:
- OS metadata (.DS_Store, Thumbs.db, etc.)
- Environment files (.env*)
- Dependencies (node_modules)
- Python artifacts (__pycache__, venvs)
- Cache and temp directories
- Editor swap files
Also updates Jacquard dependency source references in Cargo.lock.
- CLI: only skip .git directory instead of all dotfiles
- Hosting service: add logging for PDS record verification
- Hosting service: remove watch flag from dev script
Implements transparent subfs record fetching and expansion:
- extractSubfsUris() finds all subfs references in directory tree
- fetchSubfsRecord() retrieves subfs records from PDS
- expandSubfsNodes() replaces subfs nodes with actual content
When caching sites, the hosting service now:
1. Detects subfs nodes in the manifest
2. Fetches all referenced subfs records in parallel
3. Expands the tree by replacing subfs nodes with their content
4. Caches the fully expanded site normally
This makes subfs completely transparent to end users.
When a user deletes a site, now fetches the main record first to
extract any subfs references, then deletes all associated subfs
records along with the main record.
Prevents orphaned subfs records from accumulating in the PDS.
- Splits manifests >140KB or sites with 250+ files into subfs records
- Creates subfs records with TID-based rkeys for uniqueness
- Splits largest directories first until manifest fits under limit
- Fetches and merges blob maps from all subfs records for full reuse
- Deletes old subfs records after successful upload
Upload flow:
1. Fetch existing main + all subfs records
2. Build combined blob map for incremental updates
3. Upload blobs (reuses across all records)
4. Split directories into subfs if needed (auto-detect)
5. Upload main manifest with subfs references
6. Clean up orphaned subfs records from previous version
Adds extractSubfsUris() to find all subfs references in a directory tree,
tracking their mount paths for proper blob path resolution.
Also adds replaceDirectoryWithSubfs() and findLargeDirectories() to
support automatic manifest splitting when sites exceed size limits.
Adds 'subfs' as a new node type in directory entries. A subfs node
contains a subject URI pointing to a place.wisp.subfs record that
holds the actual directory content.
This allows the main manifest to reference external records instead
of embedding large directory trees directly.
Introduces a new lexicon type to split large sites into multiple records.
The subfs (sub-filesystem) record contains a directory tree that can be
referenced from the main place.wisp.fs manifest via a subfs node.
This enables sites with 400+ files or manifests >140KB to bypass PDS
record size limits by splitting directories into separate records.
- Created 7 shared packages (@wisp/*)
- lexicons: 15k+ lines deduplicated
- observability: 670 lines deduplicated
- atproto-utils, fs-utils, database, constants, safe-fetch
Prevent accidentally uploading unwanted files during site deployment:
- OS metadata (.DS_Store, Thumbs.db, etc.)
- Environment files (.env*)
- Dependencies (node_modules)
- Python artifacts (__pycache__, venvs)
- Cache and temp directories
- Editor swap files
Also updates Jacquard dependency source references in Cargo.lock.
Implements transparent subfs record fetching and expansion:
- extractSubfsUris() finds all subfs references in directory tree
- fetchSubfsRecord() retrieves subfs records from PDS
- expandSubfsNodes() replaces subfs nodes with actual content
When caching sites, the hosting service now:
1. Detects subfs nodes in the manifest
2. Fetches all referenced subfs records in parallel
3. Expands the tree by replacing subfs nodes with their content
4. Caches the fully expanded site normally
This makes subfs completely transparent to end users.
- Splits manifests >140KB or sites with 250+ files into subfs records
- Creates subfs records with TID-based rkeys for uniqueness
- Splits largest directories first until manifest fits under limit
- Fetches and merges blob maps from all subfs records for full reuse
- Deletes old subfs records after successful upload
Upload flow:
1. Fetch existing main + all subfs records
2. Build combined blob map for incremental updates
3. Upload blobs (reuses across all records)
4. Split directories into subfs if needed (auto-detect)
5. Upload main manifest with subfs references
6. Clean up orphaned subfs records from previous version
Introduces a new lexicon type to split large sites into multiple records.
The subfs (sub-filesystem) record contains a directory tree that can be
referenced from the main place.wisp.fs manifest via a subfs node.
This enables sites with 400+ files or manifests >140KB to bypass PDS
record size limits by splitting directories into separate records.