···
-
A lightweight, pluggable tiered storage library that orchestrates caching across hot (memory), warm (disk/database), and cold (S3/object storage) tiers.
-
- **Cascading Containment Model**: Hot ⊆ Warm ⊆ Cold (lower tiers contain all data from upper tiers)
-
- **Pluggable Backends**: Bring your own Redis, Postgres, SQLite, or use built-in implementations
-
- **Automatic Promotion**: Configurable eager/lazy promotion strategies for cache warming
-
- **TTL Management**: Per-key TTL with automatic expiration and renewal
-
- **Prefix Invalidation**: Efficiently delete groups of keys by prefix
-
- **Bootstrap Support**: Warm up caches from lower tiers on startup
-
- **Compression**: Optional transparent gzip compression
-
- **TypeScript First**: Full type safety with comprehensive TSDoc comments
-
- **Zero Forced Dependencies**: Only require what you use
npm install tiered-storage
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage';
const storage = new TieredStorage({
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }), // 100MB
warm: new DiskStorageTier({ directory: './cache' }),
-
cold: new S3StorageTier({
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
-
promotionStrategy: 'lazy',
-
// Store data (cascades to all tiers)
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
-
// Retrieve data (bubbles up from cold → warm → hot)
-
const user = await storage.get('user:123');
-
// Get data with metadata and source tier
-
const result = await storage.getWithMetadata('user:123');
-
console.log(`Served from ${result.source}`); // 'hot', 'warm', or 'cold'
-
// Invalidate all keys with prefix
-
await storage.invalidate('user:');
-
await storage.touch('user:123');
-
### Cascading Containment Model
-
┌──────────────────────────────────────────────────────┐
-
│ Cold Storage (S3/Object Storage) │
-
│ • Contains ALL objects (source of truth) │
-
│ • Slowest access, unlimited capacity │
-
├──────────────────────────────────────────────────────┤
-
│ Warm Storage (Disk/Database) │
-
│ • Contains ALL hot objects + additional warm objects │
-
│ • Medium access speed, large capacity │
-
├──────────────────────────────────────────────────────┤
-
│ Hot Storage (Memory) │
-
│ • Contains only the hottest objects │
-
│ • Fastest access, limited capacity │
-
└──────────────────────────────────────────────────────┘
-
**Write Strategy (Cascading Down):**
-
- Write to **hot** → also writes to **warm** and **cold**
-
- Write to **warm** → also writes to **cold**
-
- Write to **cold** → only writes to **cold**
-
**Read Strategy (Bubbling Up):**
-
- Check **hot** first → if miss, check **warm** → if miss, check **cold**
-
- On cache miss, optionally promote data up through tiers
-
### Selective Tier Placement
-
For use cases like static site hosting, you can control which files go into which tiers:
-
// Small, critical file (index.html) - store in all tiers for instant serving
-
await storage.set('site:abc/index.html', htmlContent);
-
// Large file (video) - skip hot tier to avoid memory bloat
-
await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] });
-
// Medium files (images, CSS) - skip hot, use warm + cold
-
await storage.set('site:abc/style.css', cssData, { skipTiers: ['hot'] });
-
- Hot tier stays small and fast (only critical files)
-
- Warm tier caches everything (all site files on disk)
-
- Cold tier is source of truth (all data)
-
Main orchestrator class for tiered storage.
-
new TieredStorage<T>(config: TieredStorageConfig)
-
interface TieredStorageConfig {
-
hot?: StorageTier; // Optional: fastest tier (memory/Redis)
-
warm?: StorageTier; // Optional: medium tier (disk/SQLite/Postgres)
-
cold: StorageTier; // Required: slowest tier (S3/object storage)
-
compression?: boolean; // Auto-compress before storing (default: false)
-
defaultTTL?: number; // Default TTL in milliseconds
-
promotionStrategy?: 'eager' | 'lazy'; // When to promote to upper tiers (default: 'lazy')
-
serialization?: { // Custom serialization (default: JSON)
-
serialize: (data: unknown) => Promise<Uint8Array>;
-
deserialize: (data: Uint8Array) => Promise<unknown>;
-
**`get(key: string): Promise<T | null>`**
-
Retrieve data for a key. Returns null if not found or expired.
-
**`getWithMetadata(key: string): Promise<StorageResult<T> | null>`**
-
Retrieve data with metadata and source tier information.
-
const result = await storage.getWithMetadata('user:123');
-
console.log(result.data); // The actual data
-
console.log(result.source); // 'hot' | 'warm' | 'cold'
-
console.log(result.metadata); // Metadata (size, timestamps, TTL, etc.)
-
**`set(key: string, data: T, options?: SetOptions): Promise<SetResult>`**
-
Store data with optional configuration.
-
await storage.set('key', data, {
-
ttl: 24 * 60 * 60 * 1000, // Custom TTL (24 hours)
-
metadata: { contentType: 'application/json' }, // Custom metadata
-
skipTiers: ['hot'], // Skip specific tiers
-
**`delete(key: string): Promise<void>`**
-
Delete data from all tiers.
-
**`exists(key: string): Promise<boolean>`**
-
Check if a key exists (and hasn't expired).
-
**`touch(key: string, ttlMs?: number): Promise<void>`**
-
Renew TTL for a key. Useful for "keep alive" behavior.
-
**`invalidate(prefix: string): Promise<number>`**
-
Delete all keys matching a prefix. Returns number of keys deleted.
-
await storage.invalidate('user:'); // Delete all user keys
-
await storage.invalidate('site:abc/'); // Delete all files for site 'abc'
-
await storage.invalidate(''); // Delete everything
-
**`listKeys(prefix?: string): AsyncIterableIterator<string>`**
-
List all keys, optionally filtered by prefix.
-
for await (const key of storage.listKeys('user:')) {
-
console.log(key); // 'user:123', 'user:456', etc.
-
**`getStats(): Promise<AllTierStats>`**
-
Get aggregated statistics across all tiers.
-
const stats = await storage.getStats();
-
console.log(stats.hot); // Hot tier stats (size, items, hits, misses)
-
console.log(stats.hitRate); // Overall hit rate (0-1)
-
**`bootstrapHot(limit?: number): Promise<number>`**
-
Load most frequently accessed items from warm into hot. Returns number of items loaded.
-
// On server startup: warm up hot tier
-
const loaded = await storage.bootstrapHot(1000); // Load top 1000 items
-
console.log(`Loaded ${loaded} items into hot tier`);
-
**`bootstrapWarm(options?: { limit?: number; sinceDate?: Date }): Promise<number>`**
-
Load recent items from cold into warm. Returns number of items loaded.
-
// Load items accessed in last 7 days
-
const loaded = await storage.bootstrapWarm({
-
sinceDate: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000),
-
**`export(): Promise<StorageSnapshot>`**
-
Export metadata snapshot for backup or migration.
-
**`import(snapshot: StorageSnapshot): Promise<void>`**
-
Import metadata snapshot.
-
**`clear(): Promise<void>`**
-
Clear all data from all tiers. ⚠️ Use with extreme caution!
-
**`clearTier(tier: 'hot' | 'warm' | 'cold'): Promise<void>`**
-
### Built-in Storage Tiers
-
#### `MemoryStorageTier`
-
In-memory storage using TinyLRU for efficient LRU eviction.
-
import { MemoryStorageTier } from 'tiered-storage';
-
const tier = new MemoryStorageTier({
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
-
maxItems: 1000, // Optional: max number of items
-
- Battle-tested TinyLRU library
-
- Automatic LRU eviction
-
- Size-based and count-based limits
-
- Single process only (not distributed)
-
Filesystem-based storage with `.meta` files.
-
import { DiskStorageTier } from 'tiered-storage';
-
const tier = new DiskStorageTier({
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB (optional)
-
evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
-
- Human-readable file structure
-
- Optional size-based eviction
-
- Three eviction policies: LRU, FIFO, size-based
-
- Atomic writes with `.meta` files
-
- Zero external dependencies
-
├── user%3A123 # Data file (encoded key)
-
├── user%3A123.meta # Metadata JSON
-
├── site%3Aabc%2Findex.html
-
└── site%3Aabc%2Findex.html.meta
-
AWS S3 or S3-compatible object storage.
-
import { S3StorageTier } from 'tiered-storage';
-
// AWS S3 with separate metadata bucket (RECOMMENDED!)
-
const tier = new S3StorageTier({
-
bucket: 'my-data-bucket',
-
metadataBucket: 'my-metadata-bucket', // Stores metadata separately for fast updates
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
prefix: 'cache/', // Optional key prefix
-
// Cloudflare R2 with metadata bucket
-
const r2Tier = new S3StorageTier({
-
bucket: 'my-r2-data-bucket',
-
metadataBucket: 'my-r2-metadata-bucket',
-
endpoint: 'https://account-id.r2.cloudflarestorage.com',
-
accessKeyId: process.env.R2_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
-
// Without metadata bucket (legacy mode - slower, more expensive)
-
const legacyTier = new S3StorageTier({
-
// No metadataBucket - metadata stored in S3 object metadata fields
-
- Compatible with AWS S3, Cloudflare R2, MinIO, and other S3-compatible services
-
- **Separate metadata bucket support (RECOMMENDED)** - stores metadata as JSON objects for fast, cheap updates
-
- Legacy mode: metadata in S3 object metadata fields (requires object copying for updates)
-
- Efficient batch deletions (up to 1000 keys per request)
-
- Optional key prefixing for multi-tenant scenarios
-
- Typically used as cold tier (source of truth)
-
**⚠️ Important:** Without `metadataBucket`, updating metadata (e.g., access counts) requires copying the entire object, which is slow and expensive for large files. Use a separate metadata bucket in production!
-
### Pattern 1: Simple Single-Server Setup
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
-
const storage = new TieredStorage({
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
-
warm: new DiskStorageTier({ directory: './cache' }),
-
cold: new DiskStorageTier({ directory: './storage' }),
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
-
const user = await storage.get('user:123');
-
### Pattern 2: Static Site Hosting (wisp.place-style)
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
-
const storage = new TieredStorage({
-
hot: new MemoryStorageTier({
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
-
warm: new DiskStorageTier({
-
directory: './cache/sites',
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB
-
// Cold tier is PDS (fetched on demand via custom tier implementation)
-
defaultTTL: 14 * 24 * 60 * 60 * 1000,
-
promotionStrategy: 'lazy', // Don't auto-promote large files to hot
-
// Store index.html in all tiers (fast access)
-
await storage.set(`${did}/${rkey}/index.html`, htmlBuffer, {
-
metadata: { mimeType: 'text/html', encoding: 'gzip' },
-
// Store large files only in warm + cold (skip hot)
-
await storage.set(`${did}/${rkey}/video.mp4`, videoBuffer, {
-
metadata: { mimeType: 'video/mp4' },
-
// Get file with source tracking
-
const result = await storage.getWithMetadata(`${did}/${rkey}/index.html`);
-
console.log(`Served from ${result.source}`); // Likely 'hot' for index.html
-
// Invalidate entire site
-
await storage.invalidate(`${did}/${rkey}/`);
-
// Renew TTL when site is accessed
-
await storage.touch(`${did}/${rkey}/index.html`);
-
### Pattern 3: Custom Backend (SQLite)
-
Implement the `StorageTier` interface to use any backend:
-
import { StorageTier, StorageMetadata, TierStats } from 'tiered-storage';
-
import Database from 'better-sqlite3';
-
class SQLiteStorageTier implements StorageTier {
-
private db: Database.Database;
-
constructor(dbPath: string) {
-
this.db = new Database(dbPath);
-
CREATE TABLE IF NOT EXISTS cache (
-
async get(key: string): Promise<Uint8Array | null> {
-
const row = this.db.prepare('SELECT data FROM cache WHERE key = ?').get(key);
-
return row ? new Uint8Array(row.data) : null;
-
async set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void> {
-
this.db.prepare('INSERT OR REPLACE INTO cache (key, data, metadata) VALUES (?, ?, ?)')
-
.run(key, Buffer.from(data), JSON.stringify(metadata));
-
async delete(key: string): Promise<void> {
-
this.db.prepare('DELETE FROM cache WHERE key = ?').run(key);
-
async exists(key: string): Promise<boolean> {
-
const row = this.db.prepare('SELECT 1 FROM cache WHERE key = ?').get(key);
-
async *listKeys(prefix?: string): AsyncIterableIterator<string> {
-
? this.db.prepare('SELECT key FROM cache WHERE key LIKE ?')
-
: this.db.prepare('SELECT key FROM cache');
-
const rows = prefix ? query.all(`${prefix}%`) : query.all();
-
for (const row of rows) {
-
async deleteMany(keys: string[]): Promise<void> {
-
const placeholders = keys.map(() => '?').join(',');
-
this.db.prepare(`DELETE FROM cache WHERE key IN (${placeholders})`).run(...keys);
-
async getMetadata(key: string): Promise<StorageMetadata | null> {
-
const row = this.db.prepare('SELECT metadata FROM cache WHERE key = ?').get(key);
-
return row ? JSON.parse(row.metadata) : null;
-
async setMetadata(key: string, metadata: StorageMetadata): Promise<void> {
-
this.db.prepare('UPDATE cache SET metadata = ? WHERE key = ?')
-
.run(JSON.stringify(metadata), key);
-
async getStats(): Promise<TierStats> {
-
const row = this.db.prepare('SELECT COUNT(*) as count, SUM(LENGTH(data)) as bytes FROM cache').get();
-
return { items: row.count, bytes: row.bytes || 0 };
-
async clear(): Promise<void> {
-
this.db.prepare('DELETE FROM cache').run();
-
const storage = new TieredStorage({
-
warm: new SQLiteStorageTier('./cache.db'),
-
cold: new DiskStorageTier({ directory: './storage' }),
-
### Interactive Demo Server
-
Run a **real HTTP server** that serves the example site using tiered storage:
-
# Configure S3 credentials first (copy .env.example to .env and fill in)
-
# Start the demo server
-
- **http://localhost:3000/** - The demo site served from tiered storage
-
- **http://localhost:3000/admin/stats** - Live cache statistics dashboard
-
Watch the console to see which tier serves each request:
-
- 🔥 **Hot tier (memory)** - index.html served instantly
-
- 💾 **Warm tier (disk)** - Other pages served from disk cache
-
- ☁️ **Cold tier (S3)** - First access fetches from S3, then cached
-
### Command-Line Examples
-
Or run the non-interactive examples:
-
- **Basic CRUD operations** with statistics tracking
-
- **Static site hosting** using the real site in `example-site/` directory
-
- **Bootstrap demonstrations** (warming caches from lower tiers)
-
- **Promotion strategy comparisons** (eager vs lazy)
-
The `example-site/` directory contains a complete static website with:
-
- `index.html` - Stored in hot + warm + cold (instant serving)
-
- `about.html`, `docs.html` - Stored in warm + cold (skips hot)
-
- `style.css`, `script.js` - Stored in warm + cold (skips hot)
-
This demonstrates the exact pattern you'd use for wisp.place: critical files in memory, everything else on disk/S3.
nekomimi.pet