Tiered Storage#

A lightweight, pluggable tiered storage library that orchestrates caching across hot (memory), warm (disk/database), and cold (S3/object storage) tiers.

Features#

• Cascading Containment Model: Hot ⊆ Warm ⊆ Cold (lower tiers contain all data from upper tiers)
• Pluggable Backends: Bring your own Redis, Postgres, SQLite, or use built-in implementations
• Automatic Promotion: Configurable eager/lazy promotion strategies for cache warming
• TTL Management: Per-key TTL with automatic expiration and renewal
• Prefix Invalidation: Efficiently delete groups of keys by prefix
• Bootstrap Support: Warm up caches from lower tiers on startup
• Compression: Optional transparent gzip compression
• TypeScript First: Full type safety with comprehensive TSDoc comments
• Zero Forced Dependencies: Only require what you use

Installation#

npm install tiered-storage
# or
bun add tiered-storage

Quick Start#

import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage';

const storage = new TieredStorage({
  tiers: {
    hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }), // 100MB
    warm: new DiskStorageTier({ directory: './cache' }),
    cold: new S3StorageTier({
      bucket: 'my-bucket',
      region: 'us-east-1',
      credentials: {
        accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
        secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
      },
    }),
  },
  compression: true,
  defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
  promotionStrategy: 'lazy',
});

// Store data (cascades to all tiers)
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });

// Retrieve data (bubbles up from cold → warm → hot)
const user = await storage.get('user:123');

// Get data with metadata and source tier
const result = await storage.getWithMetadata('user:123');
console.log(`Served from ${result.source}`); // 'hot', 'warm', or 'cold'

// Invalidate all keys with prefix
await storage.invalidate('user:');

// Renew TTL
await storage.touch('user:123');

Core Concepts#

Cascading Containment Model#

┌──────────────────────────────────────────────────────┐
│ Cold Storage (S3/Object Storage)                     │
│ • Contains ALL objects (source of truth)             │
│ • Slowest access, unlimited capacity                 │
├──────────────────────────────────────────────────────┤
│ Warm Storage (Disk/Database)                         │
│ • Contains ALL hot objects + additional warm objects │
│ • Medium access speed, large capacity               │
├──────────────────────────────────────────────────────┤
│ Hot Storage (Memory)                                 │
│ • Contains only the hottest objects                  │
│ • Fastest access, limited capacity                   │
└──────────────────────────────────────────────────────┘

Write Strategy (Cascading Down):

• Write to hot → also writes to warm and cold
• Write to warm → also writes to cold
• Write to cold → only writes to cold

Read Strategy (Bubbling Up):

• Check hot first → if miss, check warm → if miss, check cold
• On cache miss, optionally promote data up through tiers

Selective Tier Placement#

For use cases like static site hosting, you can control which files go into which tiers:

// Small, critical file (index.html) - store in all tiers for instant serving
await storage.set('site:abc/index.html', htmlContent);

// Large file (video) - skip hot tier to avoid memory bloat
await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] });

// Medium files (images, CSS) - skip hot, use warm + cold
await storage.set('site:abc/style.css', cssData, { skipTiers: ['hot'] });

This pattern ensures:

• Hot tier stays small and fast (only critical files)
• Warm tier caches everything (all site files on disk)
• Cold tier is source of truth (all data)

API Reference#

TieredStorage#

Main orchestrator class for tiered storage.

Constructor#

new TieredStorage<T>(config: TieredStorageConfig)

Config Options:

interface TieredStorageConfig {
  tiers: {
    hot?: StorageTier;    // Optional: fastest tier (memory/Redis)
    warm?: StorageTier;   // Optional: medium tier (disk/SQLite/Postgres)
    cold: StorageTier;    // Required: slowest tier (S3/object storage)
  };
  compression?: boolean;              // Auto-compress before storing (default: false)
  defaultTTL?: number;                // Default TTL in milliseconds
  promotionStrategy?: 'eager' | 'lazy'; // When to promote to upper tiers (default: 'lazy')
  serialization?: {                   // Custom serialization (default: JSON)
    serialize: (data: unknown) => Promise<Uint8Array>;
    deserialize: (data: Uint8Array) => Promise<unknown>;
  };
}

Methods#

get(key: string): Promise<T | null>

Retrieve data for a key. Returns null if not found or expired.

getWithMetadata(key: string): Promise<StorageResult<T> | null>

Retrieve data with metadata and source tier information.

const result = await storage.getWithMetadata('user:123');
console.log(result.data);        // The actual data
console.log(result.source);      // 'hot' | 'warm' | 'cold'
console.log(result.metadata);    // Metadata (size, timestamps, TTL, etc.)

set(key: string, data: T, options?: SetOptions): Promise<SetResult>

Store data with optional configuration.

await storage.set('key', data, {
  ttl: 24 * 60 * 60 * 1000,       // Custom TTL (24 hours)
  metadata: { contentType: 'application/json' },  // Custom metadata
  skipTiers: ['hot'],             // Skip specific tiers
});

delete(key: string): Promise<void>

Delete data from all tiers.

exists(key: string): Promise<boolean>

Check if a key exists (and hasn't expired).

touch(key: string, ttlMs?: number): Promise<void>

Renew TTL for a key. Useful for "keep alive" behavior.

invalidate(prefix: string): Promise<number>

Delete all keys matching a prefix. Returns number of keys deleted.

await storage.invalidate('user:');      // Delete all user keys
await storage.invalidate('site:abc/');  // Delete all files for site 'abc'
await storage.invalidate('');           // Delete everything

listKeys(prefix?: string): AsyncIterableIterator<string>

List all keys, optionally filtered by prefix.

for await (const key of storage.listKeys('user:')) {
  console.log(key); // 'user:123', 'user:456', etc.
}

getStats(): Promise<AllTierStats>

Get aggregated statistics across all tiers.

const stats = await storage.getStats();
console.log(stats.hot);           // Hot tier stats (size, items, hits, misses)
console.log(stats.hitRate);       // Overall hit rate (0-1)

bootstrapHot(limit?: number): Promise<number>

Load most frequently accessed items from warm into hot. Returns number of items loaded.

// On server startup: warm up hot tier
const loaded = await storage.bootstrapHot(1000); // Load top 1000 items
console.log(`Loaded ${loaded} items into hot tier`);

bootstrapWarm(options?: { limit?: number; sinceDate?: Date }): Promise<number>

Load recent items from cold into warm. Returns number of items loaded.

// Load items accessed in last 7 days
const loaded = await storage.bootstrapWarm({
  sinceDate: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000),
  limit: 10000,
});

export(): Promise<StorageSnapshot>

Export metadata snapshot for backup or migration.

import(snapshot: StorageSnapshot): Promise<void>

Import metadata snapshot.

clear(): Promise<void>

Clear all data from all tiers. ⚠️ Use with extreme caution!

clearTier(tier: 'hot' | 'warm' | 'cold'): Promise<void>

Clear a specific tier.

Built-in Storage Tiers#

MemoryStorageTier#

In-memory storage using TinyLRU for efficient LRU eviction.

import { MemoryStorageTier } from 'tiered-storage';

const tier = new MemoryStorageTier({
  maxSizeBytes: 100 * 1024 * 1024, // 100MB
  maxItems: 1000,                   // Optional: max number of items
});

Features:

• Battle-tested TinyLRU library
• Automatic LRU eviction
• Size-based and count-based limits
• Single process only (not distributed)

DiskStorageTier#

Filesystem-based storage with .meta files.

import { DiskStorageTier } from 'tiered-storage';

const tier = new DiskStorageTier({
  directory: './cache',
  maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB (optional)
  evictionPolicy: 'lru',                   // 'lru' | 'fifo' | 'size'
});

Features:

• Human-readable file structure
• Optional size-based eviction
• Three eviction policies: LRU, FIFO, size-based
• Atomic writes with .meta files
• Zero external dependencies

File structure:

cache/
├── user%3A123          # Data file (encoded key)
├── user%3A123.meta     # Metadata JSON
├── site%3Aabc%2Findex.html
└── site%3Aabc%2Findex.html.meta

S3StorageTier#

AWS S3 or S3-compatible object storage.

import { S3StorageTier } from 'tiered-storage';

// AWS S3 with separate metadata bucket (RECOMMENDED!)
const tier = new S3StorageTier({
  bucket: 'my-data-bucket',
  metadataBucket: 'my-metadata-bucket', // Stores metadata separately for fast updates
  region: 'us-east-1',
  credentials: {
    accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
    secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
  },
  prefix: 'cache/', // Optional key prefix
});

// Cloudflare R2 with metadata bucket
const r2Tier = new S3StorageTier({
  bucket: 'my-r2-data-bucket',
  metadataBucket: 'my-r2-metadata-bucket',
  region: 'auto',
  endpoint: 'https://account-id.r2.cloudflarestorage.com',
  credentials: {
    accessKeyId: process.env.R2_ACCESS_KEY_ID!,
    secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
  },
});

// Without metadata bucket (legacy mode - slower, more expensive)
const legacyTier = new S3StorageTier({
  bucket: 'my-bucket',
  region: 'us-east-1',
  // No metadataBucket - metadata stored in S3 object metadata fields
});

Features:

• Compatible with AWS S3, Cloudflare R2, MinIO, and other S3-compatible services
• Separate metadata bucket support (RECOMMENDED) - stores metadata as JSON objects for fast, cheap updates
• Legacy mode: metadata in S3 object metadata fields (requires object copying for updates)
• Efficient batch deletions (up to 1000 keys per request)
• Optional key prefixing for multi-tenant scenarios
• Typically used as cold tier (source of truth)

⚠️ Important: Without metadataBucket, updating metadata (e.g., access counts) requires copying the entire object, which is slow and expensive for large files. Use a separate metadata bucket in production!

Usage Patterns#

Pattern 1: Simple Single-Server Setup#

import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';

const storage = new TieredStorage({
  tiers: {
    hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
    warm: new DiskStorageTier({ directory: './cache' }),
    cold: new DiskStorageTier({ directory: './storage' }),
  },
  compression: true,
  defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
});

await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
const user = await storage.get('user:123');

Pattern 2: Static Site Hosting (wisp.place-style)#

import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';

const storage = new TieredStorage({
  tiers: {
    hot: new MemoryStorageTier({
      maxSizeBytes: 100 * 1024 * 1024, // 100MB
      maxItems: 500,
    }),
    warm: new DiskStorageTier({
      directory: './cache/sites',
      maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB
    }),
    // Cold tier is PDS (fetched on demand via custom tier implementation)
  },
  compression: true,
  defaultTTL: 14 * 24 * 60 * 60 * 1000,
  promotionStrategy: 'lazy', // Don't auto-promote large files to hot
});

// Store index.html in all tiers (fast access)
await storage.set(`${did}/${rkey}/index.html`, htmlBuffer, {
  metadata: { mimeType: 'text/html', encoding: 'gzip' },
});

// Store large files only in warm + cold (skip hot)
await storage.set(`${did}/${rkey}/video.mp4`, videoBuffer, {
  skipTiers: ['hot'],
  metadata: { mimeType: 'video/mp4' },
});

// Get file with source tracking
const result = await storage.getWithMetadata(`${did}/${rkey}/index.html`);
console.log(`Served from ${result.source}`); // Likely 'hot' for index.html

// Invalidate entire site
await storage.invalidate(`${did}/${rkey}/`);

// Renew TTL when site is accessed
await storage.touch(`${did}/${rkey}/index.html`);

Pattern 3: Custom Backend (SQLite)#

Implement the StorageTier interface to use any backend:

import { StorageTier, StorageMetadata, TierStats } from 'tiered-storage';
import Database from 'better-sqlite3';

class SQLiteStorageTier implements StorageTier {
  private db: Database.Database;

  constructor(dbPath: string) {
    this.db = new Database(dbPath);
    this.db.exec(`
      CREATE TABLE IF NOT EXISTS cache (
        key TEXT PRIMARY KEY,
        data BLOB NOT NULL,
        metadata TEXT NOT NULL
      )
    `);
  }

  async get(key: string): Promise<Uint8Array | null> {
    const row = this.db.prepare('SELECT data FROM cache WHERE key = ?').get(key);
    return row ? new Uint8Array(row.data) : null;
  }

  async set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void> {
    this.db.prepare('INSERT OR REPLACE INTO cache (key, data, metadata) VALUES (?, ?, ?)')
      .run(key, Buffer.from(data), JSON.stringify(metadata));
  }

  async delete(key: string): Promise<void> {
    this.db.prepare('DELETE FROM cache WHERE key = ?').run(key);
  }

  async exists(key: string): Promise<boolean> {
    const row = this.db.prepare('SELECT 1 FROM cache WHERE key = ?').get(key);
    return !!row;
  }

  async *listKeys(prefix?: string): AsyncIterableIterator<string> {
    const query = prefix
      ? this.db.prepare('SELECT key FROM cache WHERE key LIKE ?')
      : this.db.prepare('SELECT key FROM cache');

    const rows = prefix ? query.all(`${prefix}%`) : query.all();

    for (const row of rows) {
      yield row.key;
    }
  }

  async deleteMany(keys: string[]): Promise<void> {
    const placeholders = keys.map(() => '?').join(',');
    this.db.prepare(`DELETE FROM cache WHERE key IN (${placeholders})`).run(...keys);
  }

  async getMetadata(key: string): Promise<StorageMetadata | null> {
    const row = this.db.prepare('SELECT metadata FROM cache WHERE key = ?').get(key);
    return row ? JSON.parse(row.metadata) : null;
  }

  async setMetadata(key: string, metadata: StorageMetadata): Promise<void> {
    this.db.prepare('UPDATE cache SET metadata = ? WHERE key = ?')
      .run(JSON.stringify(metadata), key);
  }

  async getStats(): Promise<TierStats> {
    const row = this.db.prepare('SELECT COUNT(*) as count, SUM(LENGTH(data)) as bytes FROM cache').get();
    return { items: row.count, bytes: row.bytes || 0 };
  }

  async clear(): Promise<void> {
    this.db.prepare('DELETE FROM cache').run();
  }
}

// Use it
const storage = new TieredStorage({
  tiers: {
    warm: new SQLiteStorageTier('./cache.db'),
    cold: new DiskStorageTier({ directory: './storage' }),
  },
});

Running Examples#

Interactive Demo Server#

Run a real HTTP server that serves the example site using tiered storage:

# Configure S3 credentials first (copy .env.example to .env and fill in)
cp .env.example .env

# Start the demo server
bun run serve

Then visit:

• http://localhost:3000/ - The demo site served from tiered storage
• http://localhost:3000/admin/stats - Live cache statistics dashboard

Watch the console to see which tier serves each request:

• 🔥 Hot tier (memory) - index.html served instantly
• 💾 Warm tier (disk) - Other pages served from disk cache
• ☁️ Cold tier (S3) - First access fetches from S3, then cached

Command-Line Examples#

Or run the non-interactive examples:

bun run example

The examples include:

• Basic CRUD operations with statistics tracking
• Static site hosting using the real site in example-site/ directory
• Bootstrap demonstrations (warming caches from lower tiers)
• Promotion strategy comparisons (eager vs lazy)

The example-site/ directory contains a complete static website with:

• index.html - Stored in hot + warm + cold (instant serving)
• about.html, docs.html - Stored in warm + cold (skips hot)
• style.css, script.js - Stored in warm + cold (skips hot)

This demonstrates the exact pattern you'd use for wisp.place: critical files in memory, everything else on disk/S3.

Testing#

bun test

Development#

# Install dependencies
bun install

# Type check
bun run check

# Build
bun run build

# Run tests
bun test

License#

MIT