wip library to store cold objects in s3, warm objects on disk, and hot objects in memory
nodejs typescript

Compare changes

Choose any two refs to compare.

+148 -466
README.md
···
-
# Tiered Storage
+
# tiered-storage
-
A lightweight, pluggable tiered storage library that orchestrates caching across hot (memory), warm (disk/database), and cold (S3/object storage) tiers.
+
Cascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.
## Features
-
- **Cascading Containment Model**: Hot ⊆ Warm ⊆ Cold (lower tiers contain all data from upper tiers)
-
- **Pluggable Backends**: Bring your own Redis, Postgres, SQLite, or use built-in implementations
-
- **Automatic Promotion**: Configurable eager/lazy promotion strategies for cache warming
-
- **TTL Management**: Per-key TTL with automatic expiration and renewal
-
- **Prefix Invalidation**: Efficiently delete groups of keys by prefix
-
- **Bootstrap Support**: Warm up caches from lower tiers on startup
-
- **Compression**: Optional transparent gzip compression
-
- **TypeScript First**: Full type safety with comprehensive TSDoc comments
-
- **Zero Forced Dependencies**: Only require what you use
+
- **Cascading writes** - data flows down through all tiers
+
- **Bubbling reads** - check hot first, fall back to warm, then cold
+
- **Pluggable backends** - memory, disk, S3, or implement your own
+
- **Selective placement** - skip tiers for big files that don't need memory caching
+
- **Prefix invalidation** - `invalidate('user:')` nukes all user keys
+
- **Optional compression** - transparent gzip
-
## Installation
+
## Install
```bash
npm install tiered-storage
-
# or
-
bun add tiered-storage
```
-
## Quick Start
+
## Example
```typescript
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage';
+
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'
const storage = new TieredStorage({
tiers: {
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }), // 100MB
+
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
warm: new DiskStorageTier({ directory: './cache' }),
-
cold: new S3StorageTier({
-
bucket: 'my-bucket',
-
region: 'us-east-1',
-
credentials: {
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
},
-
}),
+
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
},
-
compression: true,
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
-
promotionStrategy: 'lazy',
-
});
+
placementRules: [
+
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
+
{ pattern: '**/*.{jpg,png,gif,mp4}', tiers: ['warm', 'cold'] },
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
+
})
-
// Store data (cascades to all tiers)
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
+
// just set - rules decide where it goes
+
await storage.set('site:abc/index.html', indexHtml) // → hot + warm + cold
+
await storage.set('site:abc/hero.png', imageData) // → warm + cold
+
await storage.set('site:abc/video.mp4', videoData) // → warm + cold
-
// Retrieve data (bubbles up from cold → warm → hot)
-
const user = await storage.get('user:123');
-
-
// Get data with metadata and source tier
-
const result = await storage.getWithMetadata('user:123');
-
console.log(`Served from ${result.source}`); // 'hot', 'warm', or 'cold'
+
// reads bubble up from wherever it lives
+
const page = await storage.getWithMetadata('site:abc/index.html')
+
console.log(page.source) // 'hot'
-
// Invalidate all keys with prefix
-
await storage.invalidate('user:');
+
const video = await storage.getWithMetadata('site:abc/video.mp4')
+
console.log(video.source) // 'warm'
-
// Renew TTL
-
await storage.touch('user:123');
+
// nuke entire site
+
await storage.invalidate('site:abc/')
```
-
## Core Concepts
+
Hot tier stays small and fast. Warm tier has everything. Cold tier is the source of truth.
-
### Cascading Containment Model
+
## How it works
```
-
┌──────────────────────────────────────────────────────┐
-
│ Cold Storage (S3/Object Storage) │
-
│ • Contains ALL objects (source of truth) │
-
│ • Slowest access, unlimited capacity │
-
├──────────────────────────────────────────────────────┤
-
│ Warm Storage (Disk/Database) │
-
│ • Contains ALL hot objects + additional warm objects │
-
│ • Medium access speed, large capacity │
-
├──────────────────────────────────────────────────────┤
-
│ Hot Storage (Memory) │
-
│ • Contains only the hottest objects │
-
│ • Fastest access, limited capacity │
-
└──────────────────────────────────────────────────────┘
+
┌─────────────────────────────────────────────┐
+
│ Cold (S3) - source of truth, all data │
+
│ ↑ │
+
│ Warm (disk) - everything hot has + more │
+
│ ↑ │
+
│ Hot (memory) - just the hottest stuff │
+
└─────────────────────────────────────────────┘
```
-
**Write Strategy (Cascading Down):**
-
- Write to **hot** → also writes to **warm** and **cold**
-
- Write to **warm** → also writes to **cold**
-
- Write to **cold** → only writes to **cold**
+
Writes cascade **down**. Reads bubble **up**.
-
**Read Strategy (Bubbling Up):**
-
- Check **hot** first → if miss, check **warm** → if miss, check **cold**
-
- On cache miss, optionally promote data up through tiers
+
## Eviction
-
### Selective Tier Placement
-
-
For use cases like static site hosting, you can control which files go into which tiers:
+
Items leave upper tiers through eviction or TTL expiration:
```typescript
-
// Small, critical file (index.html) - store in all tiers for instant serving
-
await storage.set('site:abc/index.html', htmlContent);
+
const storage = new TieredStorage({
+
tiers: {
+
// hot: LRU eviction when size/count limits hit
+
hot: new MemoryStorageTier({
+
maxSizeBytes: 100 * 1024 * 1024,
+
maxItems: 500,
+
}),
-
// Large file (video) - skip hot tier to avoid memory bloat
-
await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] });
+
// warm: evicts when maxSizeBytes hit, policy controls which items go
+
warm: new DiskStorageTier({
+
directory: './cache',
+
maxSizeBytes: 10 * 1024 * 1024 * 1024,
+
evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
+
}),
-
// Medium files (images, CSS) - skip hot, use warm + cold
-
await storage.set('site:abc/style.css', cssData, { skipTiers: ['hot'] });
+
// cold: never evicts, keeps everything
+
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
+
},
+
defaultTTL: 14 * 24 * 60 * 60 * 1000, // TTL checked on read
+
})
```
-
This pattern ensures:
-
- Hot tier stays small and fast (only critical files)
-
- Warm tier caches everything (all site files on disk)
-
- Cold tier is source of truth (all data)
-
-
## API Reference
-
-
### `TieredStorage`
+
A file that hasn't been accessed eventually gets evicted from hot (LRU), then warm (size limit + policy). Next request fetches from cold and promotes it back up.
-
Main orchestrator class for tiered storage.
+
## Placement rules
-
#### Constructor
+
Define once which keys go where, instead of passing `skipTiers` on every `set()`:
```typescript
-
new TieredStorage<T>(config: TieredStorageConfig)
-
```
-
-
**Config Options:**
-
-
```typescript
-
interface TieredStorageConfig {
+
const storage = new TieredStorage({
tiers: {
-
hot?: StorageTier; // Optional: fastest tier (memory/Redis)
-
warm?: StorageTier; // Optional: medium tier (disk/SQLite/Postgres)
-
cold: StorageTier; // Required: slowest tier (S3/object storage)
-
};
-
compression?: boolean; // Auto-compress before storing (default: false)
-
defaultTTL?: number; // Default TTL in milliseconds
-
promotionStrategy?: 'eager' | 'lazy'; // When to promote to upper tiers (default: 'lazy')
-
serialization?: { // Custom serialization (default: JSON)
-
serialize: (data: unknown) => Promise<Uint8Array>;
-
deserialize: (data: Uint8Array) => Promise<unknown>;
-
};
-
}
-
```
+
hot: new MemoryStorageTier({ maxSizeBytes: 50 * 1024 * 1024 }),
+
warm: new DiskStorageTier({ directory: './cache' }),
+
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
+
},
+
placementRules: [
+
// index.html goes everywhere for instant serving
+
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
-
#### Methods
+
// images and video skip hot
+
{ pattern: '**/*.{jpg,png,gif,webp,mp4}', tiers: ['warm', 'cold'] },
-
**`get(key: string): Promise<T | null>`**
+
// assets directory skips hot
+
{ pattern: 'assets/**', tiers: ['warm', 'cold'] },
-
Retrieve data for a key. Returns null if not found or expired.
+
// everything else: warm + cold only
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
+
})
-
**`getWithMetadata(key: string): Promise<StorageResult<T> | null>`**
-
-
Retrieve data with metadata and source tier information.
-
-
```typescript
-
const result = await storage.getWithMetadata('user:123');
-
console.log(result.data); // The actual data
-
console.log(result.source); // 'hot' | 'warm' | 'cold'
-
console.log(result.metadata); // Metadata (size, timestamps, TTL, etc.)
+
// just call set() - rules handle placement
+
await storage.set('site:abc/index.html', html) // → hot + warm + cold
+
await storage.set('site:abc/hero.png', image) // → warm + cold
+
await storage.set('site:abc/assets/font.woff', font) // → warm + cold
+
await storage.set('site:abc/about.html', html) // → warm + cold
```
-
**`set(key: string, data: T, options?: SetOptions): Promise<SetResult>`**
-
-
Store data with optional configuration.
-
-
```typescript
-
await storage.set('key', data, {
-
ttl: 24 * 60 * 60 * 1000, // Custom TTL (24 hours)
-
metadata: { contentType: 'application/json' }, // Custom metadata
-
skipTiers: ['hot'], // Skip specific tiers
-
});
-
```
-
-
**`delete(key: string): Promise<void>`**
+
Rules are evaluated in order. First match wins. Cold is always included.
-
Delete data from all tiers.
+
## API
-
**`exists(key: string): Promise<boolean>`**
+
### `storage.get(key)`
-
Check if a key exists (and hasn't expired).
+
Get data. Returns `null` if missing or expired.
-
**`touch(key: string, ttlMs?: number): Promise<void>`**
+
### `storage.getWithMetadata(key)`
-
Renew TTL for a key. Useful for "keep alive" behavior.
+
Get data plus which tier served it.
-
**`invalidate(prefix: string): Promise<number>`**
+
### `storage.set(key, data, options?)`
-
Delete all keys matching a prefix. Returns number of keys deleted.
+
Store data. Options:
```typescript
-
await storage.invalidate('user:'); // Delete all user keys
-
await storage.invalidate('site:abc/'); // Delete all files for site 'abc'
-
await storage.invalidate(''); // Delete everything
-
```
-
-
**`listKeys(prefix?: string): AsyncIterableIterator<string>`**
-
-
List all keys, optionally filtered by prefix.
-
-
```typescript
-
for await (const key of storage.listKeys('user:')) {
-
console.log(key); // 'user:123', 'user:456', etc.
+
{
+
ttl: 86400000, // custom TTL
+
skipTiers: ['hot'], // skip specific tiers
+
metadata: { ... }, // custom metadata
}
```
-
**`getStats(): Promise<AllTierStats>`**
-
-
Get aggregated statistics across all tiers.
-
-
```typescript
-
const stats = await storage.getStats();
-
console.log(stats.hot); // Hot tier stats (size, items, hits, misses)
-
console.log(stats.hitRate); // Overall hit rate (0-1)
-
```
+
### `storage.delete(key)`
-
**`bootstrapHot(limit?: number): Promise<number>`**
+
Delete from all tiers.
-
Load most frequently accessed items from warm into hot. Returns number of items loaded.
+
### `storage.invalidate(prefix)`
-
```typescript
-
// On server startup: warm up hot tier
-
const loaded = await storage.bootstrapHot(1000); // Load top 1000 items
-
console.log(`Loaded ${loaded} items into hot tier`);
-
```
+
Delete all keys matching prefix. Returns count.
-
**`bootstrapWarm(options?: { limit?: number; sinceDate?: Date }): Promise<number>`**
-
-
Load recent items from cold into warm. Returns number of items loaded.
-
-
```typescript
-
// Load items accessed in last 7 days
-
const loaded = await storage.bootstrapWarm({
-
sinceDate: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000),
-
limit: 10000,
-
});
-
```
+
### `storage.touch(key, ttl?)`
-
**`export(): Promise<StorageSnapshot>`**
+
Renew TTL.
-
Export metadata snapshot for backup or migration.
+
### `storage.listKeys(prefix?)`
-
**`import(snapshot: StorageSnapshot): Promise<void>`**
+
Async iterator over keys.
-
Import metadata snapshot.
+
### `storage.getStats()`
-
**`clear(): Promise<void>`**
+
Stats across all tiers.
-
Clear all data from all tiers. ⚠️ Use with extreme caution!
+
### `storage.bootstrapHot(limit?)`
-
**`clearTier(tier: 'hot' | 'warm' | 'cold'): Promise<void>`**
+
Warm up hot tier from warm tier. Run on startup.
-
Clear a specific tier.
+
### `storage.bootstrapWarm(options?)`
-
### Built-in Storage Tiers
+
Warm up warm tier from cold tier.
-
#### `MemoryStorageTier`
+
## Built-in tiers
-
In-memory storage using TinyLRU for efficient LRU eviction.
+
### MemoryStorageTier
```typescript
-
import { MemoryStorageTier } from 'tiered-storage';
-
-
const tier = new MemoryStorageTier({
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
-
maxItems: 1000, // Optional: max number of items
-
});
+
new MemoryStorageTier({
+
maxSizeBytes: 100 * 1024 * 1024,
+
maxItems: 1000,
+
})
```
-
**Features:**
-
- Battle-tested TinyLRU library
-
- Automatic LRU eviction
-
- Size-based and count-based limits
-
- Single process only (not distributed)
-
-
#### `DiskStorageTier`
+
LRU eviction. Fast. Single process only.
-
Filesystem-based storage with `.meta` files.
+
### DiskStorageTier
```typescript
-
import { DiskStorageTier } from 'tiered-storage';
-
-
const tier = new DiskStorageTier({
+
new DiskStorageTier({
directory: './cache',
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB (optional)
-
evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
-
});
+
maxSizeBytes: 10 * 1024 * 1024 * 1024,
+
evictionPolicy: 'lru', // or 'fifo', 'size'
+
})
```
-
**Features:**
-
- Human-readable file structure
-
- Optional size-based eviction
-
- Three eviction policies: LRU, FIFO, size-based
-
- Atomic writes with `.meta` files
-
- Zero external dependencies
+
Files on disk with `.meta` sidecars.
-
**File structure:**
-
```
-
cache/
-
├── user%3A123 # Data file (encoded key)
-
├── user%3A123.meta # Metadata JSON
-
├── site%3Aabc%2Findex.html
-
└── site%3Aabc%2Findex.html.meta
-
```
-
-
#### `S3StorageTier`
-
-
AWS S3 or S3-compatible object storage.
+
### S3StorageTier
```typescript
-
import { S3StorageTier } from 'tiered-storage';
-
-
// AWS S3 with separate metadata bucket (RECOMMENDED!)
-
const tier = new S3StorageTier({
-
bucket: 'my-data-bucket',
-
metadataBucket: 'my-metadata-bucket', // Stores metadata separately for fast updates
-
region: 'us-east-1',
-
credentials: {
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
-
},
-
prefix: 'cache/', // Optional key prefix
-
});
-
-
// Cloudflare R2 with metadata bucket
-
const r2Tier = new S3StorageTier({
-
bucket: 'my-r2-data-bucket',
-
metadataBucket: 'my-r2-metadata-bucket',
-
region: 'auto',
-
endpoint: 'https://account-id.r2.cloudflarestorage.com',
-
credentials: {
-
accessKeyId: process.env.R2_ACCESS_KEY_ID!,
-
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
-
},
-
});
-
-
// Without metadata bucket (legacy mode - slower, more expensive)
-
const legacyTier = new S3StorageTier({
-
bucket: 'my-bucket',
+
new S3StorageTier({
+
bucket: 'data',
+
metadataBucket: 'metadata', // recommended!
region: 'us-east-1',
-
// No metadataBucket - metadata stored in S3 object metadata fields
-
});
+
})
```
-
**Features:**
-
- Compatible with AWS S3, Cloudflare R2, MinIO, and other S3-compatible services
-
- **Separate metadata bucket support (RECOMMENDED)** - stores metadata as JSON objects for fast, cheap updates
-
- Legacy mode: metadata in S3 object metadata fields (requires object copying for updates)
-
- Efficient batch deletions (up to 1000 keys per request)
-
- Optional key prefixing for multi-tenant scenarios
-
- Typically used as cold tier (source of truth)
-
-
**⚠️ Important:** Without `metadataBucket`, updating metadata (e.g., access counts) requires copying the entire object, which is slow and expensive for large files. Use a separate metadata bucket in production!
+
Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.
-
## Usage Patterns
+
## Custom tiers
-
### Pattern 1: Simple Single-Server Setup
+
Implement `StorageTier`:
```typescript
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
+
interface StorageTier {
+
get(key: string): Promise<Uint8Array | null>
+
set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void>
+
delete(key: string): Promise<void>
+
exists(key: string): Promise<boolean>
+
listKeys(prefix?: string): AsyncIterableIterator<string>
+
deleteMany(keys: string[]): Promise<void>
+
getMetadata(key: string): Promise<StorageMetadata | null>
+
setMetadata(key: string, metadata: StorageMetadata): Promise<void>
+
getStats(): Promise<TierStats>
+
clear(): Promise<void>
-
const storage = new TieredStorage({
-
tiers: {
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
-
warm: new DiskStorageTier({ directory: './cache' }),
-
cold: new DiskStorageTier({ directory: './storage' }),
-
},
-
compression: true,
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
-
});
-
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
-
const user = await storage.get('user:123');
-
```
-
-
### Pattern 2: Static Site Hosting (wisp.place-style)
-
-
```typescript
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
-
-
const storage = new TieredStorage({
-
tiers: {
-
hot: new MemoryStorageTier({
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
-
maxItems: 500,
-
}),
-
warm: new DiskStorageTier({
-
directory: './cache/sites',
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB
-
}),
-
// Cold tier is PDS (fetched on demand via custom tier implementation)
-
},
-
compression: true,
-
defaultTTL: 14 * 24 * 60 * 60 * 1000,
-
promotionStrategy: 'lazy', // Don't auto-promote large files to hot
-
});
-
-
// Store index.html in all tiers (fast access)
-
await storage.set(`${did}/${rkey}/index.html`, htmlBuffer, {
-
metadata: { mimeType: 'text/html', encoding: 'gzip' },
-
});
-
-
// Store large files only in warm + cold (skip hot)
-
await storage.set(`${did}/${rkey}/video.mp4`, videoBuffer, {
-
skipTiers: ['hot'],
-
metadata: { mimeType: 'video/mp4' },
-
});
-
-
// Get file with source tracking
-
const result = await storage.getWithMetadata(`${did}/${rkey}/index.html`);
-
console.log(`Served from ${result.source}`); // Likely 'hot' for index.html
-
-
// Invalidate entire site
-
await storage.invalidate(`${did}/${rkey}/`);
-
-
// Renew TTL when site is accessed
-
await storage.touch(`${did}/${rkey}/index.html`);
-
```
-
-
### Pattern 3: Custom Backend (SQLite)
-
-
Implement the `StorageTier` interface to use any backend:
-
-
```typescript
-
import { StorageTier, StorageMetadata, TierStats } from 'tiered-storage';
-
import Database from 'better-sqlite3';
-
-
class SQLiteStorageTier implements StorageTier {
-
private db: Database.Database;
-
-
constructor(dbPath: string) {
-
this.db = new Database(dbPath);
-
this.db.exec(`
-
CREATE TABLE IF NOT EXISTS cache (
-
key TEXT PRIMARY KEY,
-
data BLOB NOT NULL,
-
metadata TEXT NOT NULL
-
)
-
`);
-
}
-
-
async get(key: string): Promise<Uint8Array | null> {
-
const row = this.db.prepare('SELECT data FROM cache WHERE key = ?').get(key);
-
return row ? new Uint8Array(row.data) : null;
-
}
-
-
async set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void> {
-
this.db.prepare('INSERT OR REPLACE INTO cache (key, data, metadata) VALUES (?, ?, ?)')
-
.run(key, Buffer.from(data), JSON.stringify(metadata));
-
}
-
-
async delete(key: string): Promise<void> {
-
this.db.prepare('DELETE FROM cache WHERE key = ?').run(key);
-
}
-
-
async exists(key: string): Promise<boolean> {
-
const row = this.db.prepare('SELECT 1 FROM cache WHERE key = ?').get(key);
-
return !!row;
-
}
-
-
async *listKeys(prefix?: string): AsyncIterableIterator<string> {
-
const query = prefix
-
? this.db.prepare('SELECT key FROM cache WHERE key LIKE ?')
-
: this.db.prepare('SELECT key FROM cache');
-
-
const rows = prefix ? query.all(`${prefix}%`) : query.all();
-
-
for (const row of rows) {
-
yield row.key;
-
}
-
}
-
-
async deleteMany(keys: string[]): Promise<void> {
-
const placeholders = keys.map(() => '?').join(',');
-
this.db.prepare(`DELETE FROM cache WHERE key IN (${placeholders})`).run(...keys);
-
}
-
-
async getMetadata(key: string): Promise<StorageMetadata | null> {
-
const row = this.db.prepare('SELECT metadata FROM cache WHERE key = ?').get(key);
-
return row ? JSON.parse(row.metadata) : null;
-
}
-
-
async setMetadata(key: string, metadata: StorageMetadata): Promise<void> {
-
this.db.prepare('UPDATE cache SET metadata = ? WHERE key = ?')
-
.run(JSON.stringify(metadata), key);
-
}
-
-
async getStats(): Promise<TierStats> {
-
const row = this.db.prepare('SELECT COUNT(*) as count, SUM(LENGTH(data)) as bytes FROM cache').get();
-
return { items: row.count, bytes: row.bytes || 0 };
-
}
-
-
async clear(): Promise<void> {
-
this.db.prepare('DELETE FROM cache').run();
-
}
+
// Optional: combine get + getMetadata for better performance
+
getWithMetadata?(key: string): Promise<{ data: Uint8Array; metadata: StorageMetadata } | null>
}
-
-
// Use it
-
const storage = new TieredStorage({
-
tiers: {
-
warm: new SQLiteStorageTier('./cache.db'),
-
cold: new DiskStorageTier({ directory: './storage' }),
-
},
-
});
```
-
## Running Examples
+
The optional `getWithMetadata` method returns both data and metadata in a single call. Implement it if your backend can fetch both efficiently (e.g., parallel I/O, single query). Falls back to separate `get()` + `getMetadata()` calls if not implemented.
-
### Interactive Demo Server
-
-
Run a **real HTTP server** that serves the example site using tiered storage:
+
## Running the demo
```bash
-
# Configure S3 credentials first (copy .env.example to .env and fill in)
-
cp .env.example .env
-
-
# Start the demo server
+
cp .env.example .env # add S3 creds
bun run serve
```
-
Then visit:
-
- **http://localhost:3000/** - The demo site served from tiered storage
-
- **http://localhost:3000/admin/stats** - Live cache statistics dashboard
-
-
Watch the console to see which tier serves each request:
-
- 🔥 **Hot tier (memory)** - index.html served instantly
-
- 💾 **Warm tier (disk)** - Other pages served from disk cache
-
- ☁️ **Cold tier (S3)** - First access fetches from S3, then cached
-
-
### Command-Line Examples
-
-
Or run the non-interactive examples:
-
-
```bash
-
bun run example
-
```
-
-
The examples include:
-
- **Basic CRUD operations** with statistics tracking
-
- **Static site hosting** using the real site in `example-site/` directory
-
- **Bootstrap demonstrations** (warming caches from lower tiers)
-
- **Promotion strategy comparisons** (eager vs lazy)
-
-
The `example-site/` directory contains a complete static website with:
-
- `index.html` - Stored in hot + warm + cold (instant serving)
-
- `about.html`, `docs.html` - Stored in warm + cold (skips hot)
-
- `style.css`, `script.js` - Stored in warm + cold (skips hot)
-
-
This demonstrates the exact pattern you'd use for wisp.place: critical files in memory, everything else on disk/S3.
-
-
## Testing
-
-
```bash
-
bun test
-
```
-
-
## Development
-
-
```bash
-
# Install dependencies
-
bun install
+
Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.
-
# Type check
-
bun run check
-
-
# Build
-
bun run build
-
-
# Run tests
-
bun test
-
-
```
## License
MIT
+17 -10
serve-example.ts
···
prefix: 'demo-sites/',
}),
},
+
placementRules: [
+
// index.html goes to all tiers for instant serving
+
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
+
+
// everything else: warm + cold only
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
compression: true,
defaultTTL: 14 * 24 * 60 * 60 * 1000,
promotionStrategy: 'lazy',
···
console.log('\n📦 Loading example site into tiered storage...\n');
const files = [
-
{ name: 'index.html', skipTiers: [], mimeType: 'text/html' },
-
{ name: 'about.html', skipTiers: ['hot'], mimeType: 'text/html' },
-
{ name: 'docs.html', skipTiers: ['hot'], mimeType: 'text/html' },
-
{ name: 'style.css', skipTiers: ['hot'], mimeType: 'text/css' },
-
{ name: 'script.js', skipTiers: ['hot'], mimeType: 'application/javascript' },
+
{ name: 'index.html', mimeType: 'text/html' },
+
{ name: 'about.html', mimeType: 'text/html' },
+
{ name: 'docs.html', mimeType: 'text/html' },
+
{ name: 'style.css', mimeType: 'text/css' },
+
{ name: 'script.js', mimeType: 'application/javascript' },
];
for (const file of files) {
···
const key = `${siteId}/${siteName}/${file.name}`;
await storage.set(key, content, {
-
skipTiers: file.skipTiers as ('hot' | 'warm')[],
metadata: { mimeType: file.mimeType },
});
-
const tierInfo =
-
file.skipTiers.length === 0
-
? '🔥 hot + 💾 warm + ☁️ cold'
-
: `💾 warm + ☁️ cold (skipped hot)`;
+
// Determine which tiers this file went to based on placement rules
+
const isIndex = file.name === 'index.html';
+
const tierInfo = isIndex
+
? '🔥 hot + 💾 warm + ☁️ cold'
+
: '💾 warm + ☁️ cold (skipped hot)';
const sizeKB = (content.length / 1024).toFixed(2);
console.log(` ✓ ${file.name.padEnd(15)} ${sizeKB.padStart(6)} KB → ${tierInfo}`);
}
+112 -55
src/TieredStorage.ts
···
StorageResult,
SetResult,
StorageMetadata,
+
StorageTier,
AllTierStats,
StorageSnapshot,
-
} from './types/index.js';
+
PlacementRule,
+
} from './types/index';
import { compress, decompress } from './utils/compression.js';
import { defaultSerialize, defaultDeserialize } from './utils/serialization.js';
import { calculateChecksum } from './utils/checksum.js';
+
import { matchGlob } from './utils/glob.js';
/**
* Main orchestrator for tiered storage system.
···
async getWithMetadata(key: string): Promise<StorageResult<T> | null> {
// 1. Check hot tier first
if (this.config.tiers.hot) {
-
const data = await this.config.tiers.hot.get(key);
-
if (data) {
-
const metadata = await this.config.tiers.hot.getMetadata(key);
-
if (!metadata) {
-
await this.delete(key);
-
} else if (this.isExpired(metadata)) {
+
const result = await this.getFromTier(this.config.tiers.hot, key);
+
if (result) {
+
if (this.isExpired(result.metadata)) {
await this.delete(key);
return null;
-
} else {
-
await this.updateAccessStats(key, 'hot');
-
return {
-
data: (await this.deserializeData(data)) as T,
-
metadata,
-
source: 'hot',
-
};
}
+
// Fire-and-forget access stats update (non-critical)
+
void this.updateAccessStats(key, 'hot');
+
return {
+
data: (await this.deserializeData(result.data)) as T,
+
metadata: result.metadata,
+
source: 'hot',
+
};
}
}
// 2. Check warm tier
if (this.config.tiers.warm) {
-
const data = await this.config.tiers.warm.get(key);
-
if (data) {
-
const metadata = await this.config.tiers.warm.getMetadata(key);
-
if (!metadata) {
-
await this.delete(key);
-
} else if (this.isExpired(metadata)) {
+
const result = await this.getFromTier(this.config.tiers.warm, key);
+
if (result) {
+
if (this.isExpired(result.metadata)) {
await this.delete(key);
return null;
-
} else {
-
if (this.config.tiers.hot && this.config.promotionStrategy === 'eager') {
-
await this.config.tiers.hot.set(key, data, metadata);
-
}
-
-
await this.updateAccessStats(key, 'warm');
-
return {
-
data: (await this.deserializeData(data)) as T,
-
metadata,
-
source: 'warm',
-
};
+
}
+
// Eager promotion to hot tier (awaited - guaranteed to complete)
+
if (this.config.tiers.hot && this.config.promotionStrategy === 'eager') {
+
await this.config.tiers.hot.set(key, result.data, result.metadata);
}
+
// Fire-and-forget access stats update (non-critical)
+
void this.updateAccessStats(key, 'warm');
+
return {
+
data: (await this.deserializeData(result.data)) as T,
+
metadata: result.metadata,
+
source: 'warm',
+
};
}
}
// 3. Check cold tier (source of truth)
-
const data = await this.config.tiers.cold.get(key);
-
if (data) {
-
const metadata = await this.config.tiers.cold.getMetadata(key);
-
if (!metadata) {
-
await this.config.tiers.cold.delete(key);
-
return null;
-
}
-
-
if (this.isExpired(metadata)) {
+
const result = await this.getFromTier(this.config.tiers.cold, key);
+
if (result) {
+
if (this.isExpired(result.metadata)) {
await this.delete(key);
return null;
}
// Promote to warm and hot (if configured)
+
// Eager promotion is awaited to guarantee completion
if (this.config.promotionStrategy === 'eager') {
+
const promotions: Promise<void>[] = [];
if (this.config.tiers.warm) {
-
await this.config.tiers.warm.set(key, data, metadata);
+
promotions.push(this.config.tiers.warm.set(key, result.data, result.metadata));
}
if (this.config.tiers.hot) {
-
await this.config.tiers.hot.set(key, data, metadata);
+
promotions.push(this.config.tiers.hot.set(key, result.data, result.metadata));
}
+
await Promise.all(promotions);
}
-
await this.updateAccessStats(key, 'cold');
+
// Fire-and-forget access stats update (non-critical)
+
void this.updateAccessStats(key, 'cold');
return {
-
data: (await this.deserializeData(data)) as T,
-
metadata,
+
data: (await this.deserializeData(result.data)) as T,
+
metadata: result.metadata,
source: 'cold',
};
}
···
}
/**
+
* Get data and metadata from a tier using the most efficient method.
+
*
+
* @remarks
+
* Uses the tier's getWithMetadata if available, otherwise falls back
+
* to separate get() and getMetadata() calls.
+
*/
+
private async getFromTier(
+
tier: StorageTier,
+
key: string
+
): Promise<{ data: Uint8Array; metadata: StorageMetadata } | null> {
+
// Use optimized combined method if available
+
if (tier.getWithMetadata) {
+
return tier.getWithMetadata(key);
+
}
+
+
// Fallback: separate calls
+
const data = await tier.get(key);
+
if (!data) {
+
return null;
+
}
+
const metadata = await tier.getMetadata(key);
+
if (!metadata) {
+
return null;
+
}
+
return { data, metadata };
+
}
+
+
/**
* Store data with optional configuration.
*
* @param key - The key to store under
···
// 3. Create metadata
const metadata = this.createMetadata(key, finalData, options);
-
// 4. Write to all tiers (cascading down)
+
// 4. Determine which tiers to write to
+
const allowedTiers = this.getTiersForKey(key, options?.skipTiers);
+
+
// 5. Write to tiers
const tiersWritten: ('hot' | 'warm' | 'cold')[] = [];
-
// Write to hot (if configured and not skipped)
-
if (this.config.tiers.hot && !options?.skipTiers?.includes('hot')) {
+
if (this.config.tiers.hot && allowedTiers.includes('hot')) {
await this.config.tiers.hot.set(key, finalData, metadata);
tiersWritten.push('hot');
+
}
-
// Hot writes cascade to warm
-
if (this.config.tiers.warm && !options?.skipTiers?.includes('warm')) {
-
await this.config.tiers.warm.set(key, finalData, metadata);
-
tiersWritten.push('warm');
-
}
-
} else if (this.config.tiers.warm && !options?.skipTiers?.includes('warm')) {
-
// Write to warm (if hot skipped)
+
if (this.config.tiers.warm && allowedTiers.includes('warm')) {
await this.config.tiers.warm.set(key, finalData, metadata);
tiersWritten.push('warm');
}
···
tiersWritten.push('cold');
return { key, metadata, tiersWritten };
+
}
+
+
/**
+
* Determine which tiers a key should be written to.
+
*
+
* @param key - The key being stored
+
* @param skipTiers - Explicit tiers to skip (overrides placement rules)
+
* @returns Array of tiers to write to
+
*
+
* @remarks
+
* Priority: skipTiers option > placementRules > all configured tiers
+
*/
+
private getTiersForKey(
+
key: string,
+
skipTiers?: ('hot' | 'warm')[]
+
): ('hot' | 'warm' | 'cold')[] {
+
// If explicit skipTiers provided, use that
+
if (skipTiers && skipTiers.length > 0) {
+
const allTiers: ('hot' | 'warm' | 'cold')[] = ['hot', 'warm', 'cold'];
+
return allTiers.filter((t) => !skipTiers.includes(t as 'hot' | 'warm'));
+
}
+
+
// Check placement rules
+
if (this.config.placementRules) {
+
for (const rule of this.config.placementRules) {
+
if (matchGlob(rule.pattern, key)) {
+
// Ensure cold is always included
+
if (!rule.tiers.includes('cold')) {
+
return [...rule.tiers, 'cold'];
+
}
+
return rule.tiers;
+
}
+
}
+
}
+
+
// Default: write to all configured tiers
+
return ['hot', 'warm', 'cold'];
}
/**
+2
src/index.ts
···
StorageTier,
StorageMetadata,
TierStats,
+
TierGetResult,
AllTierStats,
TieredStorageConfig,
+
PlacementRule,
SetOptions,
StorageResult,
SetResult,
+36 -11
src/tiers/DiskStorageTier.ts
···
import { readFile, writeFile, unlink, readdir, stat, mkdir, rm, rename } from 'node:fs/promises';
import { existsSync } from 'node:fs';
import { join, dirname } from 'node:path';
-
import type { StorageTier, StorageMetadata, TierStats } from '../types/index.js';
+
import type { StorageTier, StorageMetadata, TierStats, TierGetResult } from '../types/index.js';
import { encodeKey } from '../utils/path-encoding.js';
/**
···
try {
const data = await readFile(filePath);
+
return new Uint8Array(data);
+
} catch (error) {
+
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
+
return null;
+
}
+
throw error;
+
}
+
}
-
const metadata = await this.getMetadata(key);
-
if (metadata) {
-
metadata.lastAccessed = new Date();
-
metadata.accessCount++;
-
await this.setMetadata(key, metadata);
+
/**
+
* Retrieve data and metadata together in a single operation.
+
*
+
* @param key - The key to retrieve
+
* @returns The data and metadata, or null if not found
+
*
+
* @remarks
+
* Reads data and metadata files in parallel for better performance.
+
*/
+
async getWithMetadata(key: string): Promise<TierGetResult | null> {
+
const filePath = this.getFilePath(key);
+
const metaPath = this.getMetaPath(key);
+
+
try {
+
// Read data and metadata in parallel
+
const [dataBuffer, metaContent] = await Promise.all([
+
readFile(filePath),
+
readFile(metaPath, 'utf-8'),
+
]);
-
const entry = this.metadataIndex.get(key);
-
if (entry) {
-
entry.lastAccessed = metadata.lastAccessed;
-
}
+
const metadata = JSON.parse(metaContent) as StorageMetadata;
+
+
// Convert date strings back to Date objects
+
metadata.createdAt = new Date(metadata.createdAt);
+
metadata.lastAccessed = new Date(metadata.lastAccessed);
+
if (metadata.ttl) {
+
metadata.ttl = new Date(metadata.ttl);
}
-
return new Uint8Array(data);
+
return { data: new Uint8Array(dataBuffer), metadata };
} catch (error) {
if ((error as NodeJS.ErrnoException).code === 'ENOENT') {
return null;
+19 -1
src/tiers/MemoryStorageTier.ts
···
import { lru } from 'tiny-lru';
-
import type { StorageTier, StorageMetadata, TierStats } from '../types/index.js';
+
import type { StorageTier, StorageMetadata, TierStats, TierGetResult } from '../types/index.js';
interface CacheEntry {
data: Uint8Array;
···
this.stats.hits++;
return entry.data;
+
}
+
+
/**
+
* Retrieve data and metadata together in a single cache lookup.
+
*
+
* @param key - The key to retrieve
+
* @returns The data and metadata, or null if not found
+
*/
+
async getWithMetadata(key: string): Promise<TierGetResult | null> {
+
const entry = this.cache.get(key);
+
+
if (!entry) {
+
this.stats.misses++;
+
return null;
+
}
+
+
this.stats.hits++;
+
return { data: entry.data, metadata: entry.metadata };
}
async set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void> {
+70 -1
src/tiers/S3StorageTier.ts
···
type S3ClientConfig,
} from '@aws-sdk/client-s3';
import type { Readable } from 'node:stream';
-
import type { StorageTier, StorageMetadata, TierStats } from '../types/index.js';
+
import type { StorageTier, StorageMetadata, TierStats, TierGetResult } from '../types/index.js';
/**
* Configuration for S3StorageTier.
···
}
return await this.streamToUint8Array(response.Body as Readable);
+
} catch (error) {
+
if (this.isNoSuchKeyError(error)) {
+
return null;
+
}
+
throw error;
+
}
+
}
+
+
/**
+
* Retrieve data and metadata together in a single operation.
+
*
+
* @param key - The key to retrieve
+
* @returns The data and metadata, or null if not found
+
*
+
* @remarks
+
* When using a separate metadata bucket, fetches data and metadata in parallel.
+
* Otherwise, uses the data object's embedded metadata.
+
*/
+
async getWithMetadata(key: string): Promise<TierGetResult | null> {
+
const s3Key = this.getS3Key(key);
+
+
try {
+
if (this.metadataBucket) {
+
// Fetch data and metadata in parallel
+
const [dataResponse, metadataResponse] = await Promise.all([
+
this.client.send(new GetObjectCommand({
+
Bucket: this.config.bucket,
+
Key: s3Key,
+
})),
+
this.client.send(new GetObjectCommand({
+
Bucket: this.metadataBucket,
+
Key: s3Key + '.meta',
+
})),
+
]);
+
+
if (!dataResponse.Body || !metadataResponse.Body) {
+
return null;
+
}
+
+
const [data, metaBuffer] = await Promise.all([
+
this.streamToUint8Array(dataResponse.Body as Readable),
+
this.streamToUint8Array(metadataResponse.Body as Readable),
+
]);
+
+
const json = new TextDecoder().decode(metaBuffer);
+
const metadata = JSON.parse(json) as StorageMetadata;
+
metadata.createdAt = new Date(metadata.createdAt);
+
metadata.lastAccessed = new Date(metadata.lastAccessed);
+
if (metadata.ttl) {
+
metadata.ttl = new Date(metadata.ttl);
+
}
+
+
return { data, metadata };
+
} else {
+
// Get data with embedded metadata from response headers
+
const response = await this.client.send(new GetObjectCommand({
+
Bucket: this.config.bucket,
+
Key: s3Key,
+
}));
+
+
if (!response.Body || !response.Metadata) {
+
return null;
+
}
+
+
const data = await this.streamToUint8Array(response.Body as Readable);
+
const metadata = this.s3ToMetadata(response.Metadata);
+
+
return { data, metadata };
+
}
} catch (error) {
if (this.isNoSuchKeyError(error)) {
return null;
+67
src/types/index.ts
···
* }
* ```
*/
+
/**
+
* Result from a combined get+metadata operation on a tier.
+
*/
+
export interface TierGetResult {
+
/** The retrieved data */
+
data: Uint8Array;
+
/** Metadata associated with the data */
+
metadata: StorageMetadata;
+
}
+
export interface StorageTier {
/**
* Retrieve data for a key.
···
* @returns The data as a Uint8Array, or null if not found
*/
get(key: string): Promise<Uint8Array | null>;
+
+
/**
+
* Retrieve data and metadata together in a single operation.
+
*
+
* @param key - The key to retrieve
+
* @returns The data and metadata, or null if not found
+
*
+
* @remarks
+
* This is more efficient than calling get() and getMetadata() separately,
+
* especially for disk and network-based tiers.
+
*/
+
getWithMetadata?(key: string): Promise<TierGetResult | null>;
/**
* Store data with associated metadata.
···
}
/**
+
* Rule for automatic tier placement based on key patterns.
+
*
+
* @remarks
+
* Rules are evaluated in order. First matching rule wins.
+
* Use this to define which keys go to which tiers without
+
* specifying skipTiers on every set() call.
+
*
+
* @example
+
* ```typescript
+
* placementRules: [
+
* { pattern: 'index.html', tiers: ['hot', 'warm', 'cold'] },
+
* { pattern: '*.html', tiers: ['warm', 'cold'] },
+
* { pattern: 'assets/**', tiers: ['warm', 'cold'] },
+
* { pattern: '**', tiers: ['warm', 'cold'] }, // default
+
* ]
+
* ```
+
*/
+
export interface PlacementRule {
+
/**
+
* Glob pattern to match against keys.
+
*
+
* @remarks
+
* Supports basic globs:
+
* - `*` matches any characters except `/`
+
* - `**` matches any characters including `/`
+
* - Exact matches work too: `index.html`
+
*/
+
pattern: string;
+
+
/**
+
* Which tiers to write to for matching keys.
+
*
+
* @remarks
+
* Cold is always included (source of truth).
+
* Use `['hot', 'warm', 'cold']` for critical files.
+
* Use `['warm', 'cold']` for large files.
+
* Use `['cold']` for archival only.
+
*/
+
tiers: ('hot' | 'warm' | 'cold')[];
+
}
+
+
/**
* Configuration for the TieredStorage system.
*
* @typeParam T - The type of data being stored (for serialization)
···
/** Required cold tier - slowest, largest capacity (e.g., S3, R2, object storage) */
cold: StorageTier;
};
+
+
/** Rules for automatic tier placement based on key patterns. First match wins. */
+
placementRules?: PlacementRule[];
/**
* Whether to automatically compress data before storing.
+40
src/utils/glob.ts
···
+
/**
+
* Simple glob pattern matching for key placement rules.
+
*
+
* Supports:
+
* - `*` matches any characters except `/`
+
* - `**` matches any characters including `/` (including empty string)
+
* - `{a,b,c}` matches any of the alternatives
+
* - Exact strings match exactly
+
*/
+
export function matchGlob(pattern: string, key: string): boolean {
+
// Handle exact match
+
if (!pattern.includes('*') && !pattern.includes('{')) {
+
return pattern === key;
+
}
+
+
// Escape regex special chars (except * and {})
+
let regex = pattern.replace(/[.+^$|\\()[\]]/g, '\\$&');
+
+
// Handle {a,b,c} alternation
+
regex = regex.replace(/\{([^}]+)\}/g, (_, alts) => `(${alts.split(',').join('|')})`);
+
+
// Use placeholder to avoid double-processing
+
const DOUBLE = '\x00DOUBLE\x00';
+
const SINGLE = '\x00SINGLE\x00';
+
+
// Mark ** and * with placeholders
+
regex = regex.replace(/\*\*/g, DOUBLE);
+
regex = regex.replace(/\*/g, SINGLE);
+
+
// Replace placeholders with regex patterns
+
// ** matches anything (including /)
+
// When followed by /, it's optional (matches zero or more path segments)
+
regex = regex
+
.replace(new RegExp(`${DOUBLE}/`, 'g'), '(?:.*/)?') // **/ -> optional path prefix
+
.replace(new RegExp(`/${DOUBLE}`, 'g'), '(?:/.*)?') // /** -> optional path suffix
+
.replace(new RegExp(DOUBLE, 'g'), '.*') // ** alone -> match anything
+
.replace(new RegExp(SINGLE, 'g'), '[^/]*'); // * -> match non-slash
+
+
return new RegExp(`^${regex}$`).test(key);
+
}
+175
test/TieredStorage.test.ts
···
expect(stats.hot?.items).toBe(2);
});
});
+
+
describe('Placement Rules', () => {
+
it('should place index.html in all tiers based on rule', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
+
});
+
+
await storage.set('site:abc/index.html', { content: 'hello' });
+
+
expect(await hot.exists('site:abc/index.html')).toBe(true);
+
expect(await warm.exists('site:abc/index.html')).toBe(true);
+
expect(await cold.exists('site:abc/index.html')).toBe(true);
+
});
+
+
it('should skip hot tier for non-matching files', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
+
});
+
+
await storage.set('site:abc/about.html', { content: 'about' });
+
+
expect(await hot.exists('site:abc/about.html')).toBe(false);
+
expect(await warm.exists('site:abc/about.html')).toBe(true);
+
expect(await cold.exists('site:abc/about.html')).toBe(true);
+
});
+
+
it('should match directory patterns', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: 'assets/**', tiers: ['warm', 'cold'] },
+
{ pattern: '**', tiers: ['hot', 'warm', 'cold'] },
+
],
+
});
+
+
await storage.set('assets/images/logo.png', { data: 'png' });
+
await storage.set('index.html', { data: 'html' });
+
+
// assets/** should skip hot
+
expect(await hot.exists('assets/images/logo.png')).toBe(false);
+
expect(await warm.exists('assets/images/logo.png')).toBe(true);
+
+
// everything else goes to all tiers
+
expect(await hot.exists('index.html')).toBe(true);
+
});
+
+
it('should match file extension patterns', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: '**/*.{jpg,png,gif,mp4}', tiers: ['warm', 'cold'] },
+
{ pattern: '**', tiers: ['hot', 'warm', 'cold'] },
+
],
+
});
+
+
await storage.set('site/hero.png', { data: 'image' });
+
await storage.set('site/video.mp4', { data: 'video' });
+
await storage.set('site/index.html', { data: 'html' });
+
+
// Images and video skip hot
+
expect(await hot.exists('site/hero.png')).toBe(false);
+
expect(await hot.exists('site/video.mp4')).toBe(false);
+
+
// HTML goes everywhere
+
expect(await hot.exists('site/index.html')).toBe(true);
+
});
+
+
it('should use first matching rule', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
// Specific rule first
+
{ pattern: 'assets/critical.css', tiers: ['hot', 'warm', 'cold'] },
+
// General rule second
+
{ pattern: 'assets/**', tiers: ['warm', 'cold'] },
+
{ pattern: '**', tiers: ['warm', 'cold'] },
+
],
+
});
+
+
await storage.set('assets/critical.css', { data: 'css' });
+
await storage.set('assets/style.css', { data: 'css' });
+
+
// critical.css matches first rule -> hot
+
expect(await hot.exists('assets/critical.css')).toBe(true);
+
+
// style.css matches second rule -> no hot
+
expect(await hot.exists('assets/style.css')).toBe(false);
+
});
+
+
it('should allow skipTiers to override placement rules', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: '**', tiers: ['hot', 'warm', 'cold'] },
+
],
+
});
+
+
// Explicit skipTiers should override the rule
+
await storage.set('large-file.bin', { data: 'big' }, { skipTiers: ['hot'] });
+
+
expect(await hot.exists('large-file.bin')).toBe(false);
+
expect(await warm.exists('large-file.bin')).toBe(true);
+
expect(await cold.exists('large-file.bin')).toBe(true);
+
});
+
+
it('should always include cold tier even if not in rule', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
// Rule doesn't include cold (should be auto-added)
+
{ pattern: '**', tiers: ['hot', 'warm'] },
+
],
+
});
+
+
await storage.set('test-key', { data: 'test' });
+
+
expect(await cold.exists('test-key')).toBe(true);
+
});
+
+
it('should write to all tiers when no rules match', async () => {
+
const hot = new MemoryStorageTier({ maxSizeBytes: 1024 * 1024 });
+
const warm = new DiskStorageTier({ directory: `${testDir}/warm` });
+
const cold = new DiskStorageTier({ directory: `${testDir}/cold` });
+
+
const storage = new TieredStorage({
+
tiers: { hot, warm, cold },
+
placementRules: [
+
{ pattern: 'specific-pattern-only', tiers: ['warm', 'cold'] },
+
],
+
});
+
+
// This doesn't match any rule
+
await storage.set('other-key', { data: 'test' });
+
+
expect(await hot.exists('other-key')).toBe(true);
+
expect(await warm.exists('other-key')).toBe(true);
+
expect(await cold.exists('other-key')).toBe(true);
+
});
+
});
});
+95
test/glob.test.ts
···
+
import { describe, it, expect } from 'vitest';
+
import { matchGlob } from '../src/utils/glob.js';
+
+
describe('matchGlob', () => {
+
describe('exact matches', () => {
+
it('should match exact strings', () => {
+
expect(matchGlob('index.html', 'index.html')).toBe(true);
+
expect(matchGlob('index.html', 'about.html')).toBe(false);
+
});
+
+
it('should match paths exactly', () => {
+
expect(matchGlob('site/index.html', 'site/index.html')).toBe(true);
+
expect(matchGlob('site/index.html', 'other/index.html')).toBe(false);
+
});
+
});
+
+
describe('* wildcard', () => {
+
it('should match any characters except /', () => {
+
expect(matchGlob('*.html', 'index.html')).toBe(true);
+
expect(matchGlob('*.html', 'about.html')).toBe(true);
+
expect(matchGlob('*.html', 'style.css')).toBe(false);
+
});
+
+
it('should not match across path separators', () => {
+
expect(matchGlob('*.html', 'dir/index.html')).toBe(false);
+
});
+
+
it('should work with prefix and suffix', () => {
+
expect(matchGlob('index.*', 'index.html')).toBe(true);
+
expect(matchGlob('index.*', 'index.css')).toBe(true);
+
expect(matchGlob('index.*', 'about.html')).toBe(false);
+
});
+
});
+
+
describe('** wildcard', () => {
+
it('should match any characters including /', () => {
+
expect(matchGlob('**', 'anything')).toBe(true);
+
expect(matchGlob('**', 'path/to/file.txt')).toBe(true);
+
});
+
+
it('should match deeply nested paths', () => {
+
expect(matchGlob('**/index.html', 'index.html')).toBe(true);
+
expect(matchGlob('**/index.html', 'site/index.html')).toBe(true);
+
expect(matchGlob('**/index.html', 'a/b/c/index.html')).toBe(true);
+
expect(matchGlob('**/index.html', 'a/b/c/about.html')).toBe(false);
+
});
+
+
it('should match directory prefixes', () => {
+
expect(matchGlob('assets/**', 'assets/style.css')).toBe(true);
+
expect(matchGlob('assets/**', 'assets/images/logo.png')).toBe(true);
+
expect(matchGlob('assets/**', 'other/style.css')).toBe(false);
+
});
+
+
it('should match in the middle of a path', () => {
+
expect(matchGlob('site/**/index.html', 'site/index.html')).toBe(true);
+
expect(matchGlob('site/**/index.html', 'site/pages/index.html')).toBe(true);
+
expect(matchGlob('site/**/index.html', 'site/a/b/c/index.html')).toBe(true);
+
});
+
});
+
+
describe('{a,b,c} alternation', () => {
+
it('should match any of the alternatives', () => {
+
expect(matchGlob('*.{html,css,js}', 'index.html')).toBe(true);
+
expect(matchGlob('*.{html,css,js}', 'style.css')).toBe(true);
+
expect(matchGlob('*.{html,css,js}', 'app.js')).toBe(true);
+
expect(matchGlob('*.{html,css,js}', 'image.png')).toBe(false);
+
});
+
+
it('should work with ** and alternation', () => {
+
expect(matchGlob('**/*.{jpg,png,gif}', 'logo.png')).toBe(true);
+
expect(matchGlob('**/*.{jpg,png,gif}', 'images/logo.png')).toBe(true);
+
expect(matchGlob('**/*.{jpg,png,gif}', 'a/b/photo.jpg')).toBe(true);
+
expect(matchGlob('**/*.{jpg,png,gif}', 'style.css')).toBe(false);
+
});
+
});
+
+
describe('edge cases', () => {
+
it('should handle empty strings', () => {
+
expect(matchGlob('', '')).toBe(true);
+
expect(matchGlob('', 'something')).toBe(false);
+
expect(matchGlob('**', '')).toBe(true);
+
});
+
+
it('should escape regex special characters', () => {
+
expect(matchGlob('file.txt', 'file.txt')).toBe(true);
+
expect(matchGlob('file.txt', 'filextxt')).toBe(false);
+
expect(matchGlob('file[1].txt', 'file[1].txt')).toBe(true);
+
});
+
+
it('should handle keys with colons (common in storage)', () => {
+
expect(matchGlob('site:*/index.html', 'site:abc/index.html')).toBe(true);
+
expect(matchGlob('site:**/index.html', 'site:abc/pages/index.html')).toBe(true);
+
});
+
});
+
});