···
3
-
A lightweight, pluggable tiered storage library that orchestrates caching across hot (memory), warm (disk/database), and cold (S3/object storage) tiers.
3
+
Cascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.
7
-
- **Cascading Containment Model**: Hot ⊆ Warm ⊆ Cold (lower tiers contain all data from upper tiers)
8
-
- **Pluggable Backends**: Bring your own Redis, Postgres, SQLite, or use built-in implementations
9
-
- **Automatic Promotion**: Configurable eager/lazy promotion strategies for cache warming
10
-
- **TTL Management**: Per-key TTL with automatic expiration and renewal
11
-
- **Prefix Invalidation**: Efficiently delete groups of keys by prefix
12
-
- **Bootstrap Support**: Warm up caches from lower tiers on startup
13
-
- **Compression**: Optional transparent gzip compression
14
-
- **TypeScript First**: Full type safety with comprehensive TSDoc comments
15
-
- **Zero Forced Dependencies**: Only require what you use
7
+
- **Cascading writes** - data flows down through all tiers
8
+
- **Bubbling reads** - check hot first, fall back to warm, then cold
9
+
- **Pluggable backends** - memory, disk, S3, or implement your own
10
+
- **Selective placement** - skip tiers for big files that don't need memory caching
11
+
- **Prefix invalidation** - `invalidate('user:')` nukes all user keys
12
+
- **Optional compression** - transparent gzip
npm install tiered-storage
22
-
bun add tiered-storage
28
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage';
23
+
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'
const storage = new TieredStorage({
32
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }), // 100MB
27
+
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
warm: new DiskStorageTier({ directory: './cache' }),
34
-
cold: new S3StorageTier({
35
-
bucket: 'my-bucket',
36
-
region: 'us-east-1',
38
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
39
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
29
+
cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
44
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
45
-
promotionStrategy: 'lazy',
48
-
// Store data (cascades to all tiers)
49
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
51
-
// Retrieve data (bubbles up from cold → warm → hot)
52
-
const user = await storage.get('user:123');
54
-
// Get data with metadata and source tier
55
-
const result = await storage.getWithMetadata('user:123');
56
-
console.log(`Served from ${result.source}`); // 'hot', 'warm', or 'cold'
58
-
// Invalidate all keys with prefix
59
-
await storage.invalidate('user:');
62
-
await storage.touch('user:123');
34
+
// write cascades down
35
+
await storage.set('user:123', { name: 'Alice' })
38
+
const user = await storage.get('user:123')
67
-
### Cascading Containment Model
40
+
// see where it came from
41
+
const result = await storage.getWithMetadata('user:123')
42
+
console.log(result.source) // 'hot', 'warm', or 'cold'
45
+
await storage.invalidate('user:')
70
-
┌──────────────────────────────────────────────────────┐
71
-
│ Cold Storage (S3/Object Storage) │
72
-
│ • Contains ALL objects (source of truth) │
73
-
│ • Slowest access, unlimited capacity │
74
-
├──────────────────────────────────────────────────────┤
75
-
│ Warm Storage (Disk/Database) │
76
-
│ • Contains ALL hot objects + additional warm objects │
77
-
│ • Medium access speed, large capacity │
78
-
├──────────────────────────────────────────────────────┤
79
-
│ Hot Storage (Memory) │
80
-
│ • Contains only the hottest objects │
81
-
│ • Fastest access, limited capacity │
82
-
└──────────────────────────────────────────────────────┘
85
-
**Write Strategy (Cascading Down):**
86
-
- Write to **hot** → also writes to **warm** and **cold**
87
-
- Write to **warm** → also writes to **cold**
88
-
- Write to **cold** → only writes to **cold**
90
-
**Read Strategy (Bubbling Up):**
91
-
- Check **hot** first → if miss, check **warm** → if miss, check **cold**
92
-
- On cache miss, optionally promote data up through tiers
94
-
### Selective Tier Placement
96
-
For use cases like static site hosting, you can control which files go into which tiers:
99
-
// Small, critical file (index.html) - store in all tiers for instant serving
100
-
await storage.set('site:abc/index.html', htmlContent);
102
-
// Large file (video) - skip hot tier to avoid memory bloat
103
-
await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] });
105
-
// Medium files (images, CSS) - skip hot, use warm + cold
106
-
await storage.set('site:abc/style.css', cssData, { skipTiers: ['hot'] });
109
-
This pattern ensures:
110
-
- Hot tier stays small and fast (only critical files)
111
-
- Warm tier caches everything (all site files on disk)
112
-
- Cold tier is source of truth (all data)
116
-
### `TieredStorage`
118
-
Main orchestrator class for tiered storage.
123
-
new TieredStorage<T>(config: TieredStorageConfig)
126
-
**Config Options:**
129
-
interface TieredStorageConfig {
131
-
hot?: StorageTier; // Optional: fastest tier (memory/Redis)
132
-
warm?: StorageTier; // Optional: medium tier (disk/SQLite/Postgres)
133
-
cold: StorageTier; // Required: slowest tier (S3/object storage)
135
-
compression?: boolean; // Auto-compress before storing (default: false)
136
-
defaultTTL?: number; // Default TTL in milliseconds
137
-
promotionStrategy?: 'eager' | 'lazy'; // When to promote to upper tiers (default: 'lazy')
138
-
serialization?: { // Custom serialization (default: JSON)
139
-
serialize: (data: unknown) => Promise<Uint8Array>;
140
-
deserialize: (data: Uint8Array) => Promise<unknown>;
147
-
**`get(key: string): Promise<T | null>`**
149
-
Retrieve data for a key. Returns null if not found or expired.
151
-
**`getWithMetadata(key: string): Promise<StorageResult<T> | null>`**
153
-
Retrieve data with metadata and source tier information.
156
-
const result = await storage.getWithMetadata('user:123');
157
-
console.log(result.data); // The actual data
158
-
console.log(result.source); // 'hot' | 'warm' | 'cold'
159
-
console.log(result.metadata); // Metadata (size, timestamps, TTL, etc.)
162
-
**`set(key: string, data: T, options?: SetOptions): Promise<SetResult>`**
164
-
Store data with optional configuration.
167
-
await storage.set('key', data, {
168
-
ttl: 24 * 60 * 60 * 1000, // Custom TTL (24 hours)
169
-
metadata: { contentType: 'application/json' }, // Custom metadata
170
-
skipTiers: ['hot'], // Skip specific tiers
51
+
┌─────────────────────────────────────────────┐
52
+
│ Cold (S3) - source of truth, all data │
54
+
│ Warm (disk) - everything hot has + more │
56
+
│ Hot (memory) - just the hottest stuff │
57
+
└─────────────────────────────────────────────┘
174
-
**`delete(key: string): Promise<void>`**
60
+
Writes cascade **down**. Reads bubble **up**.
176
-
Delete data from all tiers.
178
-
**`exists(key: string): Promise<boolean>`**
64
+
### `storage.get(key)`
180
-
Check if a key exists (and hasn't expired).
66
+
Get data. Returns `null` if missing or expired.
182
-
**`touch(key: string, ttlMs?: number): Promise<void>`**
184
-
Renew TTL for a key. Useful for "keep alive" behavior.
186
-
**`invalidate(prefix: string): Promise<number>`**
188
-
Delete all keys matching a prefix. Returns number of keys deleted.
68
+
### `storage.getWithMetadata(key)`
191
-
await storage.invalidate('user:'); // Delete all user keys
192
-
await storage.invalidate('site:abc/'); // Delete all files for site 'abc'
193
-
await storage.invalidate(''); // Delete everything
70
+
Get data plus which tier served it.
196
-
**`listKeys(prefix?: string): AsyncIterableIterator<string>`**
72
+
### `storage.set(key, data, options?)`
198
-
List all keys, optionally filtered by prefix.
74
+
Store data. Options:
201
-
for await (const key of storage.listKeys('user:')) {
202
-
console.log(key); // 'user:123', 'user:456', etc.
78
+
ttl: 86400000, // custom TTL
79
+
skipTiers: ['hot'], // skip specific tiers
80
+
metadata: { ... }, // custom metadata
206
-
**`getStats(): Promise<AllTierStats>`**
84
+
### `storage.delete(key)`
208
-
Get aggregated statistics across all tiers.
86
+
Delete from all tiers.
211
-
const stats = await storage.getStats();
212
-
console.log(stats.hot); // Hot tier stats (size, items, hits, misses)
213
-
console.log(stats.hitRate); // Overall hit rate (0-1)
88
+
### `storage.invalidate(prefix)`
216
-
**`bootstrapHot(limit?: number): Promise<number>`**
218
-
Load most frequently accessed items from warm into hot. Returns number of items loaded.
221
-
// On server startup: warm up hot tier
222
-
const loaded = await storage.bootstrapHot(1000); // Load top 1000 items
223
-
console.log(`Loaded ${loaded} items into hot tier`);
226
-
**`bootstrapWarm(options?: { limit?: number; sinceDate?: Date }): Promise<number>`**
228
-
Load recent items from cold into warm. Returns number of items loaded.
90
+
Delete all keys matching prefix. Returns count.
231
-
// Load items accessed in last 7 days
232
-
const loaded = await storage.bootstrapWarm({
233
-
sinceDate: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000),
92
+
### `storage.touch(key, ttl?)`
238
-
**`export(): Promise<StorageSnapshot>`**
240
-
Export metadata snapshot for backup or migration.
96
+
### `storage.listKeys(prefix?)`
242
-
**`import(snapshot: StorageSnapshot): Promise<void>`**
98
+
Async iterator over keys.
244
-
Import metadata snapshot.
100
+
### `storage.getStats()`
246
-
**`clear(): Promise<void>`**
102
+
Stats across all tiers.
248
-
Clear all data from all tiers. ⚠️ Use with extreme caution!
104
+
### `storage.bootstrapHot(limit?)`
250
-
**`clearTier(tier: 'hot' | 'warm' | 'cold'): Promise<void>`**
106
+
Warm up hot tier from warm tier. Run on startup.
252
-
Clear a specific tier.
108
+
### `storage.bootstrapWarm(options?)`
254
-
### Built-in Storage Tiers
110
+
Warm up warm tier from cold tier.
256
-
#### `MemoryStorageTier`
258
-
In-memory storage using TinyLRU for efficient LRU eviction.
114
+
### MemoryStorageTier
261
-
import { MemoryStorageTier } from 'tiered-storage';
263
-
const tier = new MemoryStorageTier({
264
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
265
-
maxItems: 1000, // Optional: max number of items
117
+
new MemoryStorageTier({
118
+
maxSizeBytes: 100 * 1024 * 1024,
270
-
- Battle-tested TinyLRU library
271
-
- Automatic LRU eviction
272
-
- Size-based and count-based limits
273
-
- Single process only (not distributed)
123
+
LRU eviction. Fast. Single process only.
275
-
#### `DiskStorageTier`
277
-
Filesystem-based storage with `.meta` files.
125
+
### DiskStorageTier
280
-
import { DiskStorageTier } from 'tiered-storage';
282
-
const tier = new DiskStorageTier({
128
+
new DiskStorageTier({
284
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB (optional)
285
-
evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
130
+
maxSizeBytes: 10 * 1024 * 1024 * 1024,
131
+
evictionPolicy: 'lru', // or 'fifo', 'size'
290
-
- Human-readable file structure
291
-
- Optional size-based eviction
292
-
- Three eviction policies: LRU, FIFO, size-based
293
-
- Atomic writes with `.meta` files
294
-
- Zero external dependencies
135
+
Files on disk with `.meta` sidecars.
296
-
**File structure:**
299
-
├── user%3A123 # Data file (encoded key)
300
-
├── user%3A123.meta # Metadata JSON
301
-
├── site%3Aabc%2Findex.html
302
-
└── site%3Aabc%2Findex.html.meta
305
-
#### `S3StorageTier`
307
-
AWS S3 or S3-compatible object storage.
310
-
import { S3StorageTier } from 'tiered-storage';
312
-
// AWS S3 with separate metadata bucket (RECOMMENDED!)
313
-
const tier = new S3StorageTier({
314
-
bucket: 'my-data-bucket',
315
-
metadataBucket: 'my-metadata-bucket', // Stores metadata separately for fast updates
140
+
new S3StorageTier({
142
+
metadataBucket: 'metadata', // recommended!
318
-
accessKeyId: process.env.AWS_ACCESS_KEY_ID!,
319
-
secretAccessKey: process.env.AWS_SECRET_ACCESS_KEY!,
321
-
prefix: 'cache/', // Optional key prefix
324
-
// Cloudflare R2 with metadata bucket
325
-
const r2Tier = new S3StorageTier({
326
-
bucket: 'my-r2-data-bucket',
327
-
metadataBucket: 'my-r2-metadata-bucket',
329
-
endpoint: 'https://account-id.r2.cloudflarestorage.com',
331
-
accessKeyId: process.env.R2_ACCESS_KEY_ID!,
332
-
secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
336
-
// Without metadata bucket (legacy mode - slower, more expensive)
337
-
const legacyTier = new S3StorageTier({
338
-
bucket: 'my-bucket',
339
-
region: 'us-east-1',
340
-
// No metadataBucket - metadata stored in S3 object metadata fields
345
-
- Compatible with AWS S3, Cloudflare R2, MinIO, and other S3-compatible services
346
-
- **Separate metadata bucket support (RECOMMENDED)** - stores metadata as JSON objects for fast, cheap updates
347
-
- Legacy mode: metadata in S3 object metadata fields (requires object copying for updates)
348
-
- Efficient batch deletions (up to 1000 keys per request)
349
-
- Optional key prefixing for multi-tenant scenarios
350
-
- Typically used as cold tier (source of truth)
147
+
Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.
352
-
**⚠️ Important:** Without `metadataBucket`, updating metadata (e.g., access counts) requires copying the entire object, which is slow and expensive for large files. Use a separate metadata bucket in production!
356
-
### Pattern 1: Simple Single-Server Setup
151
+
Implement `StorageTier`:
359
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
361
-
const storage = new TieredStorage({
363
-
hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
364
-
warm: new DiskStorageTier({ directory: './cache' }),
365
-
cold: new DiskStorageTier({ directory: './storage' }),
368
-
defaultTTL: 14 * 24 * 60 * 60 * 1000, // 14 days
371
-
await storage.set('user:123', { name: 'Alice', email: 'alice@example.com' });
372
-
const user = await storage.get('user:123');
375
-
### Pattern 2: Static Site Hosting (wisp.place-style)
378
-
import { TieredStorage, MemoryStorageTier, DiskStorageTier } from 'tiered-storage';
380
-
const storage = new TieredStorage({
382
-
hot: new MemoryStorageTier({
383
-
maxSizeBytes: 100 * 1024 * 1024, // 100MB
386
-
warm: new DiskStorageTier({
387
-
directory: './cache/sites',
388
-
maxSizeBytes: 10 * 1024 * 1024 * 1024, // 10GB
390
-
// Cold tier is PDS (fetched on demand via custom tier implementation)
393
-
defaultTTL: 14 * 24 * 60 * 60 * 1000,
394
-
promotionStrategy: 'lazy', // Don't auto-promote large files to hot
397
-
// Store index.html in all tiers (fast access)
398
-
await storage.set(`${did}/${rkey}/index.html`, htmlBuffer, {
399
-
metadata: { mimeType: 'text/html', encoding: 'gzip' },
402
-
// Store large files only in warm + cold (skip hot)
403
-
await storage.set(`${did}/${rkey}/video.mp4`, videoBuffer, {
404
-
skipTiers: ['hot'],
405
-
metadata: { mimeType: 'video/mp4' },
408
-
// Get file with source tracking
409
-
const result = await storage.getWithMetadata(`${did}/${rkey}/index.html`);
410
-
console.log(`Served from ${result.source}`); // Likely 'hot' for index.html
412
-
// Invalidate entire site
413
-
await storage.invalidate(`${did}/${rkey}/`);
415
-
// Renew TTL when site is accessed
416
-
await storage.touch(`${did}/${rkey}/index.html`);
154
+
interface StorageTier {
155
+
get(key: string): Promise<Uint8Array | null>
156
+
set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void>
157
+
delete(key: string): Promise<void>
158
+
exists(key: string): Promise<boolean>
159
+
listKeys(prefix?: string): AsyncIterableIterator<string>
160
+
deleteMany(keys: string[]): Promise<void>
161
+
getMetadata(key: string): Promise<StorageMetadata | null>
162
+
setMetadata(key: string, metadata: StorageMetadata): Promise<void>
163
+
getStats(): Promise<TierStats>
164
+
clear(): Promise<void>
419
-
### Pattern 3: Custom Backend (SQLite)
421
-
Implement the `StorageTier` interface to use any backend:
170
+
Don't want big videos in memory? Skip hot:
424
-
import { StorageTier, StorageMetadata, TierStats } from 'tiered-storage';
425
-
import Database from 'better-sqlite3';
427
-
class SQLiteStorageTier implements StorageTier {
428
-
private db: Database.Database;
430
-
constructor(dbPath: string) {
431
-
this.db = new Database(dbPath);
433
-
CREATE TABLE IF NOT EXISTS cache (
434
-
key TEXT PRIMARY KEY,
435
-
data BLOB NOT NULL,
436
-
metadata TEXT NOT NULL
441
-
async get(key: string): Promise<Uint8Array | null> {
442
-
const row = this.db.prepare('SELECT data FROM cache WHERE key = ?').get(key);
443
-
return row ? new Uint8Array(row.data) : null;
446
-
async set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void> {
447
-
this.db.prepare('INSERT OR REPLACE INTO cache (key, data, metadata) VALUES (?, ?, ?)')
448
-
.run(key, Buffer.from(data), JSON.stringify(metadata));
451
-
async delete(key: string): Promise<void> {
452
-
this.db.prepare('DELETE FROM cache WHERE key = ?').run(key);
455
-
async exists(key: string): Promise<boolean> {
456
-
const row = this.db.prepare('SELECT 1 FROM cache WHERE key = ?').get(key);
460
-
async *listKeys(prefix?: string): AsyncIterableIterator<string> {
461
-
const query = prefix
462
-
? this.db.prepare('SELECT key FROM cache WHERE key LIKE ?')
463
-
: this.db.prepare('SELECT key FROM cache');
465
-
const rows = prefix ? query.all(`${prefix}%`) : query.all();
467
-
for (const row of rows) {
472
-
async deleteMany(keys: string[]): Promise<void> {
473
-
const placeholders = keys.map(() => '?').join(',');
474
-
this.db.prepare(`DELETE FROM cache WHERE key IN (${placeholders})`).run(...keys);
477
-
async getMetadata(key: string): Promise<StorageMetadata | null> {
478
-
const row = this.db.prepare('SELECT metadata FROM cache WHERE key = ?').get(key);
479
-
return row ? JSON.parse(row.metadata) : null;
482
-
async setMetadata(key: string, metadata: StorageMetadata): Promise<void> {
483
-
this.db.prepare('UPDATE cache SET metadata = ? WHERE key = ?')
484
-
.run(JSON.stringify(metadata), key);
487
-
async getStats(): Promise<TierStats> {
488
-
const row = this.db.prepare('SELECT COUNT(*) as count, SUM(LENGTH(data)) as bytes FROM cache').get();
489
-
return { items: row.count, bytes: row.bytes || 0 };
492
-
async clear(): Promise<void> {
493
-
this.db.prepare('DELETE FROM cache').run();
498
-
const storage = new TieredStorage({
500
-
warm: new SQLiteStorageTier('./cache.db'),
501
-
cold: new DiskStorageTier({ directory: './storage' }),
173
+
await storage.set('video.mp4', data, { skipTiers: ['hot'] })
506
-
## Running Examples
508
-
### Interactive Demo Server
510
-
Run a **real HTTP server** that serves the example site using tiered storage:
176
+
## Running the demo
513
-
# Configure S3 credentials first (copy .env.example to .env and fill in)
514
-
cp .env.example .env
516
-
# Start the demo server
179
+
cp .env.example .env # add S3 creds
521
-
- **http://localhost:3000/** - The demo site served from tiered storage
522
-
- **http://localhost:3000/admin/stats** - Live cache statistics dashboard
524
-
Watch the console to see which tier serves each request:
525
-
- 🔥 **Hot tier (memory)** - index.html served instantly
526
-
- 💾 **Warm tier (disk)** - Other pages served from disk cache
527
-
- ☁️ **Cold tier (S3)** - First access fetches from S3, then cached
529
-
### Command-Line Examples
531
-
Or run the non-interactive examples:
537
-
The examples include:
538
-
- **Basic CRUD operations** with statistics tracking
539
-
- **Static site hosting** using the real site in `example-site/` directory
540
-
- **Bootstrap demonstrations** (warming caches from lower tiers)
541
-
- **Promotion strategy comparisons** (eager vs lazy)
543
-
The `example-site/` directory contains a complete static website with:
544
-
- `index.html` - Stored in hot + warm + cold (instant serving)
545
-
- `about.html`, `docs.html` - Stored in warm + cold (skips hot)
546
-
- `style.css`, `script.js` - Stored in warm + cold (skips hot)
548
-
This demonstrates the exact pattern you'd use for wisp.place: critical files in memory, everything else on disk/S3.
559
-
# Install dependencies
183
+
Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.
nekomimi.pet