wip library to store cold objects in s3, warm objects on disk, and hot objects in memory
nekomimi.pet
nodejs
typescript
1# tiered-storage
2
3Cascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.
4
5## Features
6
7- **Cascading writes** - data flows down through all tiers
8- **Bubbling reads** - check hot first, fall back to warm, then cold
9- **Pluggable backends** - memory, disk, S3, or implement your own
10- **Selective placement** - skip tiers for big files that don't need memory caching
11- **Prefix invalidation** - `invalidate('user:')` nukes all user keys
12- **Optional compression** - transparent gzip
13
14## Install
15
16```bash
17npm install tiered-storage
18```
19
20## Example
21
22```typescript
23import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'
24
25const storage = new TieredStorage({
26 tiers: {
27 hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
28 warm: new DiskStorageTier({ directory: './cache' }),
29 cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
30 },
31 compression: true,
32})
33
34// critical file: keep in memory for instant serving
35await storage.set('site:abc/index.html', indexHtml)
36
37// big files: skip hot, let them live in warm + cold
38await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] })
39await storage.set('site:abc/hero.png', imageData, { skipTiers: ['hot'] })
40
41// on read, bubbles up from wherever it lives
42const result = await storage.getWithMetadata('site:abc/index.html')
43console.log(result.source) // 'hot' - served from memory
44
45const video = await storage.getWithMetadata('site:abc/video.mp4')
46console.log(video.source) // 'warm' - served from disk, never touches memory
47
48// nuke entire site
49await storage.invalidate('site:abc/')
50```
51
52Hot tier stays small and fast. Warm tier has everything. Cold tier is the source of truth.
53
54## How it works
55
56```
57┌─────────────────────────────────────────────┐
58│ Cold (S3) - source of truth, all data │
59│ ↑ │
60│ Warm (disk) - everything hot has + more │
61│ ↑ │
62│ Hot (memory) - just the hottest stuff │
63└─────────────────────────────────────────────┘
64```
65
66Writes cascade **down**. Reads bubble **up**.
67
68## API
69
70### `storage.get(key)`
71
72Get data. Returns `null` if missing or expired.
73
74### `storage.getWithMetadata(key)`
75
76Get data plus which tier served it.
77
78### `storage.set(key, data, options?)`
79
80Store data. Options:
81
82```typescript
83{
84 ttl: 86400000, // custom TTL
85 skipTiers: ['hot'], // skip specific tiers
86 metadata: { ... }, // custom metadata
87}
88```
89
90### `storage.delete(key)`
91
92Delete from all tiers.
93
94### `storage.invalidate(prefix)`
95
96Delete all keys matching prefix. Returns count.
97
98### `storage.touch(key, ttl?)`
99
100Renew TTL.
101
102### `storage.listKeys(prefix?)`
103
104Async iterator over keys.
105
106### `storage.getStats()`
107
108Stats across all tiers.
109
110### `storage.bootstrapHot(limit?)`
111
112Warm up hot tier from warm tier. Run on startup.
113
114### `storage.bootstrapWarm(options?)`
115
116Warm up warm tier from cold tier.
117
118## Built-in tiers
119
120### MemoryStorageTier
121
122```typescript
123new MemoryStorageTier({
124 maxSizeBytes: 100 * 1024 * 1024,
125 maxItems: 1000,
126})
127```
128
129LRU eviction. Fast. Single process only.
130
131### DiskStorageTier
132
133```typescript
134new DiskStorageTier({
135 directory: './cache',
136 maxSizeBytes: 10 * 1024 * 1024 * 1024,
137 evictionPolicy: 'lru', // or 'fifo', 'size'
138})
139```
140
141Files on disk with `.meta` sidecars.
142
143### S3StorageTier
144
145```typescript
146new S3StorageTier({
147 bucket: 'data',
148 metadataBucket: 'metadata', // recommended!
149 region: 'us-east-1',
150})
151```
152
153Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.
154
155## Custom tiers
156
157Implement `StorageTier`:
158
159```typescript
160interface StorageTier {
161 get(key: string): Promise<Uint8Array | null>
162 set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void>
163 delete(key: string): Promise<void>
164 exists(key: string): Promise<boolean>
165 listKeys(prefix?: string): AsyncIterableIterator<string>
166 deleteMany(keys: string[]): Promise<void>
167 getMetadata(key: string): Promise<StorageMetadata | null>
168 setMetadata(key: string, metadata: StorageMetadata): Promise<void>
169 getStats(): Promise<TierStats>
170 clear(): Promise<void>
171}
172```
173
174## Running the demo
175
176```bash
177cp .env.example .env # add S3 creds
178bun run serve
179```
180
181Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.
182
183## License
184
185MIT