wip library to store cold objects in s3, warm objects on disk, and hot objects in memory
nekomimi.pet
nodejs
typescript
1# tiered-storage
2
3Cascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.
4
5## Features
6
7- **Cascading writes** - data flows down through all tiers
8- **Bubbling reads** - check hot first, fall back to warm, then cold
9- **Pluggable backends** - memory, disk, S3, or implement your own
10- **Selective placement** - skip tiers for big files that don't need memory caching
11- **Prefix invalidation** - `invalidate('user:')` nukes all user keys
12- **Optional compression** - transparent gzip
13
14## Install
15
16```bash
17npm install tiered-storage
18```
19
20## Example
21
22```typescript
23import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'
24
25const storage = new TieredStorage({
26 tiers: {
27 hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
28 warm: new DiskStorageTier({ directory: './cache' }),
29 cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
30 },
31 placementRules: [
32 { pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
33 { pattern: '**/*.{jpg,png,gif,mp4}', tiers: ['warm', 'cold'] },
34 { pattern: '**', tiers: ['warm', 'cold'] },
35 ],
36})
37
38// just set - rules decide where it goes
39await storage.set('site:abc/index.html', indexHtml) // → hot + warm + cold
40await storage.set('site:abc/hero.png', imageData) // → warm + cold
41await storage.set('site:abc/video.mp4', videoData) // → warm + cold
42
43// reads bubble up from wherever it lives
44const page = await storage.getWithMetadata('site:abc/index.html')
45console.log(page.source) // 'hot'
46
47const video = await storage.getWithMetadata('site:abc/video.mp4')
48console.log(video.source) // 'warm'
49
50// nuke entire site
51await storage.invalidate('site:abc/')
52```
53
54Hot tier stays small and fast. Warm tier has everything. Cold tier is the source of truth.
55
56## How it works
57
58```
59┌─────────────────────────────────────────────┐
60│ Cold (S3) - source of truth, all data │
61│ ↑ │
62│ Warm (disk) - everything hot has + more │
63│ ↑ │
64│ Hot (memory) - just the hottest stuff │
65└─────────────────────────────────────────────┘
66```
67
68Writes cascade **down**. Reads bubble **up**.
69
70## Eviction
71
72Items leave upper tiers through eviction or TTL expiration:
73
74```typescript
75const storage = new TieredStorage({
76 tiers: {
77 // hot: LRU eviction when size/count limits hit
78 hot: new MemoryStorageTier({
79 maxSizeBytes: 100 * 1024 * 1024,
80 maxItems: 500,
81 }),
82
83 // warm: evicts when maxSizeBytes hit, policy controls which items go
84 warm: new DiskStorageTier({
85 directory: './cache',
86 maxSizeBytes: 10 * 1024 * 1024 * 1024,
87 evictionPolicy: 'lru', // 'lru' | 'fifo' | 'size'
88 }),
89
90 // cold: never evicts, keeps everything
91 cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
92 },
93 defaultTTL: 14 * 24 * 60 * 60 * 1000, // TTL checked on read
94})
95```
96
97A file that hasn't been accessed eventually gets evicted from hot (LRU), then warm (size limit + policy). Next request fetches from cold and promotes it back up.
98
99## Placement rules
100
101Define once which keys go where, instead of passing `skipTiers` on every `set()`:
102
103```typescript
104const storage = new TieredStorage({
105 tiers: {
106 hot: new MemoryStorageTier({ maxSizeBytes: 50 * 1024 * 1024 }),
107 warm: new DiskStorageTier({ directory: './cache' }),
108 cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
109 },
110 placementRules: [
111 // index.html goes everywhere for instant serving
112 { pattern: '**/index.html', tiers: ['hot', 'warm', 'cold'] },
113
114 // images and video skip hot
115 { pattern: '**/*.{jpg,png,gif,webp,mp4}', tiers: ['warm', 'cold'] },
116
117 // assets directory skips hot
118 { pattern: 'assets/**', tiers: ['warm', 'cold'] },
119
120 // everything else: warm + cold only
121 { pattern: '**', tiers: ['warm', 'cold'] },
122 ],
123})
124
125// just call set() - rules handle placement
126await storage.set('site:abc/index.html', html) // → hot + warm + cold
127await storage.set('site:abc/hero.png', image) // → warm + cold
128await storage.set('site:abc/assets/font.woff', font) // → warm + cold
129await storage.set('site:abc/about.html', html) // → warm + cold
130```
131
132Rules are evaluated in order. First match wins. Cold is always included.
133
134## API
135
136### `storage.get(key)`
137
138Get data. Returns `null` if missing or expired.
139
140### `storage.getWithMetadata(key)`
141
142Get data plus which tier served it.
143
144### `storage.set(key, data, options?)`
145
146Store data. Options:
147
148```typescript
149{
150 ttl: 86400000, // custom TTL
151 skipTiers: ['hot'], // skip specific tiers
152 metadata: { ... }, // custom metadata
153}
154```
155
156### `storage.delete(key)`
157
158Delete from all tiers.
159
160### `storage.invalidate(prefix)`
161
162Delete all keys matching prefix. Returns count.
163
164### `storage.touch(key, ttl?)`
165
166Renew TTL.
167
168### `storage.listKeys(prefix?)`
169
170Async iterator over keys.
171
172### `storage.getStats()`
173
174Stats across all tiers.
175
176### `storage.bootstrapHot(limit?)`
177
178Warm up hot tier from warm tier. Run on startup.
179
180### `storage.bootstrapWarm(options?)`
181
182Warm up warm tier from cold tier.
183
184## Built-in tiers
185
186### MemoryStorageTier
187
188```typescript
189new MemoryStorageTier({
190 maxSizeBytes: 100 * 1024 * 1024,
191 maxItems: 1000,
192})
193```
194
195LRU eviction. Fast. Single process only.
196
197### DiskStorageTier
198
199```typescript
200new DiskStorageTier({
201 directory: './cache',
202 maxSizeBytes: 10 * 1024 * 1024 * 1024,
203 evictionPolicy: 'lru', // or 'fifo', 'size'
204})
205```
206
207Files on disk with `.meta` sidecars.
208
209### S3StorageTier
210
211```typescript
212new S3StorageTier({
213 bucket: 'data',
214 metadataBucket: 'metadata', // recommended!
215 region: 'us-east-1',
216})
217```
218
219Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.
220
221## Custom tiers
222
223Implement `StorageTier`:
224
225```typescript
226interface StorageTier {
227 get(key: string): Promise<Uint8Array | null>
228 set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void>
229 delete(key: string): Promise<void>
230 exists(key: string): Promise<boolean>
231 listKeys(prefix?: string): AsyncIterableIterator<string>
232 deleteMany(keys: string[]): Promise<void>
233 getMetadata(key: string): Promise<StorageMetadata | null>
234 setMetadata(key: string, metadata: StorageMetadata): Promise<void>
235 getStats(): Promise<TierStats>
236 clear(): Promise<void>
237
238 // Optional: combine get + getMetadata for better performance
239 getWithMetadata?(key: string): Promise<{ data: Uint8Array; metadata: StorageMetadata } | null>
240}
241```
242
243The optional `getWithMetadata` method returns both data and metadata in a single call. Implement it if your backend can fetch both efficiently (e.g., parallel I/O, single query). Falls back to separate `get()` + `getMetadata()` calls if not implemented.
244
245## Running the demo
246
247```bash
248cp .env.example .env # add S3 creds
249bun run serve
250```
251
252Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.
253
254## License
255
256MIT