# tiered-storage

Cascading cache that flows hot → warm → cold. Memory, disk, S3—or bring your own.

## Features

- **Cascading writes** - data flows down through all tiers
- **Bubbling reads** - check hot first, fall back to warm, then cold
- **Pluggable backends** - memory, disk, S3, or implement your own
- **Selective placement** - skip tiers for big files that don't need memory caching
- **Prefix invalidation** - `invalidate('user:')` nukes all user keys
- **Optional compression** - transparent gzip

## Install

```bash
npm install tiered-storage
```

## Example

```typescript
import { TieredStorage, MemoryStorageTier, DiskStorageTier, S3StorageTier } from 'tiered-storage'

const storage = new TieredStorage({
  tiers: {
    hot: new MemoryStorageTier({ maxSizeBytes: 100 * 1024 * 1024 }),
    warm: new DiskStorageTier({ directory: './cache' }),
    cold: new S3StorageTier({ bucket: 'my-bucket', region: 'us-east-1' }),
  },
  compression: true,
})

// critical file: keep in memory for instant serving
await storage.set('site:abc/index.html', indexHtml)

// big files: skip hot, let them live in warm + cold
await storage.set('site:abc/video.mp4', videoData, { skipTiers: ['hot'] })
await storage.set('site:abc/hero.png', imageData, { skipTiers: ['hot'] })

// on read, bubbles up from wherever it lives
const result = await storage.getWithMetadata('site:abc/index.html')
console.log(result.source) // 'hot' - served from memory

const video = await storage.getWithMetadata('site:abc/video.mp4')
console.log(video.source) // 'warm' - served from disk, never touches memory

// nuke entire site
await storage.invalidate('site:abc/')
```

Hot tier stays small and fast. Warm tier has everything. Cold tier is the source of truth.

## How it works

```
┌─────────────────────────────────────────────┐
│  Cold (S3)     - source of truth, all data  │
│  ↑                                          │
│  Warm (disk)   - everything hot has + more  │
│  ↑                                          │
│  Hot (memory)  - just the hottest stuff     │
└─────────────────────────────────────────────┘
```

Writes cascade **down**. Reads bubble **up**.

## API

### `storage.get(key)`

Get data. Returns `null` if missing or expired.

### `storage.getWithMetadata(key)`

Get data plus which tier served it.

### `storage.set(key, data, options?)`

Store data. Options:

```typescript
{
  ttl: 86400000,           // custom TTL
  skipTiers: ['hot'],      // skip specific tiers
  metadata: { ... },       // custom metadata
}
```

### `storage.delete(key)`

Delete from all tiers.

### `storage.invalidate(prefix)`

Delete all keys matching prefix. Returns count.

### `storage.touch(key, ttl?)`

Renew TTL.

### `storage.listKeys(prefix?)`

Async iterator over keys.

### `storage.getStats()`

Stats across all tiers.

### `storage.bootstrapHot(limit?)`

Warm up hot tier from warm tier. Run on startup.

### `storage.bootstrapWarm(options?)`

Warm up warm tier from cold tier.

## Built-in tiers

### MemoryStorageTier

```typescript
new MemoryStorageTier({
  maxSizeBytes: 100 * 1024 * 1024,
  maxItems: 1000,
})
```

LRU eviction. Fast. Single process only.

### DiskStorageTier

```typescript
new DiskStorageTier({
  directory: './cache',
  maxSizeBytes: 10 * 1024 * 1024 * 1024,
  evictionPolicy: 'lru', // or 'fifo', 'size'
})
```

Files on disk with `.meta` sidecars.

### S3StorageTier

```typescript
new S3StorageTier({
  bucket: 'data',
  metadataBucket: 'metadata',  // recommended!
  region: 'us-east-1',
})
```

Works with AWS S3, Cloudflare R2, MinIO. Use a separate metadata bucket—otherwise updating access counts requires copying entire objects.

## Custom tiers

Implement `StorageTier`:

```typescript
interface StorageTier {
  get(key: string): Promise<Uint8Array | null>
  set(key: string, data: Uint8Array, metadata: StorageMetadata): Promise<void>
  delete(key: string): Promise<void>
  exists(key: string): Promise<boolean>
  listKeys(prefix?: string): AsyncIterableIterator<string>
  deleteMany(keys: string[]): Promise<void>
  getMetadata(key: string): Promise<StorageMetadata | null>
  setMetadata(key: string, metadata: StorageMetadata): Promise<void>
  getStats(): Promise<TierStats>
  clear(): Promise<void>
}
```

## Running the demo

```bash
cp .env.example .env  # add S3 creds
bun run serve
```

Visit http://localhost:3000 to see it work. Check http://localhost:3000/admin/stats for live cache stats.

## License

MIT