patch of rudimentry _redirects support, incremental uploading for cli · round #2 · pull #3 · nekomimi.pet/wisp.place-monorepo

+31 -1

README.md

···

       50
       50
        
       cargo build

     

       51
       51
        
       ```

     

       52
       52
        
       

     

       53
       53
       +
       ## Features

     

       54
       54
       +
       

     

       55
       55
       +
       ### URL Redirects and Rewrites

     

       56
       56
       +
       

     

       57
       57
       +
       The hosting service supports Netlify-style `_redirects` files for managing URLs. Place a `_redirects` file in your site root to enable:

     

       58
       58
       +
       

     

       59
       59
       +
       - **301/302 Redirects**: Permanent and temporary URL redirects

     

       60
       60
       +
       - **200 Rewrites**: Serve different content without changing the URL

     

       61
       61
       +
       - **404 Custom Pages**: Custom error pages for specific paths

     

       62
       62
       +
       - **Splats & Placeholders**: Dynamic path matching (`/blog/:year/:month/:day`, `/news/*`)

     

       63
       63
       +
       - **Query Parameter Matching**: Redirect based on URL parameters

     

       64
       64
       +
       - **Conditional Redirects**: Route by country, language, or cookie presence

     

       65
       65
       +
       - **Force Redirects**: Override existing files with redirects

     

       66
       66
       +
       

     

       67
       67
       +
       Example `_redirects`:

     

       68
       68
       +
       ```

     

       69
       69
       +
       # Single-page app routing (React, Vue, etc.)

     

       70
       70
       +
       /*                 /index.html           200

     

       71
       71
       +
       

     

       72
       72
       +
       # Simple redirects

     

       73
       73
       +
       /home              /

     

       74
       74
       +
       /old-blog/*        /blog/:splat

     

       75
       75
       +
       

     

       76
       76
       +
       # API proxy

     

       77
       77
       +
       /api/*             https://api.example.com/:splat     200

     

       78
       78
       +
       

     

       79
       79
       +
       # Country-based routing

     

       80
       80
       +
       /                  /us/                  302  Country=us

     

       81
       81
       +
       /                  /uk/                  302  Country=gb

     

       82
       82
       +
       ```

     

       83
       83
       +
       

     

       53
       84
        
       ## Limits

     

       54
       85
        
       

     

       55
       86
        
       - Max file size: 100MB (PDS limit)

     

       56
       56
       -
       - Max site size: 300MB

     

       57
       87
        
       - Max files: 2000

     

       58
       88
        
       

     

       59
       89
        
       ## Tech Stack

-123

hosting-service/EXAMPLE.md

···

       1
       1
       -
       # HTML Path Rewriting Example

     

       2
       2
       -
       

     

       3
       3
       -
       This document demonstrates how HTML path rewriting works when serving sites via the `/s/:identifier/:site/*` route.

     

       4
       4
       -
       

     

       5
       5
       -
       ## Problem

     

       6
       6
       -
       

     

       7
       7
       -
       When you create a static site with absolute paths like `/style.css` or `/images/logo.png`, these paths work fine when served from the root domain. However, when served from a subdirectory like `/s/alice.bsky.social/mysite/`, these absolute paths break because they resolve to the server root instead of the site root.

     

       8
       8
       -
       

     

       9
       9
       -
       ## Solution

     

       10
       10
       -
       

     

       11
       11
       -
       The hosting service automatically rewrites absolute paths in HTML files to work correctly in the subdirectory context.

     

       12
       12
       -
       

     

       13
       13
       -
       ## Example

     

       14
       14
       -
       

     

       15
       15
       -
       **Original HTML file (index.html):**

     

       16
       16
       -
       ```html

     

       17
       17
       -
       <!DOCTYPE html>

     

       18
       18
       -
       <html>

     

       19
       19
       -
       <head>

     

       20
       20
       -
         <meta charset="UTF-8">

     

       21
       21
       -
         <title>My Site</title>

     

       22
       22
       -
         <link rel="stylesheet" href="/style.css">

     

       23
       23
       -
         <link rel="icon" href="/favicon.ico">

     

       24
       24
       -
         <script src="/app.js"></script>

     

       25
       25
       -
       </head>

     

       26
       26
       -
       <body>

     

       27
       27
       -
         <header>

     

       28
       28
       -
           <img src="/images/logo.png" alt="Logo">

     

       29
       29
       -
           <nav>

     

       30
       30
       -
             <a href="/">Home</a>

     

       31
       31
       -
             <a href="/about">About</a>

     

       32
       32
       -
             <a href="/contact">Contact</a>

     

       33
       33
       -
           </nav>

     

       34
       34
       -
         </header>

     

       35
       35
       -
       

     

       36
       36
       -
         <main>

     

       37
       37
       -
           <h1>Welcome</h1>

     

       38
       38
       -
           <img src="/images/hero.jpg"

     

       39
       39
       -
                srcset="/images/hero.jpg 1x, /images/hero@2x.jpg 2x"

     

       40
       40
       -
                alt="Hero">

     

       41
       41
       -
       

     

       42
       42
       -
           <form action="/submit" method="post">

     

       43
       43
       -
             <input type="text" name="email">

     

       44
       44
       -
             <button>Submit</button>

     

       45
       45
       -
           </form>

     

       46
       46
       -
         </main>

     

       47
       47
       -
       

     

       48
       48
       -
         <footer>

     

       49
       49
       -
           <a href="https://example.com">External Link</a>

     

       50
       50
       -
           <a href="#top">Back to Top</a>

     

       51
       51
       -
         </footer>

     

       52
       52
       -
       </body>

     

       53
       53
       -
       </html>

     

       54
       54
       -
       ```

     

       55
       55
       -
       

     

       56
       56
       -
       **When accessed via `/s/alice.bsky.social/mysite/`, the HTML is rewritten to:**

     

       57
       57
       -
       ```html

     

       58
       58
       -
       <!DOCTYPE html>

     

       59
       59
       -
       <html>

     

       60
       60
       -
       <head>

     

       61
       61
       -
         <meta charset="UTF-8">

     

       62
       62
       -
         <title>My Site</title>

     

       63
       63
       -
         <link rel="stylesheet" href="/s/alice.bsky.social/mysite/style.css">

     

       64
       64
       -
         <link rel="icon" href="/s/alice.bsky.social/mysite/favicon.ico">

     

       65
       65
       -
         <script src="/s/alice.bsky.social/mysite/app.js"></script>

     

       66
       66
       -
       </head>

     

       67
       67
       -
       <body>

     

       68
       68
       -
         <header>

     

       69
       69
       -
           <img src="/s/alice.bsky.social/mysite/images/logo.png" alt="Logo">

     

       70
       70
       -
           <nav>

     

       71
       71
       -
             <a href="/s/alice.bsky.social/mysite/">Home</a>

     

       72
       72
       -
             <a href="/s/alice.bsky.social/mysite/about">About</a>

     

       73
       73
       -
             <a href="/s/alice.bsky.social/mysite/contact">Contact</a>

     

       74
       74
       -
           </nav>

     

       75
       75
       -
         </header>

     

       76
       76
       -
       

     

       77
       77
       -
         <main>

     

       78
       78
       -
           <h1>Welcome</h1>

     

       79
       79
       -
           <img src="/s/alice.bsky.social/mysite/images/hero.jpg"

     

       80
       80
       -
                srcset="/s/alice.bsky.social/mysite/images/hero.jpg 1x, /s/alice.bsky.social/mysite/images/hero@2x.jpg 2x"

     

       81
       81
       -
                alt="Hero">

     

       82
       82
       -
       

     

       83
       83
       -
           <form action="/s/alice.bsky.social/mysite/submit" method="post">

     

       84
       84
       -
             <input type="text" name="email">

     

       85
       85
       -
             <button>Submit</button>

     

       86
       86
       -
           </form>

     

       87
       87
       -
         </main>

     

       88
       88
       -
       

     

       89
       89
       -
         <footer>

     

       90
       90
       -
           <a href="https://example.com">External Link</a>

     

       91
       91
       -
           <a href="#top">Back to Top</a>

     

       92
       92
       -
         </footer>

     

       93
       93
       -
       </body>

     

       94
       94
       -
       </html>

     

       95
       95
       -
       ```

     

       96
       96
       -
       

     

       97
       97
       -
       ## What's Preserved

     

       98
       98
       -
       

     

       99
       99
       -
       Notice that:

     

       100
       100
       -
       - ✅ Absolute paths are rewritten: `/style.css` → `/s/alice.bsky.social/mysite/style.css`

     

       101
       101
       -
       - ✅ External URLs are preserved: `https://example.com` stays the same

     

       102
       102
       -
       - ✅ Anchors are preserved: `#top` stays the same

     

       103
       103
       -
       - ✅ The rewriting is safe and won't break your site

     

       104
       104
       -
       

     

       105
       105
       -
       ## Supported Attributes

     

       106
       106
       -
       

     

       107
       107
       -
       The rewriter handles these HTML attributes:

     

       108
       108
       -
       - `src` - images, scripts, iframes, videos, audio

     

       109
       109
       -
       - `href` - links, stylesheets

     

       110
       110
       -
       - `action` - forms

     

       111
       111
       -
       - `data` - objects

     

       112
       112
       -
       - `poster` - video posters

     

       113
       113
       -
       - `srcset` - responsive images

     

       114
       114
       -
       

     

       115
       115
       -
       ## Testing Your Site

     

       116
       116
       -
       

     

       117
       117
       -
       To test if your site works with path rewriting:

     

       118
       118
       -
       

     

       119
       119
       -
       1. Upload your site to your PDS as a `place.wisp.fs` record

     

       120
       120
       -
       2. Access it via: `https://hosting.wisp.place/s/YOUR_HANDLE/SITE_NAME/`

     

       121
       121
       -
       3. Check that all resources load correctly

     

       122
       122
       -
       

     

       123
       123
       -
       If you're using relative paths already (like `./style.css` or `../images/logo.png`), they'll work without any rewriting.

+134

hosting-service/example-_redirects

···

       1
       1
       +
       # Example _redirects file for Wisp hosting

     

       2
       2
       +
       # Place this file in the root directory of your site as "_redirects"

     

       3
       3
       +
       # Lines starting with # are comments

     

       4
       4
       +
       

     

       5
       5
       +
       # ===================================

     

       6
       6
       +
       # SIMPLE REDIRECTS

     

       7
       7
       +
       # ===================================

     

       8
       8
       +
       

     

       9
       9
       +
       # Redirect home page

     

       10
       10
       +
       # /home              /

     

       11
       11
       +
       

     

       12
       12
       +
       # Redirect old URLs to new ones

     

       13
       13
       +
       # /old-blog          /blog

     

       14
       14
       +
       # /about-us          /about

     

       15
       15
       +
       

     

       16
       16
       +
       # ===================================

     

       17
       17
       +
       # SPLAT REDIRECTS (WILDCARDS)

     

       18
       18
       +
       # ===================================

     

       19
       19
       +
       

     

       20
       20
       +
       # Redirect entire directories

     

       21
       21
       +
       # /news/*            /blog/:splat

     

       22
       22
       +
       # /old-site/*        /new-site/:splat

     

       23
       23
       +
       

     

       24
       24
       +
       # ===================================

     

       25
       25
       +
       # PLACEHOLDER REDIRECTS

     

       26
       26
       +
       # ===================================

     

       27
       27
       +
       

     

       28
       28
       +
       # Restructure blog URLs

     

       29
       29
       +
       # /blog/:year/:month/:day/:slug    /posts/:year-:month-:day/:slug

     

       30
       30
       +
       

     

       31
       31
       +
       # Capture multiple parameters

     

       32
       32
       +
       # /products/:category/:id          /shop/:category/item/:id

     

       33
       33
       +
       

     

       34
       34
       +
       # ===================================

     

       35
       35
       +
       # STATUS CODES

     

       36
       36
       +
       # ===================================

     

       37
       37
       +
       

     

       38
       38
       +
       # Permanent redirect (301) - default if not specified

     

       39
       39
       +
       # /permanent-move    /new-location    301

     

       40
       40
       +
       

     

       41
       41
       +
       # Temporary redirect (302)

     

       42
       42
       +
       # /temp-redirect     /temp-location   302

     

       43
       43
       +
       

     

       44
       44
       +
       # Rewrite (200) - serves different content, URL stays the same

     

       45
       45
       +
       # /api/*             /functions/:splat    200

     

       46
       46
       +
       

     

       47
       47
       +
       # Custom 404 page

     

       48
       48
       +
       # /shop/*            /shop-closed.html    404

     

       49
       49
       +
       

     

       50
       50
       +
       # ===================================

     

       51
       51
       +
       # FORCE REDIRECTS

     

       52
       52
       +
       # ===================================

     

       53
       53
       +
       

     

       54
       54
       +
       # Force redirect even if file exists (note the ! after status code)

     

       55
       55
       +
       # /override-file     /other-file.html    200!

     

       56
       56
       +
       

     

       57
       57
       +
       # ===================================

     

       58
       58
       +
       # CONDITIONAL REDIRECTS

     

       59
       59
       +
       # ===================================

     

       60
       60
       +
       

     

       61
       61
       +
       # Country-based redirects (ISO 3166-1 alpha-2 codes)

     

       62
       62
       +
       # /                  /us/         302  Country=us

     

       63
       63
       +
       # /                  /uk/         302  Country=gb

     

       64
       64
       +
       # /                  /anz/        302  Country=au,nz

     

       65
       65
       +
       

     

       66
       66
       +
       # Language-based redirects

     

       67
       67
       +
       # /products          /en/products      301  Language=en

     

       68
       68
       +
       # /products          /de/products      301  Language=de

     

       69
       69
       +
       # /products          /fr/products      301  Language=fr

     

       70
       70
       +
       

     

       71
       71
       +
       # Cookie-based redirects (checks if cookie exists)

     

       72
       72
       +
       # /*                 /legacy/:splat    200  Cookie=is_legacy

     

       73
       73
       +
       

     

       74
       74
       +
       # ===================================

     

       75
       75
       +
       # QUERY PARAMETERS

     

       76
       76
       +
       # ===================================

     

       77
       77
       +
       

     

       78
       78
       +
       # Match specific query parameters

     

       79
       79
       +
       # /store id=:id      /blog/:id    301

     

       80
       80
       +
       

     

       81
       81
       +
       # Multiple parameters

     

       82
       82
       +
       # /search q=:query category=:cat    /find/:cat/:query    301

     

       83
       83
       +
       

     

       84
       84
       +
       # ===================================

     

       85
       85
       +
       # DOMAIN-LEVEL REDIRECTS

     

       86
       86
       +
       # ===================================

     

       87
       87
       +
       

     

       88
       88
       +
       # Redirect to different domain (must include protocol)

     

       89
       89
       +
       # /external          https://example.com/path

     

       90
       90
       +
       

     

       91
       91
       +
       # Redirect entire subdomain

     

       92
       92
       +
       # http://blog.example.com/*     https://example.com/blog/:splat    301!

     

       93
       93
       +
       # https://blog.example.com/*    https://example.com/blog/:splat    301!

     

       94
       94
       +
       

     

       95
       95
       +
       # ===================================

     

       96
       96
       +
       # COMMON PATTERNS

     

       97
       97
       +
       # ===================================

     

       98
       98
       +
       

     

       99
       99
       +
       # Remove .html extensions

     

       100
       100
       +
       # /page.html         /page

     

       101
       101
       +
       

     

       102
       102
       +
       # Add trailing slash

     

       103
       103
       +
       # /about             /about/

     

       104
       104
       +
       

     

       105
       105
       +
       # Single-page app fallback (serve index.html for all paths)

     

       106
       106
       +
       # /*                 /index.html      200

     

       107
       107
       +
       

     

       108
       108
       +
       # API proxy

     

       109
       109
       +
       # /api/*             https://api.example.com/:splat    200

     

       110
       110
       +
       

     

       111
       111
       +
       # ===================================

     

       112
       112
       +
       # CUSTOM ERROR PAGES

     

       113
       113
       +
       # ===================================

     

       114
       114
       +
       

     

       115
       115
       +
       # Language-specific 404 pages

     

       116
       116
       +
       # /en/*              /en/404.html     404

     

       117
       117
       +
       # /de/*              /de/404.html     404

     

       118
       118
       +
       

     

       119
       119
       +
       # Section-specific 404 pages

     

       120
       120
       +
       # /shop/*            /shop/not-found.html    404

     

       121
       121
       +
       # /blog/*            /blog/404.html          404

     

       122
       122
       +
       

     

       123
       123
       +
       # ===================================

     

       124
       124
       +
       # NOTES

     

       125
       125
       +
       # ===================================

     

       126
       126
       +
       # 

     

       127
       127
       +
       # - Rules are processed in order (first match wins)

     

       128
       128
       +
       # - More specific rules should come before general ones

     

       129
       129
       +
       # - Splats (*) can only be used at the end of a path

     

       130
       130
       +
       # - Query parameters are automatically preserved for 200, 301, 302

     

       131
       131
       +
       # - Trailing slashes are normalized (/ and no / are treated the same)

     

       132
       132
       +
       # - Default status code is 301 if not specified

     

       133
       133
       +
       #

     

       134
       134
       +

+215

hosting-service/src/lib/redirects.test.ts

···

       1
       1
       +
       import { describe, it, expect } from 'bun:test'

     

       2
       2
       +
       import { parseRedirectsFile, matchRedirectRule } from './redirects';

     

       3
       3
       +
       

     

       4
       4
       +
       describe('parseRedirectsFile', () => {

     

       5
       5
       +
         it('should parse simple redirects', () => {

     

       6
       6
       +
           const content = `

     

       7
       7
       +
       # Comment line

     

       8
       8
       +
       /old-path /new-path

     

       9
       9
       +
       /home / 301

     

       10
       10
       +
       `;

     

       11
       11
       +
           const rules = parseRedirectsFile(content);

     

       12
       12
       +
           expect(rules).toHaveLength(2);

     

       13
       13
       +
           expect(rules[0]).toMatchObject({

     

       14
       14
       +
             from: '/old-path',

     

       15
       15
       +
             to: '/new-path',

     

       16
       16
       +
             status: 301,

     

       17
       17
       +
             force: false,

     

       18
       18
       +
           });

     

       19
       19
       +
           expect(rules[1]).toMatchObject({

     

       20
       20
       +
             from: '/home',

     

       21
       21
       +
             to: '/',

     

       22
       22
       +
             status: 301,

     

       23
       23
       +
             force: false,

     

       24
       24
       +
           });

     

       25
       25
       +
         });

     

       26
       26
       +
       

     

       27
       27
       +
         it('should parse redirects with different status codes', () => {

     

       28
       28
       +
           const content = `

     

       29
       29
       +
       /temp-redirect /target 302

     

       30
       30
       +
       /rewrite /content 200

     

       31
       31
       +
       /not-found /404 404

     

       32
       32
       +
       `;

     

       33
       33
       +
           const rules = parseRedirectsFile(content);

     

       34
       34
       +
           expect(rules).toHaveLength(3);

     

       35
       35
       +
           expect(rules[0]?.status).toBe(302);

     

       36
       36
       +
           expect(rules[1]?.status).toBe(200);

     

       37
       37
       +
           expect(rules[2]?.status).toBe(404);

     

       38
       38
       +
         });

     

       39
       39
       +
       

     

       40
       40
       +
         it('should parse force redirects', () => {

     

       41
       41
       +
           const content = `/force-path /target 301!`;

     

       42
       42
       +
           const rules = parseRedirectsFile(content);

     

       43
       43
       +
           expect(rules[0]?.force).toBe(true);

     

       44
       44
       +
           expect(rules[0]?.status).toBe(301);

     

       45
       45
       +
         });

     

       46
       46
       +
       

     

       47
       47
       +
         it('should parse splat redirects', () => {

     

       48
       48
       +
           const content = `/news/* /blog/:splat`;

     

       49
       49
       +
           const rules = parseRedirectsFile(content);

     

       50
       50
       +
           expect(rules[0]?.from).toBe('/news/*');

     

       51
       51
       +
           expect(rules[0]?.to).toBe('/blog/:splat');

     

       52
       52
       +
         });

     

       53
       53
       +
       

     

       54
       54
       +
         it('should parse placeholder redirects', () => {

     

       55
       55
       +
           const content = `/blog/:year/:month/:day /posts/:year-:month-:day`;

     

       56
       56
       +
           const rules = parseRedirectsFile(content);

     

       57
       57
       +
           expect(rules[0]?.from).toBe('/blog/:year/:month/:day');

     

       58
       58
       +
           expect(rules[0]?.to).toBe('/posts/:year-:month-:day');

     

       59
       59
       +
         });

     

       60
       60
       +
       

     

       61
       61
       +
         it('should parse country-based redirects', () => {

     

       62
       62
       +
           const content = `/ /anz 302 Country=au,nz`;

     

       63
       63
       +
           const rules = parseRedirectsFile(content);

     

       64
       64
       +
           expect(rules[0]?.conditions?.country).toEqual(['au', 'nz']);

     

       65
       65
       +
         });

     

       66
       66
       +
       

     

       67
       67
       +
         it('should parse language-based redirects', () => {

     

       68
       68
       +
           const content = `/products /en/products 301 Language=en`;

     

       69
       69
       +
           const rules = parseRedirectsFile(content);

     

       70
       70
       +
           expect(rules[0]?.conditions?.language).toEqual(['en']);

     

       71
       71
       +
         });

     

       72
       72
       +
       

     

       73
       73
       +
         it('should parse cookie-based redirects', () => {

     

       74
       74
       +
           const content = `/* /legacy/:splat 200 Cookie=is_legacy,my_cookie`;

     

       75
       75
       +
           const rules = parseRedirectsFile(content);

     

       76
       76
       +
           expect(rules[0]?.conditions?.cookie).toEqual(['is_legacy', 'my_cookie']);

     

       77
       77
       +
         });

     

       78
       78
       +
       });

     

       79
       79
       +
       

     

       80
       80
       +
       describe('matchRedirectRule', () => {

     

       81
       81
       +
         it('should match exact paths', () => {

     

       82
       82
       +
           const rules = parseRedirectsFile('/old-path /new-path');

     

       83
       83
       +
           const match = matchRedirectRule('/old-path', rules);

     

       84
       84
       +
           expect(match).toBeTruthy();

     

       85
       85
       +
           expect(match?.targetPath).toBe('/new-path');

     

       86
       86
       +
           expect(match?.status).toBe(301);

     

       87
       87
       +
         });

     

       88
       88
       +
       

     

       89
       89
       +
         it('should match paths with trailing slash', () => {

     

       90
       90
       +
           const rules = parseRedirectsFile('/old-path /new-path');

     

       91
       91
       +
           const match = matchRedirectRule('/old-path/', rules);

     

       92
       92
       +
           expect(match).toBeTruthy();

     

       93
       93
       +
           expect(match?.targetPath).toBe('/new-path');

     

       94
       94
       +
         });

     

       95
       95
       +
       

     

       96
       96
       +
         it('should match splat patterns', () => {

     

       97
       97
       +
           const rules = parseRedirectsFile('/news/* /blog/:splat');

     

       98
       98
       +
           const match = matchRedirectRule('/news/2024/01/15/my-post', rules);

     

       99
       99
       +
           expect(match).toBeTruthy();

     

       100
       100
       +
           expect(match?.targetPath).toBe('/blog/2024/01/15/my-post');

     

       101
       101
       +
         });

     

       102
       102
       +
       

     

       103
       103
       +
         it('should match placeholder patterns', () => {

     

       104
       104
       +
           const rules = parseRedirectsFile('/blog/:year/:month/:day /posts/:year-:month-:day');

     

       105
       105
       +
           const match = matchRedirectRule('/blog/2024/01/15', rules);

     

       106
       106
       +
           expect(match).toBeTruthy();

     

       107
       107
       +
           expect(match?.targetPath).toBe('/posts/2024-01-15');

     

       108
       108
       +
         });

     

       109
       109
       +
       

     

       110
       110
       +
         it('should preserve query strings for 301/302 redirects', () => {

     

       111
       111
       +
           const rules = parseRedirectsFile('/old /new 301');

     

       112
       112
       +
           const match = matchRedirectRule('/old', rules, {

     

       113
       113
       +
             queryParams: { foo: 'bar', baz: 'qux' },

     

       114
       114
       +
           });

     

       115
       115
       +
           expect(match?.targetPath).toContain('?');

     

       116
       116
       +
           expect(match?.targetPath).toContain('foo=bar');

     

       117
       117
       +
           expect(match?.targetPath).toContain('baz=qux');

     

       118
       118
       +
         });

     

       119
       119
       +
       

     

       120
       120
       +
         it('should match based on query parameters', () => {

     

       121
       121
       +
           const rules = parseRedirectsFile('/store id=:id /blog/:id 301');

     

       122
       122
       +
           const match = matchRedirectRule('/store', rules, {

     

       123
       123
       +
             queryParams: { id: 'my-post' },

     

       124
       124
       +
           });

     

       125
       125
       +
           expect(match).toBeTruthy();

     

       126
       126
       +
           expect(match?.targetPath).toContain('/blog/my-post');

     

       127
       127
       +
         });

     

       128
       128
       +
       

     

       129
       129
       +
         it('should not match when query params are missing', () => {

     

       130
       130
       +
           const rules = parseRedirectsFile('/store id=:id /blog/:id 301');

     

       131
       131
       +
           const match = matchRedirectRule('/store', rules, {

     

       132
       132
       +
             queryParams: {},

     

       133
       133
       +
           });

     

       134
       134
       +
           expect(match).toBeNull();

     

       135
       135
       +
         });

     

       136
       136
       +
       

     

       137
       137
       +
         it('should match based on country header', () => {

     

       138
       138
       +
           const rules = parseRedirectsFile('/ /aus 302 Country=au');

     

       139
       139
       +
           const match = matchRedirectRule('/', rules, {

     

       140
       140
       +
             headers: { 'cf-ipcountry': 'AU' },

     

       141
       141
       +
           });

     

       142
       142
       +
           expect(match).toBeTruthy();

     

       143
       143
       +
           expect(match?.targetPath).toBe('/aus');

     

       144
       144
       +
         });

     

       145
       145
       +
       

     

       146
       146
       +
         it('should not match wrong country', () => {

     

       147
       147
       +
           const rules = parseRedirectsFile('/ /aus 302 Country=au');

     

       148
       148
       +
           const match = matchRedirectRule('/', rules, {

     

       149
       149
       +
             headers: { 'cf-ipcountry': 'US' },

     

       150
       150
       +
           });

     

       151
       151
       +
           expect(match).toBeNull();

     

       152
       152
       +
         });

     

       153
       153
       +
       

     

       154
       154
       +
         it('should match based on language header', () => {

     

       155
       155
       +
           const rules = parseRedirectsFile('/products /en/products 301 Language=en');

     

       156
       156
       +
           const match = matchRedirectRule('/products', rules, {

     

       157
       157
       +
             headers: { 'accept-language': 'en-US,en;q=0.9' },

     

       158
       158
       +
           });

     

       159
       159
       +
           expect(match).toBeTruthy();

     

       160
       160
       +
           expect(match?.targetPath).toBe('/en/products');

     

       161
       161
       +
         });

     

       162
       162
       +
       

     

       163
       163
       +
         it('should match based on cookie presence', () => {

     

       164
       164
       +
           const rules = parseRedirectsFile('/* /legacy/:splat 200 Cookie=is_legacy');

     

       165
       165
       +
           const match = matchRedirectRule('/some-path', rules, {

     

       166
       166
       +
             cookies: { is_legacy: 'true' },

     

       167
       167
       +
           });

     

       168
       168
       +
           expect(match).toBeTruthy();

     

       169
       169
       +
           expect(match?.targetPath).toBe('/legacy/some-path');

     

       170
       170
       +
         });

     

       171
       171
       +
       

     

       172
       172
       +
         it('should return first matching rule', () => {

     

       173
       173
       +
           const content = `

     

       174
       174
       +
       /path /first

     

       175
       175
       +
       /path /second

     

       176
       176
       +
       `;

     

       177
       177
       +
           const rules = parseRedirectsFile(content);

     

       178
       178
       +
           const match = matchRedirectRule('/path', rules);

     

       179
       179
       +
           expect(match?.targetPath).toBe('/first');

     

       180
       180
       +
         });

     

       181
       181
       +
       

     

       182
       182
       +
         it('should match more specific rules before general ones', () => {

     

       183
       183
       +
           const content = `

     

       184
       184
       +
       /jobs/customer-ninja /careers/support

     

       185
       185
       +
       /jobs/* /careers/:splat

     

       186
       186
       +
       `;

     

       187
       187
       +
           const rules = parseRedirectsFile(content);

     

       188
       188
       +
           

     

       189
       189
       +
           const match1 = matchRedirectRule('/jobs/customer-ninja', rules);

     

       190
       190
       +
           expect(match1?.targetPath).toBe('/careers/support');

     

       191
       191
       +
           

     

       192
       192
       +
           const match2 = matchRedirectRule('/jobs/developer', rules);

     

       193
       193
       +
           expect(match2?.targetPath).toBe('/careers/developer');

     

       194
       194
       +
         });

     

       195
       195
       +
       

     

       196
       196
       +
         it('should handle SPA routing pattern', () => {

     

       197
       197
       +
           const rules = parseRedirectsFile('/* /index.html 200');

     

       198
       198
       +
           

     

       199
       199
       +
           // Should match any path

     

       200
       200
       +
           const match1 = matchRedirectRule('/about', rules);

     

       201
       201
       +
           expect(match1).toBeTruthy();

     

       202
       202
       +
           expect(match1?.targetPath).toBe('/index.html');

     

       203
       203
       +
           expect(match1?.status).toBe(200);

     

       204
       204
       +
           

     

       205
       205
       +
           const match2 = matchRedirectRule('/users/123/profile', rules);

     

       206
       206
       +
           expect(match2).toBeTruthy();

     

       207
       207
       +
           expect(match2?.targetPath).toBe('/index.html');

     

       208
       208
       +
           expect(match2?.status).toBe(200);

     

       209
       209
       +
           

     

       210
       210
       +
           const match3 = matchRedirectRule('/', rules);

     

       211
       211
       +
           expect(match3).toBeTruthy();

     

       212
       212
       +
           expect(match3?.targetPath).toBe('/index.html');

     

       213
       213
       +
         });

     

       214
       214
       +
       });

     

       215
       215
       +

+413

hosting-service/src/lib/redirects.ts

···

       1
       1
       +
       import { readFile } from 'fs/promises';

     

       2
       2
       +
       import { existsSync } from 'fs';

     

       3
       3
       +
       

     

       4
       4
       +
       export interface RedirectRule {

     

       5
       5
       +
         from: string;

     

       6
       6
       +
         to: string;

     

       7
       7
       +
         status: number;

     

       8
       8
       +
         force: boolean;

     

       9
       9
       +
         conditions?: {

     

       10
       10
       +
           country?: string[];

     

       11
       11
       +
           language?: string[];

     

       12
       12
       +
           role?: string[];

     

       13
       13
       +
           cookie?: string[];

     

       14
       14
       +
         };

     

       15
       15
       +
         // For pattern matching

     

       16
       16
       +
         fromPattern?: RegExp;

     

       17
       17
       +
         fromParams?: string[]; // Named parameters from the pattern

     

       18
       18
       +
         queryParams?: Record<string, string>; // Expected query parameters

     

       19
       19
       +
       }

     

       20
       20
       +
       

     

       21
       21
       +
       export interface RedirectMatch {

     

       22
       22
       +
         rule: RedirectRule;

     

       23
       23
       +
         targetPath: string;

     

       24
       24
       +
         status: number;

     

       25
       25
       +
       }

     

       26
       26
       +
       

     

       27
       27
       +
       /**

     

       28
       28
       +
        * Parse a _redirects file into an array of redirect rules

     

       29
       29
       +
        */

     

       30
       30
       +
       export function parseRedirectsFile(content: string): RedirectRule[] {

     

       31
       31
       +
         const lines = content.split('\n');

     

       32
       32
       +
         const rules: RedirectRule[] = [];

     

       33
       33
       +
       

     

       34
       34
       +
         for (let lineNum = 0; lineNum < lines.length; lineNum++) {

     

       35
       35
       +
           const lineRaw = lines[lineNum];

     

       36
       36
       +
           if (!lineRaw) continue;

     

       37
       37
       +
           

     

       38
       38
       +
           const line = lineRaw.trim();

     

       39
       39
       +
           

     

       40
       40
       +
           // Skip empty lines and comments

     

       41
       41
       +
           if (!line || line.startsWith('#')) {

     

       42
       42
       +
             continue;

     

       43
       43
       +
           }

     

       44
       44
       +
       

     

       45
       45
       +
           try {

     

       46
       46
       +
             const rule = parseRedirectLine(line);

     

       47
       47
       +
             if (rule && rule.fromPattern) {

     

       48
       48
       +
               rules.push(rule);

     

       49
       49
       +
             }

     

       50
       50
       +
           } catch (err) {

     

       51
       51
       +
             console.warn(`Failed to parse redirect rule on line ${lineNum + 1}: ${line}`, err);

     

       52
       52
       +
           }

     

       53
       53
       +
         }

     

       54
       54
       +
       

     

       55
       55
       +
         return rules;

     

       56
       56
       +
       }

     

       57
       57
       +
       

     

       58
       58
       +
       /**

     

       59
       59
       +
        * Parse a single redirect rule line

     

       60
       60
       +
        * Format: /from [query_params] /to [status] [conditions]

     

       61
       61
       +
        */

     

       62
       62
       +
       function parseRedirectLine(line: string): RedirectRule | null {

     

       63
       63
       +
         // Split by whitespace, but respect quoted strings (though not commonly used)

     

       64
       64
       +
         const parts = line.split(/\s+/);

     

       65
       65
       +
         

     

       66
       66
       +
         if (parts.length < 2) {

     

       67
       67
       +
           return null;

     

       68
       68
       +
         }

     

       69
       69
       +
       

     

       70
       70
       +
         let idx = 0;

     

       71
       71
       +
         const from = parts[idx++];

     

       72
       72
       +
         

     

       73
       73
       +
         if (!from) {

     

       74
       74
       +
           return null;

     

       75
       75
       +
         }

     

       76
       76
       +
         

     

       77
       77
       +
         let status = 301; // Default status

     

       78
       78
       +
         let force = false;

     

       79
       79
       +
         const conditions: NonNullable<RedirectRule['conditions']> = {};

     

       80
       80
       +
         const queryParams: Record<string, string> = {};

     

       81
       81
       +
         

     

       82
       82
       +
         // Parse query parameters that come before the destination path

     

       83
       83
       +
         // They look like: key=:value (and don't start with /)

     

       84
       84
       +
         while (idx < parts.length) {

     

       85
       85
       +
           const part = parts[idx];

     

       86
       86
       +
           if (!part) {

     

       87
       87
       +
             idx++;

     

       88
       88
       +
             continue;

     

       89
       89
       +
           }

     

       90
       90
       +
           

     

       91
       91
       +
           // If it starts with / or http, it's the destination path

     

       92
       92
       +
           if (part.startsWith('/') || part.startsWith('http://') || part.startsWith('https://')) {

     

       93
       93
       +
             break;

     

       94
       94
       +
           }

     

       95
       95
       +
           

     

       96
       96
       +
           // If it contains = and comes before the destination, it's a query param

     

       97
       97
       +
           if (part.includes('=')) {

     

       98
       98
       +
             const splitIndex = part.indexOf('=');

     

       99
       99
       +
             const key = part.slice(0, splitIndex);

     

       100
       100
       +
             const value = part.slice(splitIndex + 1);

     

       101
       101
       +
             

     

       102
       102
       +
             if (key && value) {

     

       103
       103
       +
               queryParams[key] = value;

     

       104
       104
       +
             }

     

       105
       105
       +
             idx++;

     

       106
       106
       +
           } else {

     

       107
       107
       +
             // Not a query param, must be destination or something else

     

       108
       108
       +
             break;

     

       109
       109
       +
           }

     

       110
       110
       +
         }

     

       111
       111
       +
         

     

       112
       112
       +
         // Next part should be the destination

     

       113
       113
       +
         if (idx >= parts.length) {

     

       114
       114
       +
           return null;

     

       115
       115
       +
         }

     

       116
       116
       +
         

     

       117
       117
       +
         const to = parts[idx++];

     

       118
       118
       +
         if (!to) {

     

       119
       119
       +
           return null;

     

       120
       120
       +
         }

     

       121
       121
       +
       

     

       122
       122
       +
         // Parse remaining parts for status code and conditions

     

       123
       123
       +
         for (let i = idx; i < parts.length; i++) {

     

       124
       124
       +
           const part = parts[i];

     

       125
       125
       +
           

     

       126
       126
       +
           if (!part) continue;

     

       127
       127
       +
           

     

       128
       128
       +
           // Check for status code (with optional ! for force)

     

       129
       129
       +
           if (/^\d+!?$/.test(part)) {

     

       130
       130
       +
             if (part.endsWith('!')) {

     

       131
       131
       +
               force = true;

     

       132
       132
       +
               status = parseInt(part.slice(0, -1));

     

       133
       133
       +
             } else {

     

       134
       134
       +
               status = parseInt(part);

     

       135
       135
       +
             }

     

       136
       136
       +
             continue;

     

       137
       137
       +
           }

     

       138
       138
       +
       

     

       139
       139
       +
           // Check for condition parameters (Country=, Language=, Role=, Cookie=)

     

       140
       140
       +
           if (part.includes('=')) {

     

       141
       141
       +
             const splitIndex = part.indexOf('=');

     

       142
       142
       +
             const key = part.slice(0, splitIndex);

     

       143
       143
       +
             const value = part.slice(splitIndex + 1);

     

       144
       144
       +
             

     

       145
       145
       +
             if (!key || !value) continue;

     

       146
       146
       +
             

     

       147
       147
       +
             const keyLower = key.toLowerCase();

     

       148
       148
       +
             

     

       149
       149
       +
             if (keyLower === 'country') {

     

       150
       150
       +
               conditions.country = value.split(',').map(v => v.trim().toLowerCase());

     

       151
       151
       +
             } else if (keyLower === 'language') {

     

       152
       152
       +
               conditions.language = value.split(',').map(v => v.trim().toLowerCase());

     

       153
       153
       +
             } else if (keyLower === 'role') {

     

       154
       154
       +
               conditions.role = value.split(',').map(v => v.trim());

     

       155
       155
       +
             } else if (keyLower === 'cookie') {

     

       156
       156
       +
               conditions.cookie = value.split(',').map(v => v.trim().toLowerCase());

     

       157
       157
       +
             }

     

       158
       158
       +
           }

     

       159
       159
       +
         }

     

       160
       160
       +
       

     

       161
       161
       +
         // Parse the 'from' pattern

     

       162
       162
       +
         const { pattern, params } = convertPathToRegex(from);

     

       163
       163
       +
       

     

       164
       164
       +
         return {

     

       165
       165
       +
           from,

     

       166
       166
       +
           to,

     

       167
       167
       +
           status,

     

       168
       168
       +
           force,

     

       169
       169
       +
           conditions: Object.keys(conditions).length > 0 ? conditions : undefined,

     

       170
       170
       +
           queryParams: Object.keys(queryParams).length > 0 ? queryParams : undefined,

     

       171
       171
       +
           fromPattern: pattern,

     

       172
       172
       +
           fromParams: params,

     

       173
       173
       +
         };

     

       174
       174
       +
       }

     

       175
       175
       +
       

     

       176
       176
       +
       /**

     

       177
       177
       +
        * Convert a path pattern with placeholders and splats to a regex

     

       178
       178
       +
        * Examples:

     

       179
       179
       +
        *   /blog/:year/:month/:day -> captures year, month, day

     

       180
       180
       +
        *   /news/* -> captures splat

     

       181
       181
       +
        */

     

       182
       182
       +
       function convertPathToRegex(pattern: string): { pattern: RegExp; params: string[] } {

     

       183
       183
       +
         const params: string[] = [];

     

       184
       184
       +
         let regexStr = '^';

     

       185
       185
       +
         

     

       186
       186
       +
         // Split by query string if present

     

       187
       187
       +
         const pathPart = pattern.split('?')[0] || pattern;

     

       188
       188
       +
         

     

       189
       189
       +
         // Escape special regex characters except * and :

     

       190
       190
       +
         let escaped = pathPart.replace(/[.+^${}()|[\]\\]/g, '\\$&');

     

       191
       191
       +
         

     

       192
       192
       +
         // Replace :param with named capture groups

     

       193
       193
       +
         escaped = escaped.replace(/:([a-zA-Z_][a-zA-Z0-9_]*)/g, (match, paramName) => {

     

       194
       194
       +
           params.push(paramName);

     

       195
       195
       +
           // Match path segment (everything except / and ?)

     

       196
       196
       +
           return '([^/?]+)';

     

       197
       197
       +
         });

     

       198
       198
       +
         

     

       199
       199
       +
         // Replace * with splat capture (matches everything including /)

     

       200
       200
       +
         if (escaped.includes('*')) {

     

       201
       201
       +
           escaped = escaped.replace(/\*/g, '(.*)');

     

       202
       202
       +
           params.push('splat');

     

       203
       203
       +
         }

     

       204
       204
       +
         

     

       205
       205
       +
         regexStr += escaped;

     

       206
       206
       +
         

     

       207
       207
       +
         // Make trailing slash optional

     

       208
       208
       +
         if (!regexStr.endsWith('.*')) {

     

       209
       209
       +
           regexStr += '/?';

     

       210
       210
       +
         }

     

       211
       211
       +
         

     

       212
       212
       +
         regexStr += '$';

     

       213
       213
       +
         

     

       214
       214
       +
         return {

     

       215
       215
       +
           pattern: new RegExp(regexStr),

     

       216
       216
       +
           params,

     

       217
       217
       +
         };

     

       218
       218
       +
       }

     

       219
       219
       +
       

     

       220
       220
       +
       /**

     

       221
       221
       +
        * Match a request path against redirect rules

     

       222
       222
       +
        */

     

       223
       223
       +
       export function matchRedirectRule(

     

       224
       224
       +
         requestPath: string,

     

       225
       225
       +
         rules: RedirectRule[],

     

       226
       226
       +
         context?: {

     

       227
       227
       +
           queryParams?: Record<string, string>;

     

       228
       228
       +
           headers?: Record<string, string>;

     

       229
       229
       +
           cookies?: Record<string, string>;

     

       230
       230
       +
         }

     

       231
       231
       +
       ): RedirectMatch | null {

     

       232
       232
       +
         // Normalize path: ensure leading slash, remove trailing slash (except for root)

     

       233
       233
       +
         let normalizedPath = requestPath.startsWith('/') ? requestPath : `/${requestPath}`;

     

       234
       234
       +
         

     

       235
       235
       +
         for (const rule of rules) {

     

       236
       236
       +
           // Check query parameter conditions first (if any)

     

       237
       237
       +
           if (rule.queryParams) {

     

       238
       238
       +
             // If rule requires query params but none provided, skip this rule

     

       239
       239
       +
             if (!context?.queryParams) {

     

       240
       240
       +
               continue;

     

       241
       241
       +
             }

     

       242
       242
       +
             

     

       243
       243
       +
             const queryMatches = Object.entries(rule.queryParams).every(([key, value]) => {

     

       244
       244
       +
               const actualValue = context.queryParams?.[key];

     

       245
       245
       +
               return actualValue !== undefined;

     

       246
       246
       +
             });

     

       247
       247
       +
             

     

       248
       248
       +
             if (!queryMatches) {

     

       249
       249
       +
               continue;

     

       250
       250
       +
             }

     

       251
       251
       +
           }

     

       252
       252
       +
       

     

       253
       253
       +
           // Check conditional redirects (country, language, role, cookie)

     

       254
       254
       +
           if (rule.conditions) {

     

       255
       255
       +
             if (rule.conditions.country && context?.headers) {

     

       256
       256
       +
               const cfCountry = context.headers['cf-ipcountry'];

     

       257
       257
       +
               const xCountry = context.headers['x-country'];

     

       258
       258
       +
               const country = (cfCountry?.toLowerCase() || xCountry?.toLowerCase());

     

       259
       259
       +
               if (!country || !rule.conditions.country.includes(country)) {

     

       260
       260
       +
                 continue;

     

       261
       261
       +
               }

     

       262
       262
       +
             }

     

       263
       263
       +
       

     

       264
       264
       +
             if (rule.conditions.language && context?.headers) {

     

       265
       265
       +
               const acceptLang = context.headers['accept-language'];

     

       266
       266
       +
               if (!acceptLang) {

     

       267
       267
       +
                 continue;

     

       268
       268
       +
               }

     

       269
       269
       +
               // Parse accept-language header (simplified)

     

       270
       270
       +
               const langs = acceptLang.split(',').map(l => {

     

       271
       271
       +
                 const langPart = l.split(';')[0];

     

       272
       272
       +
                 return langPart ? langPart.trim().toLowerCase() : '';

     

       273
       273
       +
               }).filter(l => l !== '');

     

       274
       274
       +
               const hasMatch = rule.conditions.language.some(lang => 

     

       275
       275
       +
                 langs.some(l => l === lang || l.startsWith(lang + '-'))

     

       276
       276
       +
               );

     

       277
       277
       +
               if (!hasMatch) {

     

       278
       278
       +
                 continue;

     

       279
       279
       +
               }

     

       280
       280
       +
             }

     

       281
       281
       +
       

     

       282
       282
       +
             if (rule.conditions.cookie && context?.cookies) {

     

       283
       283
       +
               const hasCookie = rule.conditions.cookie.some(cookieName => 

     

       284
       284
       +
                 context.cookies && cookieName in context.cookies

     

       285
       285
       +
               );

     

       286
       286
       +
               if (!hasCookie) {

     

       287
       287
       +
                 continue;

     

       288
       288
       +
               }

     

       289
       289
       +
             }

     

       290
       290
       +
       

     

       291
       291
       +
             // Role-based redirects would need JWT verification - skip for now

     

       292
       292
       +
             if (rule.conditions.role) {

     

       293
       293
       +
               continue;

     

       294
       294
       +
             }

     

       295
       295
       +
           }

     

       296
       296
       +
       

     

       297
       297
       +
           // Match the path pattern

     

       298
       298
       +
           const match = rule.fromPattern?.exec(normalizedPath);

     

       299
       299
       +
           if (!match) {

     

       300
       300
       +
             continue;

     

       301
       301
       +
           }

     

       302
       302
       +
       

     

       303
       303
       +
           // Build the target path by replacing placeholders

     

       304
       304
       +
           let targetPath = rule.to;

     

       305
       305
       +
           

     

       306
       306
       +
           // Replace captured parameters

     

       307
       307
       +
           if (rule.fromParams && match.length > 1) {

     

       308
       308
       +
             for (let i = 0; i < rule.fromParams.length; i++) {

     

       309
       309
       +
               const paramName = rule.fromParams[i];

     

       310
       310
       +
               const paramValue = match[i + 1];

     

       311
       311
       +
               

     

       312
       312
       +
               if (!paramName || !paramValue) continue;

     

       313
       313
       +
               

     

       314
       314
       +
               if (paramName === 'splat') {

     

       315
       315
       +
                 targetPath = targetPath.replace(':splat', paramValue);

     

       316
       316
       +
               } else {

     

       317
       317
       +
                 targetPath = targetPath.replace(`:${paramName}`, paramValue);

     

       318
       318
       +
               }

     

       319
       319
       +
             }

     

       320
       320
       +
           }

     

       321
       321
       +
       

     

       322
       322
       +
           // Handle query parameter replacements

     

       323
       323
       +
           if (rule.queryParams && context?.queryParams) {

     

       324
       324
       +
             for (const [key, placeholder] of Object.entries(rule.queryParams)) {

     

       325
       325
       +
               const actualValue = context.queryParams[key];

     

       326
       326
       +
               if (actualValue && placeholder && placeholder.startsWith(':')) {

     

       327
       327
       +
                 const paramName = placeholder.slice(1);

     

       328
       328
       +
                 if (paramName) {

     

       329
       329
       +
                   targetPath = targetPath.replace(`:${paramName}`, actualValue);

     

       330
       330
       +
                 }

     

       331
       331
       +
               }

     

       332
       332
       +
             }

     

       333
       333
       +
           }

     

       334
       334
       +
       

     

       335
       335
       +
           // Preserve query string for 200, 301, 302 redirects (unless target already has one)

     

       336
       336
       +
           if ([200, 301, 302].includes(rule.status) && context?.queryParams && !targetPath.includes('?')) {

     

       337
       337
       +
             const queryString = Object.entries(context.queryParams)

     

       338
       338
       +
               .map(([k, v]) => `${encodeURIComponent(k)}=${encodeURIComponent(v)}`)

     

       339
       339
       +
               .join('&');

     

       340
       340
       +
             if (queryString) {

     

       341
       341
       +
               targetPath += `?${queryString}`;

     

       342
       342
       +
             }

     

       343
       343
       +
           }

     

       344
       344
       +
       

     

       345
       345
       +
           return {

     

       346
       346
       +
             rule,

     

       347
       347
       +
             targetPath,

     

       348
       348
       +
             status: rule.status,

     

       349
       349
       +
           };

     

       350
       350
       +
         }

     

       351
       351
       +
       

     

       352
       352
       +
         return null;

     

       353
       353
       +
       }

     

       354
       354
       +
       

     

       355
       355
       +
       /**

     

       356
       356
       +
        * Load redirect rules from a cached site

     

       357
       357
       +
        */

     

       358
       358
       +
       export async function loadRedirectRules(did: string, rkey: string): Promise<RedirectRule[]> {

     

       359
       359
       +
         const CACHE_DIR = process.env.CACHE_DIR || './cache/sites';

     

       360
       360
       +
         const redirectsPath = `${CACHE_DIR}/${did}/${rkey}/_redirects`;

     

       361
       361
       +
         

     

       362
       362
       +
         if (!existsSync(redirectsPath)) {

     

       363
       363
       +
           return [];

     

       364
       364
       +
         }

     

       365
       365
       +
       

     

       366
       366
       +
         try {

     

       367
       367
       +
           const content = await readFile(redirectsPath, 'utf-8');

     

       368
       368
       +
           return parseRedirectsFile(content);

     

       369
       369
       +
         } catch (err) {

     

       370
       370
       +
           console.error('Failed to load _redirects file', err);

     

       371
       371
       +
           return [];

     

       372
       372
       +
         }

     

       373
       373
       +
       }

     

       374
       374
       +
       

     

       375
       375
       +
       /**

     

       376
       376
       +
        * Parse cookies from Cookie header

     

       377
       377
       +
        */

     

       378
       378
       +
       export function parseCookies(cookieHeader?: string): Record<string, string> {

     

       379
       379
       +
         if (!cookieHeader) return {};

     

       380
       380
       +
         

     

       381
       381
       +
         const cookies: Record<string, string> = {};

     

       382
       382
       +
         const parts = cookieHeader.split(';');

     

       383
       383
       +
         

     

       384
       384
       +
         for (const part of parts) {

     

       385
       385
       +
           const [key, ...valueParts] = part.split('=');

     

       386
       386
       +
           if (key && valueParts.length > 0) {

     

       387
       387
       +
             cookies[key.trim()] = valueParts.join('=').trim();

     

       388
       388
       +
           }

     

       389
       389
       +
         }

     

       390
       390
       +
         

     

       391
       391
       +
         return cookies;

     

       392
       392
       +
       }

     

       393
       393
       +
       

     

       394
       394
       +
       /**

     

       395
       395
       +
        * Parse query string into object

     

       396
       396
       +
        */

     

       397
       397
       +
       export function parseQueryString(url: string): Record<string, string> {

     

       398
       398
       +
         const queryStart = url.indexOf('?');

     

       399
       399
       +
         if (queryStart === -1) return {};

     

       400
       400
       +
         

     

       401
       401
       +
         const queryString = url.slice(queryStart + 1);

     

       402
       402
       +
         const params: Record<string, string> = {};

     

       403
       403
       +
         

     

       404
       404
       +
         for (const pair of queryString.split('&')) {

     

       405
       405
       +
           const [key, value] = pair.split('=');

     

       406
       406
       +
           if (key) {

     

       407
       407
       +
             params[decodeURIComponent(key)] = value ? decodeURIComponent(value) : '';

     

       408
       408
       +
           }

     

       409
       409
       +
         }

     

       410
       410
       +
         

     

       411
       411
       +
         return params;

     

       412
       412
       +
       }

     

       413
       413
       +

+168 -6

hosting-service/src/server.ts

···

       7
       7
        
       import { lookup } from 'mime-types';

     

       8
       8
        
       import { logger, observabilityMiddleware, observabilityErrorHandler, logCollector, errorTracker, metricsCollector } from './lib/observability';

     

       9
       9
        
       import { fileCache, metadataCache, rewrittenHtmlCache, getCacheKey, type FileMetadata } from './lib/cache';

     

       10
       10
       +
       import { loadRedirectRules, matchRedirectRule, parseCookies, parseQueryString, type RedirectRule } from './lib/redirects';

     

       10
       11
        
       

     

       11
       12
        
       const BASE_HOST = process.env.BASE_HOST || 'wisp.place';

     

       12
       13
        
       

     
···

       35
       36
        
         }

     

       36
       37
        
       }

     

       37
       38
        
       

     

       39
       39
       +
       // Cache for redirect rules (per site)

     

       40
       40
       +
       const redirectRulesCache = new Map<string, RedirectRule[]>();

     

       41
       41
       +
       

     

       42
       42
       +
       /**

     

       43
       43
       +
        * Clear redirect rules cache for a specific site

     

       44
       44
       +
        * Should be called when a site is updated/recached

     

       45
       45
       +
        */

     

       46
       46
       +
       export function clearRedirectRulesCache(did: string, rkey: string) {

     

       47
       47
       +
         const cacheKey = `${did}:${rkey}`;

     

       48
       48
       +
         redirectRulesCache.delete(cacheKey);

     

       49
       49
       +
       }

     

       50
       50
       +
       

     

       38
       51
        
       // Helper to serve files from cache

     

       39
       39
       -
       async function serveFromCache(did: string, rkey: string, filePath: string) {

     

       52
       52
       +
       async function serveFromCache(

     

       53
       53
       +
         did: string, 

     

       54
       54
       +
         rkey: string, 

     

       55
       55
       +
         filePath: string,

     

       56
       56
       +
         fullUrl?: string,

     

       57
       57
       +
         headers?: Record<string, string>

     

       58
       58
       +
       ) {

     

       59
       59
       +
         // Check for redirect rules first

     

       60
       60
       +
         const redirectCacheKey = `${did}:${rkey}`;

     

       61
       61
       +
         let redirectRules = redirectRulesCache.get(redirectCacheKey);

     

       62
       62
       +
         

     

       63
       63
       +
         if (redirectRules === undefined) {

     

       64
       64
       +
           // Load rules for the first time

     

       65
       65
       +
           redirectRules = await loadRedirectRules(did, rkey);

     

       66
       66
       +
           redirectRulesCache.set(redirectCacheKey, redirectRules);

     

       67
       67
       +
         }

     

       68
       68
       +
       

     

       69
       69
       +
         // Apply redirect rules if any exist

     

       70
       70
       +
         if (redirectRules.length > 0) {

     

       71
       71
       +
           const requestPath = '/' + (filePath || '');

     

       72
       72
       +
           const queryParams = fullUrl ? parseQueryString(fullUrl) : {};

     

       73
       73
       +
           const cookies = parseCookies(headers?.['cookie']);

     

       74
       74
       +
           

     

       75
       75
       +
           const redirectMatch = matchRedirectRule(requestPath, redirectRules, {

     

       76
       76
       +
             queryParams,

     

       77
       77
       +
             headers,

     

       78
       78
       +
             cookies,

     

       79
       79
       +
           });

     

       80
       80
       +
       

     

       81
       81
       +
           if (redirectMatch) {

     

       82
       82
       +
             const { targetPath, status } = redirectMatch;

     

       83
       83
       +
             

     

       84
       84
       +
             // Handle different status codes

     

       85
       85
       +
             if (status === 200) {

     

       86
       86
       +
               // Rewrite: serve different content but keep URL the same

     

       87
       87
       +
               // Remove leading slash for internal path resolution

     

       88
       88
       +
               const rewritePath = targetPath.startsWith('/') ? targetPath.slice(1) : targetPath;

     

       89
       89
       +
               return serveFileInternal(did, rkey, rewritePath);

     

       90
       90
       +
             } else if (status === 301 || status === 302) {

     

       91
       91
       +
               // External redirect: change the URL

     

       92
       92
       +
               return new Response(null, {

     

       93
       93
       +
                 status,

     

       94
       94
       +
                 headers: {

     

       95
       95
       +
                   'Location': targetPath,

     

       96
       96
       +
                   'Cache-Control': status === 301 ? 'public, max-age=31536000' : 'public, max-age=0',

     

       97
       97
       +
                 },

     

       98
       98
       +
               });

     

       99
       99
       +
             } else if (status === 404) {

     

       100
       100
       +
               // Custom 404 page

     

       101
       101
       +
               const custom404Path = targetPath.startsWith('/') ? targetPath.slice(1) : targetPath;

     

       102
       102
       +
               const response = await serveFileInternal(did, rkey, custom404Path);

     

       103
       103
       +
               // Override status to 404

     

       104
       104
       +
               return new Response(response.body, {

     

       105
       105
       +
                 status: 404,

     

       106
       106
       +
                 headers: response.headers,

     

       107
       107
       +
               });

     

       108
       108
       +
             }

     

       109
       109
       +
           }

     

       110
       110
       +
         }

     

       111
       111
       +
       

     

       112
       112
       +
         // No redirect matched, serve normally

     

       113
       113
       +
         return serveFileInternal(did, rkey, filePath);

     

       114
       114
       +
       }

     

       115
       115
       +
       

     

       116
       116
       +
       // Internal function to serve a file (used by both normal serving and rewrites)

     

       117
       117
       +
       async function serveFileInternal(did: string, rkey: string, filePath: string) {

     

       40
       118
        
         // Default to index.html if path is empty or ends with /

     

       41
       119
        
         let requestPath = filePath || 'index.html';

     

       42
       120
        
         if (requestPath.endsWith('/')) {

     
···

       138
       216
        
         did: string,

     

       139
       217
        
         rkey: string,

     

       140
       218
        
         filePath: string,

     

       141
       141
       -
         basePath: string

     

       219
       219
       +
         basePath: string,

     

       220
       220
       +
         fullUrl?: string,

     

       221
       221
       +
         headers?: Record<string, string>

     

       142
       222
        
       ) {

     

       223
       223
       +
         // Check for redirect rules first

     

       224
       224
       +
         const redirectCacheKey = `${did}:${rkey}`;

     

       225
       225
       +
         let redirectRules = redirectRulesCache.get(redirectCacheKey);

     

       226
       226
       +
         

     

       227
       227
       +
         if (redirectRules === undefined) {

     

       228
       228
       +
           // Load rules for the first time

     

       229
       229
       +
           redirectRules = await loadRedirectRules(did, rkey);

     

       230
       230
       +
           redirectRulesCache.set(redirectCacheKey, redirectRules);

     

       231
       231
       +
         }

     

       232
       232
       +
       

     

       233
       233
       +
         // Apply redirect rules if any exist

     

       234
       234
       +
         if (redirectRules.length > 0) {

     

       235
       235
       +
           const requestPath = '/' + (filePath || '');

     

       236
       236
       +
           const queryParams = fullUrl ? parseQueryString(fullUrl) : {};

     

       237
       237
       +
           const cookies = parseCookies(headers?.['cookie']);

     

       238
       238
       +
           

     

       239
       239
       +
           const redirectMatch = matchRedirectRule(requestPath, redirectRules, {

     

       240
       240
       +
             queryParams,

     

       241
       241
       +
             headers,

     

       242
       242
       +
             cookies,

     

       243
       243
       +
           });

     

       244
       244
       +
       

     

       245
       245
       +
           if (redirectMatch) {

     

       246
       246
       +
             const { targetPath, status } = redirectMatch;

     

       247
       247
       +
             

     

       248
       248
       +
             // Handle different status codes

     

       249
       249
       +
             if (status === 200) {

     

       250
       250
       +
               // Rewrite: serve different content but keep URL the same

     

       251
       251
       +
               const rewritePath = targetPath.startsWith('/') ? targetPath.slice(1) : targetPath;

     

       252
       252
       +
               return serveFileInternalWithRewrite(did, rkey, rewritePath, basePath);

     

       253
       253
       +
             } else if (status === 301 || status === 302) {

     

       254
       254
       +
               // External redirect: change the URL

     

       255
       255
       +
               // For sites.wisp.place, we need to adjust the target path to include the base path

     

       256
       256
       +
               // unless it's an absolute URL

     

       257
       257
       +
               let redirectTarget = targetPath;

     

       258
       258
       +
               if (!targetPath.startsWith('http://') && !targetPath.startsWith('https://')) {

     

       259
       259
       +
                 redirectTarget = basePath + (targetPath.startsWith('/') ? targetPath.slice(1) : targetPath);

     

       260
       260
       +
               }

     

       261
       261
       +
               return new Response(null, {

     

       262
       262
       +
                 status,

     

       263
       263
       +
                 headers: {

     

       264
       264
       +
                   'Location': redirectTarget,

     

       265
       265
       +
                   'Cache-Control': status === 301 ? 'public, max-age=31536000' : 'public, max-age=0',

     

       266
       266
       +
                 },

     

       267
       267
       +
               });

     

       268
       268
       +
             } else if (status === 404) {

     

       269
       269
       +
               // Custom 404 page

     

       270
       270
       +
               const custom404Path = targetPath.startsWith('/') ? targetPath.slice(1) : targetPath;

     

       271
       271
       +
               const response = await serveFileInternalWithRewrite(did, rkey, custom404Path, basePath);

     

       272
       272
       +
               // Override status to 404

     

       273
       273
       +
               return new Response(response.body, {

     

       274
       274
       +
                 status: 404,

     

       275
       275
       +
                 headers: response.headers,

     

       276
       276
       +
               });

     

       277
       277
       +
             }

     

       278
       278
       +
           }

     

       279
       279
       +
         }

     

       280
       280
       +
       

     

       281
       281
       +
         // No redirect matched, serve normally

     

       282
       282
       +
         return serveFileInternalWithRewrite(did, rkey, filePath, basePath);

     

       283
       283
       +
       }

     

       284
       284
       +
       

     

       285
       285
       +
       // Internal function to serve a file with rewriting

     

       286
       286
       +
       async function serveFileInternalWithRewrite(did: string, rkey: string, filePath: string, basePath: string) {

     

       143
       287
        
         // Default to index.html if path is empty or ends with /

     

       144
       288
        
         let requestPath = filePath || 'index.html';

     

       145
       289
        
         if (requestPath.endsWith('/')) {

     
···

       317
       461
        
       

     

       318
       462
        
         try {

     

       319
       463
        
           await downloadAndCacheSite(did, rkey, siteData.record, pdsEndpoint, siteData.cid);

     

       464
       464
       +
           // Clear redirect rules cache since the site was updated

     

       465
       465
       +
           clearRedirectRulesCache(did, rkey);

     

       320
       466
        
           logger.info('Site cached successfully', { did, rkey });

     

       321
       467
        
           return true;

     

       322
       468
        
         } catch (err) {

     
···

       384
       530
        
       

     

       385
       531
        
           // Serve with HTML path rewriting to handle absolute paths

     

       386
       532
        
           const basePath = `/${identifier}/${site}/`;

     

       387
       387
       -
           return serveFromCacheWithRewrite(did, site, filePath, basePath);

     

       533
       533
       +
           const headers: Record<string, string> = {};

     

       534
       534
       +
           c.req.raw.headers.forEach((value, key) => {

     

       535
       535
       +
             headers[key.toLowerCase()] = value;

     

       536
       536
       +
           });

     

       537
       537
       +
           return serveFromCacheWithRewrite(did, site, filePath, basePath, c.req.url, headers);

     

       388
       538
        
         }

     

       389
       539
        
       

     

       390
       540
        
         // Check if this is a DNS hash subdomain

     
···

       420
       570
        
             return c.text('Site not found', 404);

     

       421
       571
        
           }

     

       422
       572
        
       

     

       423
       423
       -
           return serveFromCache(customDomain.did, rkey, path);

     

       573
       573
       +
           const headers: Record<string, string> = {};

     

       574
       574
       +
           c.req.raw.headers.forEach((value, key) => {

     

       575
       575
       +
             headers[key.toLowerCase()] = value;

     

       576
       576
       +
           });

     

       577
       577
       +
           return serveFromCache(customDomain.did, rkey, path, c.req.url, headers);

     

       424
       578
        
         }

     

       425
       579
        
       

     

       426
       580
        
         // Route 2: Registered subdomains - /*.wisp.place/*

     
···

       444
       598
        
             return c.text('Site not found', 404);

     

       445
       599
        
           }

     

       446
       600
        
       

     

       447
       447
       -
           return serveFromCache(domainInfo.did, rkey, path);

     

       601
       601
       +
           const headers: Record<string, string> = {};

     

       602
       602
       +
           c.req.raw.headers.forEach((value, key) => {

     

       603
       603
       +
             headers[key.toLowerCase()] = value;

     

       604
       604
       +
           });

     

       605
       605
       +
           return serveFromCache(domainInfo.did, rkey, path, c.req.url, headers);

     

       448
       606
        
         }

     

       449
       607
        
       

     

       450
       608
        
         // Route 1: Custom domains - /*

     
···

       467
       625
        
           return c.text('Site not found', 404);

     

       468
       626
        
         }

     

       469
       627
        
       

     

       470
       470
       -
         return serveFromCache(customDomain.did, rkey, path);

     

       628
       628
       +
         const headers: Record<string, string> = {};

     

       629
       629
       +
         c.req.raw.headers.forEach((value, key) => {

     

       630
       630
       +
           headers[key.toLowerCase()] = value;

     

       631
       631
       +
         });

     

       632
       632
       +
         return serveFromCache(customDomain.did, rkey, path, c.req.url, headers);

     

       471
       633
        
       });

     

       472
       634
        
       

     

       473
       635
        
       // Internal observability endpoints (for admin panel)

cli/.gitignore

+66

cli/src/cid.rs

···

       1
       1
       +
       use jacquard_common::types::cid::IpldCid;

     

       2
       2
       +
       use sha2::{Digest, Sha256};

     

       3
       3
       +
       

     

       4
       4
       +
       /// Compute CID (Content Identifier) for blob content

     

       5
       5
       +
       /// Uses the same algorithm as AT Protocol: CIDv1 with raw codec (0x55) and SHA-256

     

       6
       6
       +
       /// 

     

       7
       7
       +
       /// CRITICAL: This must be called on BASE64-ENCODED GZIPPED content, not just gzipped content

     

       8
       8
       +
       /// 

     

       9
       9
       +
       /// Based on @atproto/common/src/ipld.ts sha256RawToCid implementation

     

       10
       10
       +
       pub fn compute_cid(content: &[u8]) -> String {

     

       11
       11
       +
           // Use node crypto to compute sha256 hash (same as AT Protocol)

     

       12
       12
       +
           let hash = Sha256::digest(content);

     

       13
       13
       +
           

     

       14
       14
       +
           // Create multihash (code 0x12 = sha2-256)

     

       15
       15
       +
           let multihash = multihash::Multihash::wrap(0x12, &hash)

     

       16
       16
       +
               .expect("SHA-256 hash should always fit in multihash");

     

       17
       17
       +
           

     

       18
       18
       +
           // Create CIDv1 with raw codec (0x55)

     

       19
       19
       +
           let cid = IpldCid::new_v1(0x55, multihash);

     

       20
       20
       +
           

     

       21
       21
       +
           // Convert to base32 string representation

     

       22
       22
       +
           cid.to_string_of_base(multibase::Base::Base32Lower)

     

       23
       23
       +
               .unwrap_or_else(|_| cid.to_string())

     

       24
       24
       +
       }

     

       25
       25
       +
       

     

       26
       26
       +
       #[cfg(test)]

     

       27
       27
       +
       mod tests {

     

       28
       28
       +
           use super::*;

     

       29
       29
       +
           use base64::Engine;

     

       30
       30
       +
       

     

       31
       31
       +
           #[test]

     

       32
       32
       +
           fn test_compute_cid() {

     

       33
       33
       +
               // Test with a simple string: "hello"

     

       34
       34
       +
               let content = b"hello";

     

       35
       35
       +
               let cid = compute_cid(content);

     

       36
       36
       +
               

     

       37
       37
       +
               // CID should start with 'baf' for raw codec base32

     

       38
       38
       +
               assert!(cid.starts_with("baf"));

     

       39
       39
       +
           }

     

       40
       40
       +
       

     

       41
       41
       +
           #[test]

     

       42
       42
       +
           fn test_compute_cid_base64_encoded() {

     

       43
       43
       +
               // Simulate the actual use case: gzipped then base64 encoded

     

       44
       44
       +
               use flate2::write::GzEncoder;

     

       45
       45
       +
               use flate2::Compression;

     

       46
       46
       +
               use std::io::Write;

     

       47
       47
       +
               

     

       48
       48
       +
               let original = b"hello world";

     

       49
       49
       +
               

     

       50
       50
       +
               // Gzip compress

     

       51
       51
       +
               let mut encoder = GzEncoder::new(Vec::new(), Compression::default());

     

       52
       52
       +
               encoder.write_all(original).unwrap();

     

       53
       53
       +
               let gzipped = encoder.finish().unwrap();

     

       54
       54
       +
               

     

       55
       55
       +
               // Base64 encode the gzipped data

     

       56
       56
       +
               let base64_bytes = base64::prelude::BASE64_STANDARD.encode(&gzipped).into_bytes();

     

       57
       57
       +
               

     

       58
       58
       +
               // Compute CID on the base64 bytes

     

       59
       59
       +
               let cid = compute_cid(&base64_bytes);

     

       60
       60
       +
               

     

       61
       61
       +
               // Should be a valid CID

     

       62
       62
       +
               assert!(cid.starts_with("baf"));

     

       63
       63
       +
               assert!(cid.len() > 10);

     

       64
       64
       +
           }

     

       65
       65
       +
       }

     

       66
       66
       +

+71

cli/src/download.rs

···

       1
       1
       +
       use base64::Engine;

     

       2
       2
       +
       use bytes::Bytes;

     

       3
       3
       +
       use flate2::read::GzDecoder;

     

       4
       4
       +
       use jacquard_common::types::blob::BlobRef;

     

       5
       5
       +
       use miette::IntoDiagnostic;

     

       6
       6
       +
       use std::io::Read;

     

       7
       7
       +
       use url::Url;

     

       8
       8
       +
       

     

       9
       9
       +
       /// Download a blob from the PDS

     

       10
       10
       +
       pub async fn download_blob(pds_url: &Url, blob_ref: &BlobRef<'_>, did: &str) -> miette::Result<Bytes> {

     

       11
       11
       +
           // Extract CID from blob ref

     

       12
       12
       +
           let cid = blob_ref.blob().r#ref.to_string();

     

       13
       13
       +
           

     

       14
       14
       +
           // Construct blob download URL

     

       15
       15
       +
           // The correct endpoint is: /xrpc/com.atproto.sync.getBlob?did={did}&cid={cid}

     

       16
       16
       +
           let blob_url = pds_url

     

       17
       17
       +
               .join(&format!("/xrpc/com.atproto.sync.getBlob?did={}&cid={}", did, cid))

     

       18
       18
       +
               .into_diagnostic()?;

     

       19
       19
       +
           

     

       20
       20
       +
           let client = reqwest::Client::new();

     

       21
       21
       +
           let response = client

     

       22
       22
       +
               .get(blob_url)

     

       23
       23
       +
               .send()

     

       24
       24
       +
               .await

     

       25
       25
       +
               .into_diagnostic()?;

     

       26
       26
       +
           

     

       27
       27
       +
           if !response.status().is_success() {

     

       28
       28
       +
               return Err(miette::miette!(

     

       29
       29
       +
                   "Failed to download blob: {}",

     

       30
       30
       +
                   response.status()

     

       31
       31
       +
               ));

     

       32
       32
       +
           }

     

       33
       33
       +
           

     

       34
       34
       +
           let bytes = response.bytes().await.into_diagnostic()?;

     

       35
       35
       +
           Ok(bytes)

     

       36
       36
       +
       }

     

       37
       37
       +
       

     

       38
       38
       +
       /// Decompress and decode a blob (base64 + gzip)

     

       39
       39
       +
       pub fn decompress_blob(data: &[u8], is_base64: bool, is_gzipped: bool) -> miette::Result<Vec<u8>> {

     

       40
       40
       +
           let mut current_data = data.to_vec();

     

       41
       41
       +
           

     

       42
       42
       +
           // First, decode base64 if needed

     

       43
       43
       +
           if is_base64 {

     

       44
       44
       +
               current_data = base64::prelude::BASE64_STANDARD

     

       45
       45
       +
                   .decode(&current_data)

     

       46
       46
       +
                   .into_diagnostic()?;

     

       47
       47
       +
           }

     

       48
       48
       +
           

     

       49
       49
       +
           // Then, decompress gzip if needed

     

       50
       50
       +
           if is_gzipped {

     

       51
       51
       +
               let mut decoder = GzDecoder::new(&current_data[..]);

     

       52
       52
       +
               let mut decompressed = Vec::new();

     

       53
       53
       +
               decoder.read_to_end(&mut decompressed).into_diagnostic()?;

     

       54
       54
       +
               current_data = decompressed;

     

       55
       55
       +
           }

     

       56
       56
       +
           

     

       57
       57
       +
           Ok(current_data)

     

       58
       58
       +
       }

     

       59
       59
       +
       

     

       60
       60
       +
       /// Download and decompress a blob

     

       61
       61
       +
       pub async fn download_and_decompress_blob(

     

       62
       62
       +
           pds_url: &Url,

     

       63
       63
       +
           blob_ref: &BlobRef<'_>,

     

       64
       64
       +
           did: &str,

     

       65
       65
       +
           is_base64: bool,

     

       66
       66
       +
           is_gzipped: bool,

     

       67
       67
       +
       ) -> miette::Result<Vec<u8>> {

     

       68
       68
       +
           let data = download_blob(pds_url, blob_ref, did).await?;

     

       69
       69
       +
           decompress_blob(&data, is_base64, is_gzipped)

     

       70
       70
       +
       }

     

       71
       71
       +

+109 -16

cli/src/main.rs

···

       2
       2
        
       mod place_wisp;

     

       3
       3
        
       mod cid;

     

       4
       4
        
       mod blob_map;

     

       5
       5
       +
       mod metadata;

     

       6
       6
       +
       mod download;

     

       7
       7
       +
       mod pull;

     

       8
       8
       +
       mod serve;

     

       5
       9
        
       

     

       6
       6
       -
       use clap::Parser;

     

       10
       10
       +
       use clap::{Parser, Subcommand};

     

       7
       11
        
       use jacquard::CowStr;

     

       8
       12
        
       use jacquard::client::{Agent, FileAuthStore, AgentSessionExt, MemoryCredentialSession, AgentSession};

     

       9
       13
        
       use jacquard::oauth::client::OAuthClient;

     
···

       23
       27
        
       use place_wisp::fs::*;

     

       24
       28
        
       

     

       25
       29
        
       #[derive(Parser, Debug)]

     

       26
       26
       -
       #[command(author, version, about = "Deploy a static site to wisp.place")]

     

       30
       30
       +
       #[command(author, version, about = "wisp.place CLI tool")]

     

       27
       31
        
       struct Args {

     

       32
       32
       +
           #[command(subcommand)]

     

       33
       33
       +
           command: Option<Commands>,

     

       34
       34
       +
           

     

       35
       35
       +
           // Deploy arguments (when no subcommand is specified)

     

       28
       36
        
           /// Handle (e.g., alice.bsky.social), DID, or PDS URL

     

       29
       29
       -
           input: CowStr<'static>,

     

       37
       37
       +
           #[arg(global = true, conflicts_with = "command")]

     

       38
       38
       +
           input: Option<CowStr<'static>>,

     

       30
       39
        
       

     

       31
       40
        
           /// Path to the directory containing your static site

     

       32
       32
       -
           #[arg(short, long, default_value = ".")]

     

       33
       33
       -
           path: PathBuf,

     

       41
       41
       +
           #[arg(short, long, global = true, conflicts_with = "command")]

     

       42
       42
       +
           path: Option<PathBuf>,

     

       34
       43
        
       

     

       35
       44
        
           /// Site name (defaults to directory name)

     

       36
       36
       -
           #[arg(short, long)]

     

       45
       45
       +
           #[arg(short, long, global = true, conflicts_with = "command")]

     

       37
       46
        
           site: Option<String>,

     

       38
       47
        
       

     

       39
       39
       -
           /// Path to auth store file (will be created if missing, only used with OAuth)

     

       40
       40
       -
           #[arg(long, default_value = "/tmp/wisp-oauth-session.json")]

     

       41
       41
       -
           store: String,

     

       48
       48
       +
           /// Path to auth store file

     

       49
       49
       +
           #[arg(long, global = true, conflicts_with = "command")]

     

       50
       50
       +
           store: Option<String>,

     

       42
       51
        
       

     

       43
       43
       -
           /// App Password for authentication (alternative to OAuth)

     

       44
       44
       -
           #[arg(long)]

     

       52
       52
       +
           /// App Password for authentication

     

       53
       53
       +
           #[arg(long, global = true, conflicts_with = "command")]

     

       45
       54
        
           password: Option<CowStr<'static>>,

     

       46
       55
        
       }

     

       47
       56
        
       

     

       57
       57
       +
       #[derive(Subcommand, Debug)]

     

       58
       58
       +
       enum Commands {

     

       59
       59
       +
           /// Deploy a static site to wisp.place (default command)

     

       60
       60
       +
           Deploy {

     

       61
       61
       +
               /// Handle (e.g., alice.bsky.social), DID, or PDS URL

     

       62
       62
       +
               input: CowStr<'static>,

     

       63
       63
       +
       

     

       64
       64
       +
               /// Path to the directory containing your static site

     

       65
       65
       +
               #[arg(short, long, default_value = ".")]

     

       66
       66
       +
               path: PathBuf,

     

       67
       67
       +
       

     

       68
       68
       +
               /// Site name (defaults to directory name)

     

       69
       69
       +
               #[arg(short, long)]

     

       70
       70
       +
               site: Option<String>,

     

       71
       71
       +
       

     

       72
       72
       +
               /// Path to auth store file (will be created if missing, only used with OAuth)

     

       73
       73
       +
               #[arg(long, default_value = "/tmp/wisp-oauth-session.json")]

     

       74
       74
       +
               store: String,

     

       75
       75
       +
       

     

       76
       76
       +
               /// App Password for authentication (alternative to OAuth)

     

       77
       77
       +
               #[arg(long)]

     

       78
       78
       +
               password: Option<CowStr<'static>>,

     

       79
       79
       +
           },

     

       80
       80
       +
           /// Pull a site from the PDS to a local directory

     

       81
       81
       +
           Pull {

     

       82
       82
       +
               /// Handle (e.g., alice.bsky.social) or DID

     

       83
       83
       +
               input: CowStr<'static>,

     

       84
       84
       +
       

     

       85
       85
       +
               /// Site name (record key)

     

       86
       86
       +
               #[arg(short, long)]

     

       87
       87
       +
               site: String,

     

       88
       88
       +
       

     

       89
       89
       +
               /// Output directory for the downloaded site

     

       90
       90
       +
               #[arg(short, long, default_value = ".")]

     

       91
       91
       +
               output: PathBuf,

     

       92
       92
       +
           },

     

       93
       93
       +
           /// Serve a site locally with real-time firehose updates

     

       94
       94
       +
           Serve {

     

       95
       95
       +
               /// Handle (e.g., alice.bsky.social) or DID

     

       96
       96
       +
               input: CowStr<'static>,

     

       97
       97
       +
       

     

       98
       98
       +
               /// Site name (record key)

     

       99
       99
       +
               #[arg(short, long)]

     

       100
       100
       +
               site: String,

     

       101
       101
       +
       

     

       102
       102
       +
               /// Output directory for the site files

     

       103
       103
       +
               #[arg(short, long, default_value = ".")]

     

       104
       104
       +
               output: PathBuf,

     

       105
       105
       +
       

     

       106
       106
       +
               /// Port to serve on

     

       107
       107
       +
               #[arg(short, long, default_value = "8080")]

     

       108
       108
       +
               port: u16,

     

       109
       109
       +
           },

     

       110
       110
       +
       }

     

       111
       111
       +
       

     

       48
       112
        
       #[tokio::main]

     

       49
       113
        
       async fn main() -> miette::Result<()> {

     

       50
       114
        
           let args = Args::parse();

     

       51
       115
        
       

     

       52
       52
       -
           // Dispatch to appropriate authentication method

     

       53
       53
       -
           if let Some(password) = args.password {

     

       54
       54
       -
               run_with_app_password(args.input, password, args.path, args.site).await

     

       55
       55
       -
           } else {

     

       56
       56
       -
               run_with_oauth(args.input, args.store, args.path, args.site).await

     

       116
       116
       +
           match args.command {

     

       117
       117
       +
               Some(Commands::Deploy { input, path, site, store, password }) => {

     

       118
       118
       +
                   // Dispatch to appropriate authentication method

     

       119
       119
       +
                   if let Some(password) = password {

     

       120
       120
       +
                       run_with_app_password(input, password, path, site).await

     

       121
       121
       +
                   } else {

     

       122
       122
       +
                       run_with_oauth(input, store, path, site).await

     

       123
       123
       +
                   }

     

       124
       124
       +
               }

     

       125
       125
       +
               Some(Commands::Pull { input, site, output }) => {

     

       126
       126
       +
                   pull::pull_site(input, CowStr::from(site), output).await

     

       127
       127
       +
               }

     

       128
       128
       +
               Some(Commands::Serve { input, site, output, port }) => {

     

       129
       129
       +
                   serve::serve_site(input, CowStr::from(site), output, port).await

     

       130
       130
       +
               }

     

       131
       131
       +
               None => {

     

       132
       132
       +
                   // Legacy mode: if input is provided, assume deploy command

     

       133
       133
       +
                   if let Some(input) = args.input {

     

       134
       134
       +
                       let path = args.path.unwrap_or_else(|| PathBuf::from("."));

     

       135
       135
       +
                       let store = args.store.unwrap_or_else(|| "/tmp/wisp-oauth-session.json".to_string());

     

       136
       136
       +
                       

     

       137
       137
       +
                       // Dispatch to appropriate authentication method

     

       138
       138
       +
                       if let Some(password) = args.password {

     

       139
       139
       +
                           run_with_app_password(input, password, path, args.site).await

     

       140
       140
       +
                       } else {

     

       141
       141
       +
                           run_with_oauth(input, store, path, args.site).await

     

       142
       142
       +
                       }

     

       143
       143
       +
                   } else {

     

       144
       144
       +
                       // No command and no input, show help

     

       145
       145
       +
                       use clap::CommandFactory;

     

       146
       146
       +
                       Args::command().print_help().into_diagnostic()?;

     

       147
       147
       +
                       Ok(())

     

       148
       148
       +
                   }

     

       149
       149
       +
               }

     

       57
       150
        
           }

     

       58
       151
        
       }

     

       59
       152

+46

cli/src/metadata.rs

···

       1
       1
       +
       use serde::{Deserialize, Serialize};

     

       2
       2
       +
       use std::collections::HashMap;

     

       3
       3
       +
       use std::path::Path;

     

       4
       4
       +
       use miette::IntoDiagnostic;

     

       5
       5
       +
       

     

       6
       6
       +
       /// Metadata tracking file CIDs for incremental updates

     

       7
       7
       +
       #[derive(Debug, Clone, Serialize, Deserialize)]

     

       8
       8
       +
       pub struct SiteMetadata {

     

       9
       9
       +
           /// Record CID from the PDS

     

       10
       10
       +
           pub record_cid: String,

     

       11
       11
       +
           /// Map of file paths to their blob CIDs

     

       12
       12
       +
           pub file_cids: HashMap<String, String>,

     

       13
       13
       +
           /// Timestamp when the site was last synced

     

       14
       14
       +
           pub last_sync: i64,

     

       15
       15
       +
       }

     

       16
       16
       +
       

     

       17
       17
       +
       impl SiteMetadata {

     

       18
       18
       +
           pub fn new(record_cid: String, file_cids: HashMap<String, String>) -> Self {

     

       19
       19
       +
               Self {

     

       20
       20
       +
                   record_cid,

     

       21
       21
       +
                   file_cids,

     

       22
       22
       +
                   last_sync: chrono::Utc::now().timestamp(),

     

       23
       23
       +
               }

     

       24
       24
       +
           }

     

       25
       25
       +
       

     

       26
       26
       +
           /// Load metadata from a directory

     

       27
       27
       +
           pub fn load(dir: &Path) -> miette::Result<Option<Self>> {

     

       28
       28
       +
               let metadata_path = dir.join(".wisp-metadata.json");

     

       29
       29
       +
               if !metadata_path.exists() {

     

       30
       30
       +
                   return Ok(None);

     

       31
       31
       +
               }

     

       32
       32
       +
       

     

       33
       33
       +
               let contents = std::fs::read_to_string(&metadata_path).into_diagnostic()?;

     

       34
       34
       +
               let metadata: SiteMetadata = serde_json::from_str(&contents).into_diagnostic()?;

     

       35
       35
       +
               Ok(Some(metadata))

     

       36
       36
       +
           }

     

       37
       37
       +
       

     

       38
       38
       +
           /// Save metadata to a directory

     

       39
       39
       +
           pub fn save(&self, dir: &Path) -> miette::Result<()> {

     

       40
       40
       +
               let metadata_path = dir.join(".wisp-metadata.json");

     

       41
       41
       +
               let contents = serde_json::to_string_pretty(self).into_diagnostic()?;

     

       42
       42
       +
               std::fs::write(&metadata_path, contents).into_diagnostic()?;

     

       43
       43
       +
               Ok(())

     

       44
       44
       +
           }

     

       45
       45
       +
       }

     

       46
       46
       +

+305

cli/src/pull.rs

···

       1
       1
       +
       use crate::blob_map;

     

       2
       2
       +
       use crate::download;

     

       3
       3
       +
       use crate::metadata::SiteMetadata;

     

       4
       4
       +
       use crate::place_wisp::fs::*;

     

       5
       5
       +
       use jacquard::CowStr;

     

       6
       6
       +
       use jacquard::prelude::IdentityResolver;

     

       7
       7
       +
       use jacquard_common::types::string::Did;

     

       8
       8
       +
       use jacquard_common::xrpc::XrpcExt;

     

       9
       9
       +
       use jacquard_identity::PublicResolver;

     

       10
       10
       +
       use miette::IntoDiagnostic;

     

       11
       11
       +
       use std::collections::HashMap;

     

       12
       12
       +
       use std::path::{Path, PathBuf};

     

       13
       13
       +
       use url::Url;

     

       14
       14
       +
       

     

       15
       15
       +
       /// Pull a site from the PDS to a local directory

     

       16
       16
       +
       pub async fn pull_site(

     

       17
       17
       +
           input: CowStr<'static>,

     

       18
       18
       +
           rkey: CowStr<'static>,

     

       19
       19
       +
           output_dir: PathBuf,

     

       20
       20
       +
       ) -> miette::Result<()> {

     

       21
       21
       +
           println!("Pulling site {} from {}...", rkey, input);

     

       22
       22
       +
       

     

       23
       23
       +
           // Resolve handle to DID if needed

     

       24
       24
       +
           let resolver = PublicResolver::default();

     

       25
       25
       +
           let did = if input.starts_with("did:") {

     

       26
       26
       +
               Did::new(&input).into_diagnostic()?

     

       27
       27
       +
           } else {

     

       28
       28
       +
               // It's a handle, resolve it

     

       29
       29
       +
               let handle = jacquard_common::types::string::Handle::new(&input).into_diagnostic()?;

     

       30
       30
       +
               resolver.resolve_handle(&handle).await.into_diagnostic()?

     

       31
       31
       +
           };

     

       32
       32
       +
       

     

       33
       33
       +
           // Resolve PDS endpoint for the DID

     

       34
       34
       +
           let pds_url = resolver.pds_for_did(&did).await.into_diagnostic()?;

     

       35
       35
       +
           println!("Resolved PDS: {}", pds_url);

     

       36
       36
       +
       

     

       37
       37
       +
           // Fetch the place.wisp.fs record

     

       38
       38
       +
       

     

       39
       39
       +
           println!("Fetching record from PDS...");

     

       40
       40
       +
           let client = reqwest::Client::new();

     

       41
       41
       +
           

     

       42
       42
       +
           // Use com.atproto.repo.getRecord

     

       43
       43
       +
           use jacquard::api::com_atproto::repo::get_record::GetRecord;

     

       44
       44
       +
           use jacquard_common::types::string::Rkey as RkeyType;

     

       45
       45
       +
           let rkey_parsed = RkeyType::new(&rkey).into_diagnostic()?;

     

       46
       46
       +
           

     

       47
       47
       +
           use jacquard_common::types::ident::AtIdentifier;

     

       48
       48
       +
           use jacquard_common::types::string::RecordKey;

     

       49
       49
       +
           let request = GetRecord::new()

     

       50
       50
       +
               .repo(AtIdentifier::Did(did.clone()))

     

       51
       51
       +
               .collection(CowStr::from("place.wisp.fs"))

     

       52
       52
       +
               .rkey(RecordKey::from(rkey_parsed))

     

       53
       53
       +
               .build();

     

       54
       54
       +
       

     

       55
       55
       +
           let response = client

     

       56
       56
       +
               .xrpc(pds_url.clone())

     

       57
       57
       +
               .send(&request)

     

       58
       58
       +
               .await

     

       59
       59
       +
               .into_diagnostic()?;

     

       60
       60
       +
       

     

       61
       61
       +
           let record_output = response.into_output().into_diagnostic()?;

     

       62
       62
       +
           let record_cid = record_output.cid.as_ref().map(|c| c.to_string()).unwrap_or_default();

     

       63
       63
       +
       

     

       64
       64
       +
           // Parse the record value as Fs

     

       65
       65
       +
           use jacquard_common::types::value::from_data;

     

       66
       66
       +
           let fs_record: Fs = from_data(&record_output.value).into_diagnostic()?;

     

       67
       67
       +
       

     

       68
       68
       +
           let file_count = fs_record.file_count.map(|c| c.to_string()).unwrap_or_else(|| "?".to_string());

     

       69
       69
       +
           println!("Found site '{}' with {} files", fs_record.site, file_count);

     

       70
       70
       +
       

     

       71
       71
       +
           // Load existing metadata for incremental updates

     

       72
       72
       +
           let existing_metadata = SiteMetadata::load(&output_dir)?;

     

       73
       73
       +
           let existing_file_cids = existing_metadata

     

       74
       74
       +
               .as_ref()

     

       75
       75
       +
               .map(|m| m.file_cids.clone())

     

       76
       76
       +
               .unwrap_or_default();

     

       77
       77
       +
       

     

       78
       78
       +
           // Extract blob map from the new manifest

     

       79
       79
       +
           let new_blob_map = blob_map::extract_blob_map(&fs_record.root);

     

       80
       80
       +
           let new_file_cids: HashMap<String, String> = new_blob_map

     

       81
       81
       +
               .iter()

     

       82
       82
       +
               .map(|(path, (_blob_ref, cid))| (path.clone(), cid.clone()))

     

       83
       83
       +
               .collect();

     

       84
       84
       +
       

     

       85
       85
       +
           // Clean up any leftover temp directories from previous failed attempts

     

       86
       86
       +
           let parent = output_dir.parent().unwrap_or_else(|| std::path::Path::new("."));

     

       87
       87
       +
           let output_name = output_dir.file_name().unwrap_or_else(|| std::ffi::OsStr::new("site")).to_string_lossy();

     

       88
       88
       +
           let temp_prefix = format!(".tmp-{}-", output_name);

     

       89
       89
       +
           

     

       90
       90
       +
           if let Ok(entries) = parent.read_dir() {

     

       91
       91
       +
               for entry in entries.flatten() {

     

       92
       92
       +
                   let name = entry.file_name();

     

       93
       93
       +
                   if name.to_string_lossy().starts_with(&temp_prefix) {

     

       94
       94
       +
                       let _ = std::fs::remove_dir_all(entry.path());

     

       95
       95
       +
                   }

     

       96
       96
       +
               }

     

       97
       97
       +
           }

     

       98
       98
       +
       

     

       99
       99
       +
           // Check if we need to update (but only if output directory actually exists with files)

     

       100
       100
       +
           if let Some(metadata) = &existing_metadata {

     

       101
       101
       +
               if metadata.record_cid == record_cid {

     

       102
       102
       +
                   // Verify that the output directory actually exists and has content

     

       103
       103
       +
                   let has_content = output_dir.exists() && 

     

       104
       104
       +
                       output_dir.read_dir()

     

       105
       105
       +
                           .map(|mut entries| entries.any(|e| {

     

       106
       106
       +
                               if let Ok(entry) = e {

     

       107
       107
       +
                                   !entry.file_name().to_string_lossy().starts_with(".wisp-metadata")

     

       108
       108
       +
                               } else {

     

       109
       109
       +
                                   false

     

       110
       110
       +
                               }

     

       111
       111
       +
                           }))

     

       112
       112
       +
                           .unwrap_or(false);

     

       113
       113
       +
                   

     

       114
       114
       +
                   if has_content {

     

       115
       115
       +
                       println!("Site is already up to date!");

     

       116
       116
       +
                       return Ok(());

     

       117
       117
       +
                   }

     

       118
       118
       +
               }

     

       119
       119
       +
           }

     

       120
       120
       +
       

     

       121
       121
       +
           // Create temporary directory for atomic update

     

       122
       122
       +
           // Place temp dir in parent directory to avoid issues with non-existent output_dir

     

       123
       123
       +
           let parent = output_dir.parent().unwrap_or_else(|| std::path::Path::new("."));

     

       124
       124
       +
           let temp_dir_name = format!(

     

       125
       125
       +
               ".tmp-{}-{}",

     

       126
       126
       +
               output_dir.file_name().unwrap_or_else(|| std::ffi::OsStr::new("site")).to_string_lossy(),

     

       127
       127
       +
               chrono::Utc::now().timestamp()

     

       128
       128
       +
           );

     

       129
       129
       +
           let temp_dir = parent.join(temp_dir_name);

     

       130
       130
       +
           std::fs::create_dir_all(&temp_dir).into_diagnostic()?;

     

       131
       131
       +
       

     

       132
       132
       +
           println!("Downloading files...");

     

       133
       133
       +
           let mut downloaded = 0;

     

       134
       134
       +
           let mut reused = 0;

     

       135
       135
       +
       

     

       136
       136
       +
           // Download files recursively

     

       137
       137
       +
           let download_result = download_directory(

     

       138
       138
       +
               &fs_record.root,

     

       139
       139
       +
               &temp_dir,

     

       140
       140
       +
               &pds_url,

     

       141
       141
       +
               did.as_str(),

     

       142
       142
       +
               &new_blob_map,

     

       143
       143
       +
               &existing_file_cids,

     

       144
       144
       +
               &output_dir,

     

       145
       145
       +
               String::new(),

     

       146
       146
       +
               &mut downloaded,

     

       147
       147
       +
               &mut reused,

     

       148
       148
       +
           )

     

       149
       149
       +
           .await;

     

       150
       150
       +
       

     

       151
       151
       +
           // If download failed, clean up temp directory

     

       152
       152
       +
           if let Err(e) = download_result {

     

       153
       153
       +
               let _ = std::fs::remove_dir_all(&temp_dir);

     

       154
       154
       +
               return Err(e);

     

       155
       155
       +
           }

     

       156
       156
       +
       

     

       157
       157
       +
           println!(

     

       158
       158
       +
               "Downloaded {} files, reused {} files",

     

       159
       159
       +
               downloaded, reused

     

       160
       160
       +
           );

     

       161
       161
       +
       

     

       162
       162
       +
           // Save metadata

     

       163
       163
       +
           let metadata = SiteMetadata::new(record_cid, new_file_cids);

     

       164
       164
       +
           metadata.save(&temp_dir)?;

     

       165
       165
       +
       

     

       166
       166
       +
           // Move files from temp to output directory

     

       167
       167
       +
           let output_abs = std::fs::canonicalize(&output_dir).unwrap_or_else(|_| output_dir.clone());

     

       168
       168
       +
           let current_dir = std::env::current_dir().into_diagnostic()?;

     

       169
       169
       +
           

     

       170
       170
       +
           // Special handling for pulling to current directory

     

       171
       171
       +
           if output_abs == current_dir {

     

       172
       172
       +
               // Move files from temp to current directory

     

       173
       173
       +
               for entry in std::fs::read_dir(&temp_dir).into_diagnostic()? {

     

       174
       174
       +
                   let entry = entry.into_diagnostic()?;

     

       175
       175
       +
                   let dest = current_dir.join(entry.file_name());

     

       176
       176
       +
                   

     

       177
       177
       +
                   // Remove existing file/dir if it exists

     

       178
       178
       +
                   if dest.exists() {

     

       179
       179
       +
                       if dest.is_dir() {

     

       180
       180
       +
                           std::fs::remove_dir_all(&dest).into_diagnostic()?;

     

       181
       181
       +
                       } else {

     

       182
       182
       +
                           std::fs::remove_file(&dest).into_diagnostic()?;

     

       183
       183
       +
                       }

     

       184
       184
       +
                   }

     

       185
       185
       +
                   

     

       186
       186
       +
                   // Move from temp to current dir

     

       187
       187
       +
                   std::fs::rename(entry.path(), dest).into_diagnostic()?;

     

       188
       188
       +
               }

     

       189
       189
       +
               

     

       190
       190
       +
               // Clean up temp directory

     

       191
       191
       +
               std::fs::remove_dir_all(&temp_dir).into_diagnostic()?;

     

       192
       192
       +
           } else {

     

       193
       193
       +
               // If output directory exists and has content, remove it first

     

       194
       194
       +
               if output_dir.exists() {

     

       195
       195
       +
                   std::fs::remove_dir_all(&output_dir).into_diagnostic()?;

     

       196
       196
       +
               }

     

       197
       197
       +
               

     

       198
       198
       +
               // Ensure parent directory exists

     

       199
       199
       +
               if let Some(parent) = output_dir.parent() {

     

       200
       200
       +
                   if !parent.as_os_str().is_empty() && !parent.exists() {

     

       201
       201
       +
                       std::fs::create_dir_all(parent).into_diagnostic()?;

     

       202
       202
       +
                   }

     

       203
       203
       +
               }

     

       204
       204
       +
               

     

       205
       205
       +
               // Rename temp to final location

     

       206
       206
       +
               match std::fs::rename(&temp_dir, &output_dir) {

     

       207
       207
       +
                   Ok(_) => {},

     

       208
       208
       +
                   Err(e) => {

     

       209
       209
       +
                       // Clean up temp directory on failure

     

       210
       210
       +
                       let _ = std::fs::remove_dir_all(&temp_dir);

     

       211
       211
       +
                       return Err(miette::miette!("Failed to move temp directory: {}", e));

     

       212
       212
       +
                   }

     

       213
       213
       +
               }

     

       214
       214
       +
           }

     

       215
       215
       +
       

     

       216
       216
       +
           println!("✓ Site pulled successfully to {}", output_dir.display());

     

       217
       217
       +
       

     

       218
       218
       +
           Ok(())

     

       219
       219
       +
       }

     

       220
       220
       +
       

     

       221
       221
       +
       /// Recursively download a directory

     

       222
       222
       +
       fn download_directory<'a>(

     

       223
       223
       +
           dir: &'a Directory<'_>,

     

       224
       224
       +
           output_dir: &'a Path,

     

       225
       225
       +
           pds_url: &'a Url,

     

       226
       226
       +
           did: &'a str,

     

       227
       227
       +
           new_blob_map: &'a HashMap<String, (jacquard_common::types::blob::BlobRef<'static>, String)>,

     

       228
       228
       +
           existing_file_cids: &'a HashMap<String, String>,

     

       229
       229
       +
           existing_output_dir: &'a Path,

     

       230
       230
       +
           path_prefix: String,

     

       231
       231
       +
           downloaded: &'a mut usize,

     

       232
       232
       +
           reused: &'a mut usize,

     

       233
       233
       +
       ) -> std::pin::Pin<Box<dyn std::future::Future<Output = miette::Result<()>> + Send + 'a>> {

     

       234
       234
       +
           Box::pin(async move {

     

       235
       235
       +
           for entry in &dir.entries {

     

       236
       236
       +
               let entry_name = entry.name.as_str();

     

       237
       237
       +
               let current_path = if path_prefix.is_empty() {

     

       238
       238
       +
                   entry_name.to_string()

     

       239
       239
       +
               } else {

     

       240
       240
       +
                   format!("{}/{}", path_prefix, entry_name)

     

       241
       241
       +
               };

     

       242
       242
       +
       

     

       243
       243
       +
               match &entry.node {

     

       244
       244
       +
                   EntryNode::File(file) => {

     

       245
       245
       +
                       let output_path = output_dir.join(entry_name);

     

       246
       246
       +
       

     

       247
       247
       +
                       // Check if file CID matches existing

     

       248
       248
       +
                       if let Some((_blob_ref, new_cid)) = new_blob_map.get(&current_path) {

     

       249
       249
       +
                           if let Some(existing_cid) = existing_file_cids.get(&current_path) {

     

       250
       250
       +
                               if existing_cid == new_cid {

     

       251
       251
       +
                                   // File unchanged, copy from existing directory

     

       252
       252
       +
                                   let existing_path = existing_output_dir.join(&current_path);

     

       253
       253
       +
                                   if existing_path.exists() {

     

       254
       254
       +
                                       std::fs::copy(&existing_path, &output_path).into_diagnostic()?;

     

       255
       255
       +
                                       *reused += 1;

     

       256
       256
       +
                                       println!("  ✓ Reused {}", current_path);

     

       257
       257
       +
                                       continue;

     

       258
       258
       +
                                   }

     

       259
       259
       +
                               }

     

       260
       260
       +
                           }

     

       261
       261
       +
                       }

     

       262
       262
       +
       

     

       263
       263
       +
                       // File is new or changed, download it

     

       264
       264
       +
                       println!("  ↓ Downloading {}", current_path);

     

       265
       265
       +
                       let data = download::download_and_decompress_blob(

     

       266
       266
       +
                           pds_url,

     

       267
       267
       +
                           &file.blob,

     

       268
       268
       +
                           did,

     

       269
       269
       +
                           file.base64.unwrap_or(false),

     

       270
       270
       +
                           file.encoding.as_ref().map(|e| e.as_str() == "gzip").unwrap_or(false),

     

       271
       271
       +
                       )

     

       272
       272
       +
                       .await?;

     

       273
       273
       +
       

     

       274
       274
       +
                       std::fs::write(&output_path, data).into_diagnostic()?;

     

       275
       275
       +
                       *downloaded += 1;

     

       276
       276
       +
                   }

     

       277
       277
       +
                   EntryNode::Directory(subdir) => {

     

       278
       278
       +
                       let subdir_path = output_dir.join(entry_name);

     

       279
       279
       +
                       std::fs::create_dir_all(&subdir_path).into_diagnostic()?;

     

       280
       280
       +
       

     

       281
       281
       +
                       download_directory(

     

       282
       282
       +
                           subdir,

     

       283
       283
       +
                           &subdir_path,

     

       284
       284
       +
                           pds_url,

     

       285
       285
       +
                           did,

     

       286
       286
       +
                           new_blob_map,

     

       287
       287
       +
                           existing_file_cids,

     

       288
       288
       +
                           existing_output_dir,

     

       289
       289
       +
                           current_path,

     

       290
       290
       +
                           downloaded,

     

       291
       291
       +
                           reused,

     

       292
       292
       +
                       )

     

       293
       293
       +
                       .await?;

     

       294
       294
       +
                   }

     

       295
       295
       +
                   EntryNode::Unknown(_) => {

     

       296
       296
       +
                       // Skip unknown node types

     

       297
       297
       +
                       println!("  ⚠ Skipping unknown node type for {}", current_path);

     

       298
       298
       +
                   }

     

       299
       299
       +
               }

     

       300
       300
       +
           }

     

       301
       301
       +
       

     

       302
       302
       +
           Ok(())

     

       303
       303
       +
           })

     

       304
       304
       +
       }

     

       305
       305
       +

+202

cli/src/serve.rs

···

       1
       1
       +
       use crate::pull::pull_site;

     

       2
       2
       +
       use axum::Router;

     

       3
       3
       +
       use jacquard::CowStr;

     

       4
       4
       +
       use jacquard_common::jetstream::{CommitOperation, JetstreamMessage, JetstreamParams};

     

       5
       5
       +
       use jacquard_common::types::string::Did;

     

       6
       6
       +
       use jacquard_common::xrpc::{SubscriptionClient, TungsteniteSubscriptionClient};

     

       7
       7
       +
       use miette::IntoDiagnostic;

     

       8
       8
       +
       use n0_future::StreamExt;

     

       9
       9
       +
       use std::path::PathBuf;

     

       10
       10
       +
       use std::sync::Arc;

     

       11
       11
       +
       use tokio::sync::RwLock;

     

       12
       12
       +
       use tower_http::compression::CompressionLayer;

     

       13
       13
       +
       use tower_http::services::ServeDir;

     

       14
       14
       +
       use url::Url;

     

       15
       15
       +
       

     

       16
       16
       +
       /// Shared state for the server

     

       17
       17
       +
       #[derive(Clone)]

     

       18
       18
       +
       struct ServerState {

     

       19
       19
       +
           did: CowStr<'static>,

     

       20
       20
       +
           rkey: CowStr<'static>,

     

       21
       21
       +
           output_dir: PathBuf,

     

       22
       22
       +
           last_cid: Arc<RwLock<Option<String>>>,

     

       23
       23
       +
       }

     

       24
       24
       +
       

     

       25
       25
       +
       /// Serve a site locally with real-time firehose updates

     

       26
       26
       +
       pub async fn serve_site(

     

       27
       27
       +
           input: CowStr<'static>,

     

       28
       28
       +
           rkey: CowStr<'static>,

     

       29
       29
       +
           output_dir: PathBuf,

     

       30
       30
       +
           port: u16,

     

       31
       31
       +
       ) -> miette::Result<()> {

     

       32
       32
       +
           println!("Serving site {} from {} on port {}...", rkey, input, port);

     

       33
       33
       +
       

     

       34
       34
       +
           // Resolve handle to DID if needed

     

       35
       35
       +
           use jacquard_identity::PublicResolver;

     

       36
       36
       +
           use jacquard::prelude::IdentityResolver;

     

       37
       37
       +
           

     

       38
       38
       +
           let resolver = PublicResolver::default();

     

       39
       39
       +
           let did = if input.starts_with("did:") {

     

       40
       40
       +
               Did::new(&input).into_diagnostic()?

     

       41
       41
       +
           } else {

     

       42
       42
       +
               // It's a handle, resolve it

     

       43
       43
       +
               let handle = jacquard_common::types::string::Handle::new(&input).into_diagnostic()?;

     

       44
       44
       +
               resolver.resolve_handle(&handle).await.into_diagnostic()?

     

       45
       45
       +
           };

     

       46
       46
       +
           

     

       47
       47
       +
           println!("Resolved to DID: {}", did.as_str());

     

       48
       48
       +
       

     

       49
       49
       +
           // Create output directory if it doesn't exist

     

       50
       50
       +
           std::fs::create_dir_all(&output_dir).into_diagnostic()?;

     

       51
       51
       +
       

     

       52
       52
       +
           // Initial pull of the site

     

       53
       53
       +
           println!("Performing initial pull...");

     

       54
       54
       +
           let did_str = CowStr::from(did.as_str().to_string());

     

       55
       55
       +
           pull_site(did_str.clone(), rkey.clone(), output_dir.clone()).await?;

     

       56
       56
       +
       

     

       57
       57
       +
           // Create shared state

     

       58
       58
       +
           let state = ServerState {

     

       59
       59
       +
               did: did_str.clone(),

     

       60
       60
       +
               rkey: rkey.clone(),

     

       61
       61
       +
               output_dir: output_dir.clone(),

     

       62
       62
       +
               last_cid: Arc::new(RwLock::new(None)),

     

       63
       63
       +
           };

     

       64
       64
       +
       

     

       65
       65
       +
           // Start firehose listener in background

     

       66
       66
       +
           let firehose_state = state.clone();

     

       67
       67
       +
           tokio::spawn(async move {

     

       68
       68
       +
               if let Err(e) = watch_firehose(firehose_state).await {

     

       69
       69
       +
                   eprintln!("Firehose error: {}", e);

     

       70
       70
       +
               }

     

       71
       71
       +
           });

     

       72
       72
       +
       

     

       73
       73
       +
           // Create HTTP server with gzip compression

     

       74
       74
       +
           let app = Router::new()

     

       75
       75
       +
               .fallback_service(

     

       76
       76
       +
                   ServeDir::new(&output_dir)

     

       77
       77
       +
                       .precompressed_gzip()

     

       78
       78
       +
               )

     

       79
       79
       +
               .layer(CompressionLayer::new())

     

       80
       80
       +
               .with_state(state);

     

       81
       81
       +
       

     

       82
       82
       +
           let addr = format!("0.0.0.0:{}", port);

     

       83
       83
       +
           let listener = tokio::net::TcpListener::bind(&addr)

     

       84
       84
       +
               .await

     

       85
       85
       +
               .into_diagnostic()?;

     

       86
       86
       +
       

     

       87
       87
       +
           println!("\n✓ Server running at http://localhost:{}", port);

     

       88
       88
       +
           println!("  Watching for updates on the firehose...\n");

     

       89
       89
       +
       

     

       90
       90
       +
           axum::serve(listener, app).await.into_diagnostic()?;

     

       91
       91
       +
       

     

       92
       92
       +
           Ok(())

     

       93
       93
       +
       }

     

       94
       94
       +
       

     

       95
       95
       +
       /// Watch the firehose for updates to the specific site

     

       96
       96
       +
       fn watch_firehose(state: ServerState) -> std::pin::Pin<Box<dyn std::future::Future<Output = miette::Result<()>> + Send>> {

     

       97
       97
       +
           Box::pin(async move {

     

       98
       98
       +
           let jetstream_url = Url::parse("wss://jetstream1.us-east.fire.hose.cam")

     

       99
       99
       +
               .into_diagnostic()?;

     

       100
       100
       +
       

     

       101
       101
       +
           println!("[Firehose] Connecting to Jetstream...");

     

       102
       102
       +
       

     

       103
       103
       +
           // Create subscription client

     

       104
       104
       +
           let client = TungsteniteSubscriptionClient::from_base_uri(jetstream_url);

     

       105
       105
       +
       

     

       106
       106
       +
           // Subscribe with no filters (we'll filter manually)

     

       107
       107
       +
           // Jetstream doesn't support filtering by collection in the params builder

     

       108
       108
       +
           let params = JetstreamParams::new().build();

     

       109
       109
       +
       

     

       110
       110
       +
           let stream = client.subscribe(&params).await.into_diagnostic()?;

     

       111
       111
       +
           println!("[Firehose] Connected! Watching for updates...");

     

       112
       112
       +
       

     

       113
       113
       +
           // Convert to typed message stream

     

       114
       114
       +
           let (_sink, mut messages) = stream.into_stream();

     

       115
       115
       +
       

     

       116
       116
       +
           loop {

     

       117
       117
       +
               match messages.next().await {

     

       118
       118
       +
                   Some(Ok(msg)) => {

     

       119
       119
       +
                       if let Err(e) = handle_firehose_message(&state, msg).await {

     

       120
       120
       +
                           eprintln!("[Firehose] Error handling message: {}", e);

     

       121
       121
       +
                       }

     

       122
       122
       +
                   }

     

       123
       123
       +
                   Some(Err(e)) => {

     

       124
       124
       +
                       eprintln!("[Firehose] Stream error: {}", e);

     

       125
       125
       +
                       // Try to reconnect after a delay

     

       126
       126
       +
                       tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;

     

       127
       127
       +
                       return Box::pin(watch_firehose(state)).await;

     

       128
       128
       +
                   }

     

       129
       129
       +
                   None => {

     

       130
       130
       +
                       println!("[Firehose] Stream ended, reconnecting...");

     

       131
       131
       +
                       tokio::time::sleep(tokio::time::Duration::from_secs(5)).await;

     

       132
       132
       +
                       return Box::pin(watch_firehose(state)).await;

     

       133
       133
       +
                   }

     

       134
       134
       +
               }

     

       135
       135
       +
           }

     

       136
       136
       +
           })

     

       137
       137
       +
       }

     

       138
       138
       +
       

     

       139
       139
       +
       /// Handle a firehose message

     

       140
       140
       +
       async fn handle_firehose_message(

     

       141
       141
       +
           state: &ServerState,

     

       142
       142
       +
           msg: JetstreamMessage<'_>,

     

       143
       143
       +
       ) -> miette::Result<()> {

     

       144
       144
       +
           match msg {

     

       145
       145
       +
               JetstreamMessage::Commit {

     

       146
       146
       +
                   did,

     

       147
       147
       +
                   commit,

     

       148
       148
       +
                   ..

     

       149
       149
       +
               } => {

     

       150
       150
       +
                   // Check if this is our site

     

       151
       151
       +
                   if did.as_str() == state.did.as_str()

     

       152
       152
       +
                       && commit.collection.as_str() == "place.wisp.fs"

     

       153
       153
       +
                       && commit.rkey.as_str() == state.rkey.as_str()

     

       154
       154
       +
                   {

     

       155
       155
       +
                       match commit.operation {

     

       156
       156
       +
                           CommitOperation::Create | CommitOperation::Update => {

     

       157
       157
       +
                               let new_cid = commit.cid.as_ref().map(|c| c.to_string());

     

       158
       158
       +
                               

     

       159
       159
       +
                               // Check if CID changed

     

       160
       160
       +
                               let should_update = {

     

       161
       161
       +
                                   let last_cid = state.last_cid.read().await;

     

       162
       162
       +
                                   new_cid != *last_cid

     

       163
       163
       +
                               };

     

       164
       164
       +
       

     

       165
       165
       +
                               if should_update {

     

       166
       166
       +
                                   println!("\n[Update] Detected change to site {} (CID: {:?})", state.rkey, new_cid);

     

       167
       167
       +
                                   println!("[Update] Pulling latest version...");

     

       168
       168
       +
       

     

       169
       169
       +
                                   // Pull the updated site

     

       170
       170
       +
                                   match pull_site(

     

       171
       171
       +
                                       state.did.clone(),

     

       172
       172
       +
                                       state.rkey.clone(),

     

       173
       173
       +
                                       state.output_dir.clone(),

     

       174
       174
       +
                                   )

     

       175
       175
       +
                                   .await

     

       176
       176
       +
                                   {

     

       177
       177
       +
                                       Ok(_) => {

     

       178
       178
       +
                                           // Update last CID

     

       179
       179
       +
                                           let mut last_cid = state.last_cid.write().await;

     

       180
       180
       +
                                           *last_cid = new_cid;

     

       181
       181
       +
                                           println!("[Update] ✓ Site updated successfully!\n");

     

       182
       182
       +
                                       }

     

       183
       183
       +
                                       Err(e) => {

     

       184
       184
       +
                                           eprintln!("[Update] Failed to pull site: {}", e);

     

       185
       185
       +
                                       }

     

       186
       186
       +
                                   }

     

       187
       187
       +
                               }

     

       188
       188
       +
                           }

     

       189
       189
       +
                           CommitOperation::Delete => {

     

       190
       190
       +
                               println!("\n[Update] Site {} was deleted", state.rkey);

     

       191
       191
       +
                           }

     

       192
       192
       +
                       }

     

       193
       193
       +
                   }

     

       194
       194
       +
               }

     

       195
       195
       +
               _ => {

     

       196
       196
       +
                   // Ignore identity and account messages

     

       197
       197
       +
               }

     

       198
       198
       +
           }

     

       199
       199
       +
       

     

       200
       200
       +
           Ok(())

     

       201
       201
       +
       }

     

       202
       202
       +

-3

.gitmodules

···

       1
       1
       -
       [submodule "cli/jacquard"]

     

       2
       2
       -
       	path = cli/jacquard

     

       3
       3
       -
       	url = https://tangled.org/@nonbinary.computer/jacquard

-1

cli/jacquard

···

       1
       1
       -
       Subproject commit d533482a61f540586b1eea620b8e9a01a59d5650

+1 -1

cli/Cargo.toml

···

       1
       1
        
       [package]

     

       2
       2
        
       name = "wisp-cli"

     

       3
       3
       -
       version = "0.1.0"

     

       3
       3
       +
       version = "0.2.0"

     

       4
       4
        
       edition = "2024"

     

       5
       5
        
       

     

       6
       6
        
       [features]

+28 -1

crates.nix

···

       19
       19
        
                 targets.x86_64-pc-windows-gnu.latest.rust-std

     

       20
       20
        
                 targets.x86_64-unknown-linux-gnu.latest.rust-std

     

       21
       21
        
                 targets.aarch64-apple-darwin.latest.rust-std

     

       22
       22
       +
                 targets.aarch64-unknown-linux-gnu.latest.rust-std

     

       22
       23
        
               ];

     

       23
       24
        
           # configure crates

     

       24
       25
        
           nci.crates."wisp-cli" = {

     
···

       26
       27
        
               dev.runTests = false;

     

       27
       28
        
               release.runTests = false;

     

       28
       29
        
             };

     

       29
       29
       -
             targets."x86_64-unknown-linux-gnu" = {

     

       30
       30
       +
             targets."x86_64-unknown-linux-gnu" = let

     

       31
       31
       +
               targetPkgs = pkgs.pkgsCross.gnu64;

     

       32
       32
       +
               targetCC = targetPkgs.stdenv.cc;

     

       33
       33
       +
               targetCargoEnvVarTarget = targetPkgs.stdenv.hostPlatform.rust.cargoEnvVarTarget;

     

       34
       34
       +
             in rec {

     

       30
       35
        
               default = true;

     

       36
       36
       +
               depsDrvConfig.mkDerivation = {

     

       37
       37
       +
                 nativeBuildInputs = [targetCC];

     

       38
       38
       +
               };

     

       39
       39
       +
               depsDrvConfig.env = rec {

     

       40
       40
       +
                 TARGET_CC = "${targetCC.targetPrefix}cc";

     

       41
       41
       +
                 "CARGO_TARGET_${targetCargoEnvVarTarget}_LINKER" = TARGET_CC;

     

       42
       42
       +
               };

     

       43
       43
       +
               drvConfig = depsDrvConfig;

     

       31
       44
        
             };

     

       32
       45
        
             targets."x86_64-pc-windows-gnu" = let

     

       33
       46
        
               targetPkgs = pkgs.pkgsCross.mingwW64;

     
···

       58
       71
        
               };

     

       59
       72
        
               drvConfig = depsDrvConfig;

     

       60
       73
        
             };

     

       74
       74
       +
             targets."aarch64-unknown-linux-gnu" = let

     

       75
       75
       +
               targetPkgs = pkgs.pkgsCross.aarch64-multiplatform;

     

       76
       76
       +
               targetCC = targetPkgs.stdenv.cc;

     

       77
       77
       +
               targetCargoEnvVarTarget = targetPkgs.stdenv.hostPlatform.rust.cargoEnvVarTarget;

     

       78
       78
       +
             in rec {

     

       79
       79
       +
               depsDrvConfig.mkDerivation = {

     

       80
       80
       +
                 nativeBuildInputs = [targetCC];

     

       81
       81
       +
               };

     

       82
       82
       +
               depsDrvConfig.env = rec {

     

       83
       83
       +
                 TARGET_CC = "${targetCC.targetPrefix}cc";

     

       84
       84
       +
                 "CARGO_TARGET_${targetCargoEnvVarTarget}_LINKER" = TARGET_CC;

     

       85
       85
       +
               };

     

       86
       86
       +
               drvConfig = depsDrvConfig;

     

       87
       87
       +
             };

     

       61
       88
        
           };

     

       62
       89
        
         };

     

       63
       90
        
       }

+17 -2

flake.nix

···

       26
       26
        
               ...

     

       27
       27
        
             }: let

     

       28
       28
        
               crateOutputs = config.nci.outputs."wisp-cli";

     

       29
       29
       +
               mkRenamedPackage = name: pkg: pkgs.runCommand name {} ''

     

       30
       30
       +
                 mkdir -p $out/bin

     

       31
       31
       +
                 cp ${pkg}/bin/wisp-cli $out/bin/${name}

     

       32
       32
       +
               '';

     

       29
       33
        
             in {

     

       30
       34
        
               devShells.default = crateOutputs.devShell;

     

       31
       35
        
               packages.default = crateOutputs.packages.release;

     

       32
       32
       -
               packages.wisp-cli-windows = crateOutputs.allTargets."x86_64-pc-windows-gnu".packages.release;

     

       33
       33
       -
               packages.wisp-cli-darwin = crateOutputs.allTargets."aarch64-apple-darwin".packages.release;

     

       36
       36
       +
               packages.wisp-cli-x86_64-linux = mkRenamedPackage "wisp-cli-x86_64-linux" crateOutputs.packages.release;

     

       37
       37
       +
               packages.wisp-cli-aarch64-linux = mkRenamedPackage "wisp-cli-aarch64-linux" crateOutputs.allTargets."aarch64-unknown-linux-gnu".packages.release;

     

       38
       38
       +
               packages.wisp-cli-x86_64-windows = mkRenamedPackage "wisp-cli-x86_64-windows.exe" crateOutputs.allTargets."x86_64-pc-windows-gnu".packages.release;

     

       39
       39
       +
               packages.wisp-cli-aarch64-darwin = mkRenamedPackage "wisp-cli-aarch64-darwin" crateOutputs.allTargets."aarch64-apple-darwin".packages.release;

     

       40
       40
       +
               packages.all = pkgs.symlinkJoin {

     

       41
       41
       +
                 name = "wisp-cli-all";

     

       42
       42
       +
                 paths = [

     

       43
       43
       +
                   config.packages.wisp-cli-x86_64-linux

     

       44
       44
       +
                   config.packages.wisp-cli-aarch64-linux

     

       45
       45
       +
                   config.packages.wisp-cli-x86_64-windows

     

       46
       46
       +
                   config.packages.wisp-cli-aarch64-darwin

     

       47
       47
       +
                 ];

     

       48
       48
       +
               };

     

       34
       49
        
             };

     

       35
       50
        
           };

     

       36
       51
        
       }