back round 0 view raw

docs/spindle: arch, hosting and pipeline spec #264

merged

opened by

anirudh.fi 5 months ago targeting master from push-ytzxyplmvunn

Signed-off-by: Anirudh Oppiliappan anirudh@tangled.sh

options

unified split

Changed files

+126

docs

spindle

architecture.md

hosting.md

pipeline.md

+24

docs/spindle/architecture.md

···

       1
       1
       +
       # spindle architecture

     

       2
       2
       +
       

     

       3
       3
       +
       Spindle is a small CI runner service. Here's a high level overview of how it operates:

     

       4
       4
       +
       

     

       5
       5
       +
       * listens for [`sh.tangled.spindle.member`](/lexicons/spindle/member.json) and

     

       6
       6
       +
       [`sh.tangled.repo`](/lexicons/repo.json) records on the Jetstream.

     

       7
       7
       +
       * when a new repo record comes through (typically when you add a spindle to a

     

       8
       8
       +
       repo from the settings), spindle then resolves the underlying knot and

     

       9
       9
       +
       subscribes to repo events (see:

     

       10
       10
       +
       [`sh.tangled.pipeline`](/lexicons/pipeline.json)).

     

       11
       11
       +
       * the spindle engine then handles execution of the pipeline, with results and

     

       12
       12
       +
       logs beamed on the spindle event stream over wss

     

       13
       13
       +
       

     

       14
       14
       +
       ### the engine

     

       15
       15
       +
       

     

       16
       16
       +
       At present, the only supported backend is Docker. Spindle executes each step in

     

       17
       17
       +
       the pipeline in a fresh container, with state persisted across steps within the

     

       18
       18
       +
       `/tangled/workspace` directory.

     

       19
       19
       +
       

     

       20
       20
       +
       The base image for the container is constructed on the fly using

     

       21
       21
       +
       [Nixery](https://nixery.dev), which is handy for caching layers for frequently

     

       22
       22
       +
       used packages.

     

       23
       23
       +
       

     

       24
       24
       +
       The pipeline manifest is [specified here](/docs/spindle/pipeline.md).

+43

docs/spindle/hosting.md

···

       1
       1
       +
       # spindle self-hosting guide

     

       2
       2
       +
       

     

       3
       3
       +
       ## prerequisites

     

       4
       4
       +
       

     

       5
       5
       +
       * Go

     

       6
       6
       +
       * Docker (the only supported backend currently)

     

       7
       7
       +
       

     

       8
       8
       +
       ## configuration

     

       9
       9
       +
       

     

       10
       10
       +
       Spindle is configured using environment variables. The following environment variables are available:

     

       11
       11
       +
       

     

       12
       12
       +
       * `SPINDLE_SERVER_LISTEN_ADDR`: The address the server listens on (default: `"0.0.0.0:6555"`).

     

       13
       13
       +
       * `SPINDLE_SERVER_DB_PATH`: The path to the SQLite database file (default: `"spindle.db"`).

     

       14
       14
       +
       * `SPINDLE_SERVER_HOSTNAME`: The hostname of the server (required).

     

       15
       15
       +
       * `SPINDLE_SERVER_JETSTREAM_ENDPOINT`: The endpoint of the Jetstream server (default: `"wss://jetstream1.us-west.bsky.network/subscribe"`).

     

       16
       16
       +
       * `SPINDLE_SERVER_DEV`: A boolean indicating whether the server is running in development mode (default: `false`).

     

       17
       17
       +
       * `SPINDLE_SERVER_OWNER`: The DID of the owner (required).

     

       18
       18
       +
       * `SPINDLE_PIPELINES_NIXERY`: The Nixery URL (default: `"nixery.tangled.sh"`).

     

       19
       19
       +
       * `SPINDLE_PIPELINES_WORKFLOW_TIMEOUT`: The default workflow timeout (default: `"5m"`).

     

       20
       20
       +
       * `SPINDLE_PIPELINES_LOG_DIR`: The directory to store workflow logs (default: `"/var/log/spindle"`).

     

       21
       21
       +
       

     

       22
       22
       +
       ## running spindle

     

       23
       23
       +
       

     

       24
       24
       +
       1.  **Set the environment variables.**  For example:

     

       25
       25
       +
       

     

       26
       26
       +
           ```shell

     

       27
       27
       +
           export SPINDLE_SERVER_HOSTNAME="your-hostname"

     

       28
       28
       +
           export SPINDLE_SERVER_OWNER="your-did"

     

       29
       29
       +
           ```

     

       30
       30
       +
       

     

       31
       31
       +
       2.  **Build the Spindle binary.**

     

       32
       32
       +
       

     

       33
       33
       +
           ```shell

     

       34
       34
       +
           go build -o spindle core/spindle/server.go

     

       35
       35
       +
           ```

     

       36
       36
       +
       

     

       37
       37
       +
       3.  **Run the Spindle binary.**

     

       38
       38
       +
       

     

       39
       39
       +
           ```shell

     

       40
       40
       +
           ./spindle

     

       41
       41
       +
           ```

     

       42
       42
       +
       

     

       43
       43
       +
       Spindle will now start, connect to the Jetstream server, and begin processing pipelines.

+59

docs/spindle/pipeline.md

···

       1
       1
       +
       # spindle pipeline manifest

     

       2
       2
       +
       

     

       3
       3
       +
       Spindle pipelines are defined under the `.tangled/workflows` directory in a

     

       4
       4
       +
       repo. Generally:

     

       5
       5
       +
       

     

       6
       6
       +
       * Pipelines are defined in YAML.

     

       7
       7
       +
       * Dependencies can be specified from

     

       8
       8
       +
       [Nixpkgs](https://search.nixos.org) or custom registries.

     

       9
       9
       +
       * Environment variables can be set globally or per-step.

     

       10
       10
       +
       

     

       11
       11
       +
       Here's an example that uses all fields:

     

       12
       12
       +
       

     

       13
       13
       +
       ```yaml

     

       14
       14
       +
       # build_and_test.yaml

     

       15
       15
       +
       when:

     

       16
       16
       +
         - event: ["push", "pull_request"]

     

       17
       17
       +
           branch: ["main", "develop"]

     

       18
       18
       +
         - event: ["manual"]

     

       19
       19
       +
       

     

       20
       20
       +
       dependencies:

     

       21
       21
       +
         ## from nixpkgs

     

       22
       22
       +
         nixpkgs:

     

       23
       23
       +
           - nodejs

     

       24
       24
       +
         ## custom registry

     

       25
       25
       +
         git+https://tangled.sh/@oppi.li/statix:

     

       26
       26
       +
           - statix

     

       27
       27
       +
       

     

       28
       28
       +
       steps:

     

       29
       29
       +
         - name: "Install dependencies"

     

       30
       30
       +
           command: "npm install"

     

       31
       31
       +
           environment:

     

       32
       32
       +
             NODE_ENV: "development"

     

       33
       33
       +
             CI: "true"

     

       34
       34
       +
       

     

       35
       35
       +
         - name: "Run linter"

     

       36
       36
       +
           command: "npm run lint"

     

       37
       37
       +
       

     

       38
       38
       +
         - name: "Run tests"

     

       39
       39
       +
           command: "npm test"

     

       40
       40
       +
           environment:

     

       41
       41
       +
             NODE_ENV: "test"

     

       42
       42
       +
             JEST_WORKERS: "2"

     

       43
       43
       +
       

     

       44
       44
       +
         - name: "Build application"

     

       45
       45
       +
           command: "npm run build"

     

       46
       46
       +
           environment:

     

       47
       47
       +
             NODE_ENV: "production"

     

       48
       48
       +
       

     

       49
       49
       +
       environment:

     

       50
       50
       +
         BUILD_NUMBER: "123"

     

       51
       51
       +
         GIT_BRANCH: "main"

     

       52
       52
       +
       

     

       53
       53
       +
       ## current repository is cloned and checked out at the target ref

     

       54
       54
       +
       ## by default.

     

       55
       55
       +
       clone:

     

       56
       56
       +
         skip: false

     

       57
       57
       +
         depth: 50

     

       58
       58
       +
         submodules: true

     

       59
       59
       +
       ```