commit 45cd444e2d42524fbe24f4146701450bac9b9d21 · kot.pink/core

+130 -54
docs/spindle/pipeline.md
···

       1
       1
       -
       # spindle pipeline manifest

     

       1
       1
       +
       # spindle pipelines

     

       2
       2
       +
       

     

       3
       3
       +
       Spindle workflows allow you to write CI/CD pipelines in a simple format. They're located in the `.tangled/workflows` directory at the root of your repository, and are defined using YAML.

     

       4
       4
       +
       

     

       5
       5
       +
       The fields are:

     

       2
       6
        
       

     

       3
       3
       -
       Spindle pipelines are defined under the `.tangled/workflows` directory in a

     

       4
       4
       -
       repo. Generally:

     

       7
       7
       +
       - [Trigger](#trigger): A **required** field that defines when a workflow should be triggered.

     

       8
       8
       +
       - [Engine](#engine): A **required** field that defines which engine a workflow should run on.

     

       9
       9
       +
       - [Clone options](#clone-options): An **optional** field that defines how the repository should be cloned.

     

       10
       10
       +
       - [Dependencies](#dependencies): An **optional** field that allows you to list dependencies you may need.

     

       11
       11
       +
       - [Environment](#environment): An **optional** field that allows you to define environment variables.

     

       12
       12
       +
       - [Steps](#steps): An **optional** field that allows you to define what steps should run in the workflow.

     

       5
       13
        
       

     

       6
       6
       -
       * Pipelines are defined in YAML.

     

       7
       7
       -
       * Workflows can run using different *engines*.

     

       14
       14
       +
       ## Trigger

     

       8
       15
        
       

     

       9
       9
       -
       The most barebones workflow looks like this:

     

       16
       16
       +
       The first thing to add to a workflow is the trigger, which defines when a workflow runs. This is defined using a `when` field, which takes in a list of conditions. Each condition has the following fields:

     

       17
       17
       +
       

     

       18
       18
       +
       - `event`: This is a **required** field that defines when your workflow should run. It's a list that can take one or more of the following values:

     

       19
       19
       +
         - `push`: The workflow should run every time a commit is pushed to the repository.

     

       20
       20
       +
         - `pull_request`: The workflow should run every time a pull request is made or updated.

     

       21
       21
       +
         - `manual`: The workflow can be triggered manually.

     

       22
       22
       +
       - `branch`: This is a **required** field that defines which branches the workflow should run for. If used with the `push` event, commits to the branch(es) listed here will trigger the workflow. If used with the `pull_request` event, updates to pull requests targeting the branch(es) listed here will trigger the workflow. This field has no effect with the `manual` event.

     

       23
       23
       +
       

     

       24
       24
       +
       For example, if you'd like define a workflow that runs when commits are pushed to the `main` and `develop` branches, or when pull requests that target the `main` branch are updated, or manually, you can do so with:

     

       10
       25
        
       

     

       11
       26
        
       ```yaml

     

       12
       27
        
       when:

     

       13
       13
       -
         - event: ["push"]

     

       28
       28
       +
         - event: ["push", "manual"]

     

       29
       29
       +
           branch: ["main", "develop"]

     

       30
       30
       +
         - event: ["pull_request"]

     

       14
       31
        
           branch: ["main"]

     

       32
       32
       +
       ```

     

       15
       33
        
       

     

       34
       34
       +
       ## Engine

     

       35
       35
       +
       

     

       36
       36
       +
       Next is the engine on which the workflow should run, defined using the **required** `engine` field. The currently supported engines are:

     

       37
       37
       +
       

     

       38
       38
       +
       - `nixery`: This uses an instance of [Nixery](https://nixery.dev) to run steps, which allows you to add [dependencies](#dependencies) from [Nixpkgs](https://github.com/NixOS/nixpkgs). You can search for packages on https://search.nixos.org, and there's a pretty good chance the package(s) you're looking for will be there.

     

       39
       39
       +
       

     

       40
       40
       +
       Example:

     

       41
       41
       +
       

     

       42
       42
       +
       ```yaml

     

       16
       43
        
       engine: "nixery"

     

       44
       44
       +
       ```

     

       45
       45
       +
       

     

       46
       46
       +
       ## Clone options

     

       17
       47
        
       

     

       18
       18
       -
       # optional

     

       48
       48
       +
       When a workflow starts, the first step is to clone the repository. You can customize this behavior using the **optional** `clone` field. It has the following fields:

     

       49
       49
       +
       

     

       50
       50
       +
       - `skip`: Setting this to `true` will skip cloning the repository. This can be useful if your workflow is doing something that doesn't require anything from the repository itself. This is `false` by default.

     

       51
       51
       +
       - `depth`: This sets the number of commits, or the "clone depth", to fetch from the repository. For example, if you set this to 2, the last 2 commits will be fetched. By default, the depth is set to 1, meaning only the most recent commit will be fetched, which is the commit that triggered the workflow.

     

       52
       52
       +
       - `submodules`: If you use [git submodules](https://git-scm.com/book/en/v2/Git-Tools-Submodules) in your repository, setting this field to `true` will recursively fetch all submodules. This is `false` by default.

     

       53
       53
       +
       

     

       54
       54
       +
       The default settings are:

     

       55
       55
       +
       

     

       56
       56
       +
       ```yaml

     

       19
       57
        
       clone:

     

       20
       58
        
         skip: false

     

       21
       21
       -
         depth: 50

     

       22
       22
       -
         submodules: true

     

       59
       59
       +
         depth: 1

     

       60
       60
       +
         submodules: false

     

       23
       61
        
       ```

     

       24
       62
        
       

     

       25
       25
       -
       The `when` and `engine` fields are required, while every other aspect

     

       26
       26
       -
       of how the definition is parsed is up to the engine. Currently, a spindle

     

       27
       27
       -
       provides at least one of these built-in engines:

     

       63
       63
       +
       ## Dependencies

     

       28
       64
        
       

     

       29
       29
       -
       ## `nixery`

     

       65
       65
       +
       Usually when you're running a workflow, you'll need additional dependencies. The `dependencies` field lets you define which dependencies to get, and from where. It's a key-value map, with the key being the registry to fetch dependencies from, and the value being the list of dependencies to fetch.

     

       30
       66
        
       

     

       31
       31
       -
       The Nixery engine uses an instance of [Nixery](https://nixery.dev) to run

     

       32
       32
       -
       steps that use dependencies from [Nixpkgs](https://github.com/NixOS/nixpkgs).

     

       33
       33
       -
       

     

       34
       34
       -
       Here's an example that uses all fields:

     

       67
       67
       +
       Say you want to fetch Node.js and Go from `nixpkgs`, and a package called `my_pkg` you've made from your own registry at your repository at `https://tangled.sh/@example.com/my_pkg`. You can define those dependencies like so:

     

       35
       68
        
       

     

       36
       69
        
       ```yaml

     

       37
       37
       -
       # build_and_test.yaml

     

       38
       38
       -
       when:

     

       39
       39
       -
         - event: ["push", "pull_request"]

     

       40
       40
       -
           branch: ["main", "develop"]

     

       41
       41
       -
         - event: ["manual"]

     

       42
       42
       -
       

     

       43
       70
        
       dependencies:

     

       44
       44
       -
         ## from nixpkgs

     

       71
       71
       +
         # nixpkgs

     

       45
       72
        
         nixpkgs:

     

       46
       73
        
           - nodejs

     

       47
       47
       -
         ## custom registry

     

       48
       48
       -
         git+https://tangled.sh/@oppi.li/statix:

     

       49
       49
       -
           - statix

     

       74
       74
       +
           - go

     

       75
       75
       +
         # custom registry

     

       76
       76
       +
         git+https://tangled.sh/@example.com/my_pkg:

     

       77
       77
       +
           - my_pkg

     

       78
       78
       +
       ```

     

       50
       79
        
       

     

       51
       51
       -
       steps:

     

       52
       52
       -
         - name: "Install dependencies"

     

       53
       53
       -
           command: "npm install"

     

       54
       54
       -
           environment:

     

       55
       55
       -
             NODE_ENV: "development"

     

       56
       56
       -
             CI: "true"

     

       80
       80
       +
       Now these dependencies are available to use in your workflow!

     

       57
       81
        
       

     

       58
       58
       -
         - name: "Run linter"

     

       59
       59
       -
           command: "npm run lint"

     

       82
       82
       +
       ## Environment

     

       83
       83
       +
       

     

       84
       84
       +
       The `environment` field allows you define environment variables that will be available throughout the entire workflow. **Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.**

     

       85
       85
       +
       

     

       86
       86
       +
       Example:

     

       87
       87
       +
       

     

       88
       88
       +
       ```yaml

     

       89
       89
       +
       environment:

     

       90
       90
       +
         GOOS: "linux"

     

       91
       91
       +
         GOARCH: "arm64"

     

       92
       92
       +
         NODE_ENV: "production"

     

       93
       93
       +
         MY_ENV_VAR: "MY_ENV_VALUE"

     

       94
       94
       +
       ```

     

       60
       95
        
       

     

       61
       61
       -
         - name: "Run tests"

     

       62
       62
       -
           command: "npm test"

     

       63
       63
       -
           environment:

     

       64
       64
       -
             NODE_ENV: "test"

     

       65
       65
       -
             JEST_WORKERS: "2"

     

       96
       96
       +
       ## Steps

     

       66
       97
        
       

     

       67
       67
       -
         - name: "Build application"

     

       98
       98
       +
       The `steps` field allows you to define what steps should run in the workflow. It's a list of step objects, each with the following fields:

     

       99
       99
       +
       

     

       100
       100
       +
       - `name`: This field allows you to give your step a name. This name is visible in your workflow runs, and is used to describe what the step is doing.

     

       101
       101
       +
       - `command`: This field allows you to define a command to run in that step. The step is run in a Bash shell, and the logs from the command will be visible in the pipelines page on the Tangled website. The [dependencies](#dependencies) you added will be available to use here.

     

       102
       102
       +
       - `environment`: Similar to the global [environment](#environment) config, this **optional** field is a key-value map that allows you to set environment variables for the step. **Do not put secrets here, these environment variables are visible to anyone viewing the repository. You can add secrets for pipelines in your repository's settings.**

     

       103
       103
       +
       

     

       104
       104
       +
       Example:

     

       105
       105
       +
       

     

       106
       106
       +
       ```yaml

     

       107
       107
       +
       steps:

     

       108
       108
       +
         - name: "Build backend"

     

       109
       109
       +
           command: "go build"

     

       110
       110
       +
           environment:

     

       111
       111
       +
             GOOS: "darwin"

     

       112
       112
       +
             GOARCH: "arm64"

     

       113
       113
       +
         - name: "Build frontend"

     

       68
       114
        
           command: "npm run build"

     

       69
       115
        
           environment:

     

       70
       116
        
             NODE_ENV: "production"

     

       117
       117
       +
       ```

     

       71
       118
        
       

     

       72
       72
       -
       environment:

     

       73
       73
       -
         BUILD_NUMBER: "123"

     

       74
       74
       -
         GIT_BRANCH: "main"

     

       119
       119
       +
       ## Complete workflow

     

       75
       120
        
       

     

       76
       76
       -
       ## current repository is cloned and checked out at the target ref

     

       77
       77
       -
       ## by default.

     

       121
       121
       +
       ```yaml

     

       122
       122
       +
       # .tangled/workflows/build.yml

     

       123
       123
       +
       

     

       124
       124
       +
       when:

     

       125
       125
       +
         - event: ["push", "manual"]

     

       126
       126
       +
           branch: ["main", "develop"]

     

       127
       127
       +
         - event: ["pull_request"]

     

       128
       128
       +
           branch: ["main"]

     

       129
       129
       +
       

     

       130
       130
       +
       engine: "nixery"

     

       131
       131
       +
       

     

       132
       132
       +
       # using the default values

     

       78
       133
        
       clone:

     

       79
       134
        
         skip: false

     

       80
       80
       -
         depth: 50

     

       81
       81
       -
         submodules: true

     

       82
       82
       -
       ```

     

       135
       135
       +
         depth: 1

     

       136
       136
       +
         submodules: false

     

       83
       137
        
       

     

       84
       84
       -
       ## git push options

     

       138
       138
       +
       dependencies:

     

       139
       139
       +
         # nixpkgs

     

       140
       140
       +
         nixpkgs:

     

       141
       141
       +
           - nodejs

     

       142
       142
       +
           - go

     

       143
       143
       +
         # custom registry

     

       144
       144
       +
         git+https://tangled.sh/@example.com/my_pkg:

     

       145
       145
       +
           - my_pkg

     

       85
       146
        
       

     

       86
       86
       -
       These are push options that can be used with the `--push-option (-o)` flag of git push:

     

       147
       147
       +
       environment:

     

       148
       148
       +
         GOOS: "linux"

     

       149
       149
       +
         GOARCH: "arm64"

     

       150
       150
       +
         NODE_ENV: "production"

     

       151
       151
       +
         MY_ENV_VAR: "MY_ENV_VALUE"

     

       152
       152
       +
       

     

       153
       153
       +
       steps:

     

       154
       154
       +
         - name: "Build backend"

     

       155
       155
       +
           command: "go build"

     

       156
       156
       +
           environment:

     

       157
       157
       +
             GOOS: "darwin"

     

       158
       158
       +
             GOARCH: "arm64"

     

       159
       159
       +
         - name: "Build frontend"

     

       160
       160
       +
           command: "npm run build"

     

       161
       161
       +
           environment:

     

       162
       162
       +
             NODE_ENV: "production"

     

       163
       163
       +
       ```

     

       87
       164
        
       

     

       88
       88
       -
       - `verbose-ci`, `ci-verbose`: enables diagnostics reporting for the CI pipeline, allowing you to see any issues when you push.

     

       89
       89
       -
       - `skip-ci`, `ci-skip`: skips triggering the CI pipeline.

     

       165
       165
       +
       If you want another example of a workflow, you can look at the one [Tangled uses to build the project](https://tangled.sh/@tangled.sh/core/blob/master/.tangled/workflows/build.yml).