commit 99648914400685392377abc98f29a0a90a9b48c2 · pyrox.dev/nixpkgs

+43

nixos/doc/manual/development/developing-the-test-driver.chapter.md

···

       1
       1
       +
       

     

       2
       2
       +
       # Developing the NixOS Test Driver {#chap-developing-the-test-driver}

     

       3
       3
       +
       

     

       4
       4
       +
       The NixOS test framework is a project of its own.

     

       5
       5
       +
       

     

       6
       6
       +
       It consists of roughly the following components:

     

       7
       7
       +
       

     

       8
       8
       +
        - `nixos/lib/test-driver`: The Python framework that sets up the test and runs the [`testScript`](#test-opt-testScript)

     

       9
       9
       +
        - `nixos/lib/testing`: The Nix code responsible for the wiring, written using the (NixOS) Module System.

     

       10
       10
       +
       

     

       11
       11
       +
       These components are exposed publicly through:

     

       12
       12
       +
       

     

       13
       13
       +
        - `nixos/lib/default.nix`: The public interface that exposes the `nixos/lib/testing` entrypoint.

     

       14
       14
       +
        - `flake.nix`: Exposes the `lib.nixos`, including the public test interface.

     

       15
       15
       +
       

     

       16
       16
       +
       Beyond the test driver itself, its integration into NixOS and Nixpkgs is important.

     

       17
       17
       +
       

     

       18
       18
       +
        - `pkgs/top-level/all-packages.nix`: Defines the `nixosTests` attribute, used

     

       19
       19
       +
          by the package `tests` attributes and OfBorg.

     

       20
       20
       +
        - `nixos/release.nix`: Defines the `tests` attribute built by Hydra, independently, but analogous to `nixosTests`

     

       21
       21
       +
        - `nixos/release-combined.nix`: Defines which tests are channel blockers.

     

       22
       22
       +
       

     

       23
       23
       +
       Finally, we have legacy entrypoints that users should move away from, but are cared for on a best effort basis.

     

       24
       24
       +
       These include `pkgs.nixosTest`, `testing-python.nix` and `make-test-python.nix`.

     

       25
       25
       +
       

     

       26
       26
       +
       ## Testing changes to the test framework {#sec-test-the-test-framework}

     

       27
       27
       +
       

     

       28
       28
       +
       When making significant changes to the test framework, we run the tests on Hydra, to avoid disrupting the larger NixOS project.

     

       29
       29
       +
       

     

       30
       30
       +
       For this, we use the `python-test-refactoring` branch in the `NixOS/nixpkgs` repository, and its [corresponding Hydra jobset](https://hydra.nixos.org/jobset/nixos/python-test-refactoring).

     

       31
       31
       +
       This branch is used as a pointer, and not as a feature branch.

     

       32
       32
       +
       

     

       33
       33
       +
       1. Rebase the PR onto a recent, good evaluation of `nixos-unstable`

     

       34
       34
       +
       2. Create a baseline evaluation by force-pushing this revision of `nixos-unstable` to `python-test-refactoring`.

     

       35
       35
       +
       3. Note the evaluation number (we'll call it `<previous>`)

     

       36
       36
       +
       4. Push the PR to `python-test-refactoring` and evaluate the PR on Hydra

     

       37
       37
       +
       5. Create a comparison URL by navigating to the latest build of the PR and adding to the URL `?compare=<previous>`. This is not necessary for the evaluation that comes right after the baseline.

     

       38
       38
       +
       

     

       39
       39
       +
       Review the removed tests and newly failed tests using the constructed URL; otherwise you will accidentally compare iterations of the PR instead of changes to the PR base.

     

       40
       40
       +
       

     

       41
       41
       +
       As we currently have some flaky tests, newly failing tests are expected, but should be reviewed to make sure that

     

       42
       42
       +
        - The number of failures did not increase significantly.

     

       43
       43
       +
        - All failures that do occur can reasonably be assumed to fail for a different reason than the changes.

nixos/doc/manual/development/development.md

···

       10
       10
        
       what-happens-during-a-system-switch.chapter.md

     

       11
       11
        
       writing-documentation.chapter.md

     

       12
       12
        
       nixos-tests.chapter.md

     

       13
       13
       +
       developing-the-test-driver.chapter.md

     

       13
       14
        
       testing-installer.chapter.md

     

       14
       15
        
       ```