commit 6db96122047fe7bdc1d58e201c9d161c0c479aba · pyrox.dev/nixpkgs

+1 -1

doc/build-helpers/special.md

···

       7
       7
        
       special/makesetuphook.section.md

     

       8
       8
        
       special/mkshell.section.md

     

       9
       9
        
       special/vm-tools.section.md

     

       10
       10
       -
       special/incremental-build.section.md

     

       10
       10
       +
       special/checkpoint-build.section.md

     

       11
       11
        
       ```

+28

doc/build-helpers/special/checkpoint-build.section.md

···

       1
       1
       +
       # pkgs.checkpointBuildTools.*  {#sec-checkpoint-build}

     

       2
       2
       +
       

     

       3
       3
       +
       `pkgs.checkpointBuildTools` provides a way to build derivations incrementally. It consists of two functions to make checkpoint builds using nix possible.

     

       4
       4
       +
       

     

       5
       5
       +
       For hermeticity, Nix derivations do not allow any state to carry over between builds, making a transparent incremental build within a derivation impossible.

     

       6
       6
       +
       

     

       7
       7
       +
       However, we can tell Nix explicitly what the previous build state was, by representing that previous state as a derivation output. This allows the passed build state to be used for an incremental build.

     

       8
       8
       +
       

     

       9
       9
       +
       To build a derivation based on build checkpoints, the following steps needs to be fullfilled:

     

       10
       10
       +
           * - run prepareCheckpointBuild on the desired derivation

     

       11
       11
       +
           *   e.G `checkpointArtifacts = (pkgs.checkpointBuildTools.prepareCheckpointBuild pkgs.virtualbox);`

     

       12
       12
       +
           * - change something you want in the sources of the package( e.G using source override)

     

       13
       13
       +
           *   changedVBox = pkgs.virtuabox.overrideAttrs (old: {

     

       14
       14
       +
           *      src = path/to/vbox/sources;

     

       15
       15
       +
           *   }

     

       16
       16
       +
           * - use `mkCheckpointedBuild changedVBox buildOutput`

     

       17
       17
       +
           * enjoy shorter build times

     

       18
       18
       +
       

     

       19
       19
       +
       As Nix has no builtin support for the detection of the previous built derivation, a base version needs to be declared.  

     

       20
       20
       +
       To create the outputs later used as base version for checkpoint builds, the function `pkgs.checkpointBuildTools.prepareCheckpointBuild` is used.  

     

       21
       21
       +
       The function takes the original derivation as an argument and transforms the output to a base version for an checkpoint build build.  

     

       22
       22
       +
       While doing so, the original output is not created and the installation phase is overwritten to produce the checkpoint artifacts.  

     

       23
       23
       +
       

     

       24
       24
       +
       When the built artifacts of the base version of the derivation are created, the code can be modified and changes are built using the `pkgs.checkpointBuildTools.mkCheckpointedBuild` function.  

     

       25
       25
       +
       The `pkgs.checkpointBuildTools.mkCheckpointedBuild` function detects the changes in the code and places the output of the base version derivation within the build folder.   

     

       26
       26
       +
       Then, the build tool is able to detect the changes and makes the decision of which parts of the derivation needs to be recompiled and produces the output, as expected in the derivation, without checkpoint build support.

     

       27
       27
       +
       

     

       28
       28
       +

-28

doc/build-helpers/special/incremental-build.section.md

···

       1
       1
       -
       # pkgs.buildIncremental.*  {#sec-incremental-build}

     

       2
       2
       -
       

     

       3
       3
       -
       `pkgs.buildIncremental` provides a way to build derivations incrementally. It consists of two functions to make incremental builds using nix possible.

     

       4
       4
       -
       

     

       5
       5
       -
       For hermeticity, Nix derivations do not allow any state to carry over between builds, making a transparent incremental build within a derivation impossible.

     

       6
       6
       -
       

     

       7
       7
       -
       However, we can tell Nix explicitly what the previous build state was, by representing that previous state as a derivation output. This allows the passed build state to be used for an incremental build.

     

       8
       8
       -
       

     

       9
       9
       -
       To build a derivation incrementally, the following steps needs to be fullfilled:

     

       10
       10
       -
           * - run prepareIncrementalBuild on the desired derivation

     

       11
       11
       -
           *   e.G `incrementalBuildArtifacts = (pkgs.buildIncremental.prepareIncrementalBuild pkgs.virtualbox);`

     

       12
       12
       -
           * - change something you want in the sources of the package( e.G using source override)

     

       13
       13
       -
           *   changedVBox = pkgs.virtuabox.overrideAttrs (old: {

     

       14
       14
       -
           *      src = path/to/vbox/sources;

     

       15
       15
       -
           *   }

     

       16
       16
       -
           * - use `mkIncrementalBuild changedVBox buildOutput`

     

       17
       17
       -
           * enjoy shorter build times

     

       18
       18
       -
       

     

       19
       19
       -
       As Nix has no builtin support for the detection of the previous built derivation, a base version needs to be declared.  

     

       20
       20
       -
       To create the outputs later used as base version for incremental builds, the function `pkgs.buildIncremental.prepareIncrementalBuild` is used.  

     

       21
       21
       -
       The function takes the original derivation as an argument and transforms the output to a base version for an incremental build.  

     

       22
       22
       -
       While doing so, the original output is not created and the installation phase is overwritten to produce the incremental build artifacts.  

     

       23
       23
       -
       

     

       24
       24
       -
       When the built artifacts of the base version of the derivation are created, the code can be modified and changes are built using the `pkgs.buildIncremental.mkIncrementalBuild` function.  

     

       25
       25
       -
       The `pkgs.buildIncremental.mkIncrementalBuild` function detects the changes in the code and places the output of the base version derivation within the build folder.   

     

       26
       26
       -
       Then, the build tool is able to detect the changes and makes the decision of which parts of the derivation needs to be recompiled and produces the output, as expected in the derivation, without incremental build support.

     

       27
       27
       -
       

     

       28
       28
       -

+14 -14

pkgs/build-support/build-incremental.nix pkgs/build-support/checkpoint-build.nix

···

       2
       2
        
       rec {

     

       3
       3
        
         /* Prepare a derivation for local builds.

     

       4
       4
        
           *

     

       5
       5
       -
           * This function prepares incremental builds by provinding,

     

       5
       5
       +
           * This function prepares checkpoint builds by provinding,

     

       6
       6
        
           * containing the build output and the sources for cross checking.

     

       7
       7
       -
           * The build output can be used later to allow incremental builds

     

       8
       8
       -
           * by passing the derivation output to the `mkIncrementalBuild` function.

     

       7
       7
       +
           * The build output can be used later to allow checkpoint builds

     

       8
       8
       +
           * by passing the derivation output to the `mkCheckpointBuild` function.

     

       9
       9
        
           *

     

       10
       10
       -
           * To build a project incrementaly follow these steps:

     

       10
       10
       +
           * To build a project with checkpoints follow these steps:

     

       11
       11
        
           * - run prepareIncrementalBuild on the desired derivation

     

       12
       12
       -
           *   e.G `incrementalBuildArtifacts = (pkgs.buildIncremental.prepareIncrementalBuild pkgs.virtualbox);`

     

       12
       12
       +
           *   e.G `incrementalBuildArtifacts = (pkgs.checkpointBuildTools.prepareCheckpointBuild pkgs.virtualbox);`

     

       13
       13
        
           * - change something you want in the sources of the package( e.G using source override)

     

       14
       14
        
           *   changedVBox = pkgs.virtuabox.overrideAttrs (old: {

     

       15
       15
        
           *      src = path/to/vbox/sources;

     

       16
       16
        
           *   }

     

       17
       17
       -
           * - use `mkIncrementalBuild changedVBox buildOutput`

     

       17
       17
       +
           * - use `mkCheckpointedBuild changedVBox buildOutput`

     

       18
       18
        
           * - enjoy shorter build times

     

       19
       19
        
         */

     

       20
       20
       -
         prepareIncrementalBuild = drv: drv.overrideAttrs (old: {

     

       20
       20
       +
         prepareCheckpointBuild = drv: drv.overrideAttrs (old: {

     

       21
       21
        
           outputs = [ "out" ];

     

       22
       22
       -
           name = drv.name + "-incrementalBuildArtifacts";

     

       22
       22
       +
           name = drv.name + "-checkpointArtifacts";

     

       23
       23
        
           # To determine differences between the state of the build directory

     

       24
       24
        
           # from an earlier build and  a later one we store the state of the build

     

       25
       25
        
           # directory before build, but after patch phases.

     
···

       41
       41
        
           '';

     

       42
       42
        
         });

     

       43
       43
        
       

     

       44
       44
       -
         /* Build a derivation incrementally based on the output generated by

     

       45
       45
       -
           * the `prepareIncrementalBuild function.

     

       44
       44
       +
         /* Build a derivation based on the checkpoint output generated by

     

       45
       45
       +
           * the `prepareCheckpointBuild function.

     

       46
       46
        
           *

     

       47
       47
        
           * Usage:

     

       48
       48
        
           * let

     

       49
       49
       -
           *   incrementalBuildArtifacts = prepareIncrementalBuild drv

     

       50
       50
       -
           * in mkIncrementalBuild drv incrementalBuildArtifacts

     

       49
       49
       +
           *   checkpointArtifacts = prepareCheckpointBuild drv

     

       50
       50
       +
           * in mkCheckpointedBuild drv checkpointArtifacts

     

       51
       51
        
         */

     

       52
       52
       -
         mkIncrementalBuild = drv: previousBuildArtifacts: drv.overrideAttrs (old: {

     

       53
       53
       -
           # The actual incremental build phase.

     

       52
       52
       +
         mkCheckpointedBuild = drv: previousBuildArtifacts: drv.overrideAttrs (old: {

     

       53
       53
       +
           # The actual checkpoint build phase.

     

       54
       54
        
           # We compare the changed sources from a previous build with the current and create a patch

     

       55
       55
        
           # Afterwards we clean the build directory to copy the previous output files (Including the sources)

     

       56
       56
        
           # The source difference patch is applied to get the latest changes again to allow short build times.

+1 -1

pkgs/test/default.nix

···

       113
       113
        
       

     

       114
       114
        
         install-shell-files = callPackage ./install-shell-files {};

     

       115
       115
        
       

     

       116
       116
       -
         incremental-build = callPackage ./incrementalBuild {};

     

       116
       116
       +
         checkpoint-build = callPackage ./checkpointBuild {};

     

       117
       117
        
       

     

       118
       118
        
         kernel-config = callPackage ./kernel.nix {};

     

       119
       119

+8 -8

pkgs/test/incrementalBuild/default.nix pkgs/test/checkpointBuild/default.nix

···

       1
       1
       -
       { hello, incrementalBuildTools, runCommandNoCC, texinfo, stdenv, rsync }:

     

       1
       1
       +
       { hello, checkpointBuildTools, runCommandNoCC, texinfo, stdenv, rsync }:

     

       2
       2
        
       let

     

       3
       3
       -
         baseHelloArtifacts = incrementalBuildTools.prepareIncrementalBuild hello;

     

       3
       3
       +
         baseHelloArtifacts = checkpointBuildTools.prepareCheckpointBuild hello;

     

       4
       4
        
         patchedHello = hello.overrideAttrs (old: {

     

       5
       5
        
           buildInputs = [ texinfo ];

     

       6
       6
        
           src = runCommandNoCC "patch-hello-src" { } ''

     
···

       10
       10
        
             patch -p1 < ${./hello.patch}

     

       11
       11
        
           '';

     

       12
       12
        
         });

     

       13
       13
       -
         incrementalBuiltHello = incrementalBuildTools.mkIncrementalBuild patchedHello baseHelloArtifacts;

     

       13
       13
       +
         checkpointBuiltHello = checkpointBuildTools.mkCheckpointedBuild patchedHello baseHelloArtifacts;

     

       14
       14
        
       

     

       15
       15
       -
         incrementalBuiltHelloWithCheck = incrementalBuiltHello.overrideAttrs (old: {

     

       15
       15
       +
         checkpointBuiltHelloWithCheck = checkpointBuiltHello.overrideAttrs (old: {

     

       16
       16
        
           doCheck = true;

     

       17
       17
        
           checkPhase = ''

     

       18
       18
        
             echo "checking if unchanged source file is not recompiled"

     
···

       20
       20
        
           '';

     

       21
       21
        
         });

     

       22
       22
        
       

     

       23
       23
       -
         baseHelloRemoveFileArtifacts = incrementalBuildTools.prepareIncrementalBuild (hello.overrideAttrs (old: {

     

       23
       23
       +
         baseHelloRemoveFileArtifacts = checkpointBuildTools.prepareCheckpointBuild (hello.overrideAttrs (old: {

     

       24
       24
        
           patches = [ ./hello-additionalFile.patch ];

     

       25
       25
        
         }));

     

       26
       26
        
       

     
···

       41
       41
        
           '';

     

       42
       42
        
         });

     

       43
       43
        
       

     

       44
       44
       -
         incrementalBuiltHelloWithRemovedFile = incrementalBuildTools.mkIncrementalBuild patchedHelloRemoveFile baseHelloRemoveFileArtifacts;

     

       44
       44
       +
         checkpointBuiltHelloWithRemovedFile = checkpointBuildTools.mkCheckpointedBuild patchedHelloRemoveFile baseHelloRemoveFileArtifacts;

     

       45
       45
        
       in

     

       46
       46
        
       stdenv.mkDerivation {

     

       47
       47
        
         name = "patched-hello-returns-correct-output";

     
···

       49
       49
        
           touch $out

     

       50
       50
        
       

     

       51
       51
        
           echo "testing output of hello binary"

     

       52
       52
       -
           [ "$(${incrementalBuiltHelloWithCheck}/bin/hello)" = "Hello, incremental world!" ]

     

       52
       52
       +
           [ "$(${checkpointBuiltHelloWithCheck}/bin/hello)" = "Hello, incremental world!" ]

     

       53
       53
        
           echo "testing output of hello with removed file"

     

       54
       54
       -
           [ "$(${incrementalBuiltHelloWithRemovedFile}/bin/hello)" = "Hello, incremental world!" ]

     

       54
       54
       +
           [ "$(${checkpointBuiltHelloWithRemovedFile}/bin/hello)" = "Hello, incremental world!" ]

     

       55
       55
        
         '';

     

       56
       56
        
       }

     

       57
       57

pkgs/test/incrementalBuild/hello-additionalFile.patch pkgs/test/checkpointBuild/hello-additionalFile.patch

pkgs/test/incrementalBuild/hello-removeFile.patch pkgs/test/checkpointBuild/hello-removeFile.patch

pkgs/test/incrementalBuild/hello.patch pkgs/test/checkpointBuild/hello.patch

+1 -1

pkgs/top-level/all-packages.nix

···

       429
       429
        
       

     

       430
       430
        
         camunda-modeler = callPackage ../applications/misc/camunda-modeler { };

     

       431
       431
        
       

     

       432
       432
       -
         incrementalBuildTools = callPackage ../build-support/build-incremental.nix {};

     

       432
       432
       +
         checkpointBuildTools = callPackage ../build-support/checkpoint-build.nix {};

     

       433
       433
        
       

     

       434
       434
        
         caroline = callPackage ../development/libraries/caroline { };

     

       435
       435