pyrox.dev
1# Javascript {#language-javascript}
2
3## Introduction {#javascript-introduction}
4
5This contains instructions on how to package javascript applications.
6
7The various tools available will be listed in the [tools-overview](#javascript-tools-overview).
8Some general principles for packaging will follow.
9Finally some tool specific instructions will be given.
10
11## Getting unstuck / finding code examples {#javascript-finding-examples}
12
13If you find you are lacking inspiration for packaging javascript applications, the links below might prove useful.
14Searching online for prior art can be helpful if you are running into solved problems.
15
16### Github {#javascript-finding-examples-github}
17
18- Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code>
19- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+path%3A**%2Fflake.nix&type=code>
20
21### Gitlab {#javascript-finding-examples-gitlab}
22
23- Searching Nix files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix>
24- Searching just `flake.nix` files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix>
25
26## Tools overview {#javascript-tools-overview}
27
28## General principles {#javascript-general-principles}
29
30The following principles are given in order of importance with potential exceptions.
31
32### Try to use the same node version used upstream {#javascript-upstream-node-version}
33
34It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging.
35
36This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node.
37Some cryptic errors regarding V8 may appear.
38
39### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager}
40
41A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of `node_modules` for each tool.
42
43Guidelines of package managers, recommend to commit those lock files to the repos.
44If a particular lock file is present, it is a strong indication of which package manager is used upstream.
45
46It's better to try to use a Nix tool that understand the lock file.
47Using a different tool might give you hard to understand error because different packages have been installed.
48An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629).
49Upstream use npm, but this is an attempt to package it with `yarn2nix` (that uses yarn.lock).
50
51Using a different tool forces to commit a lock file to the repository.
52Those files are fairly large, so when packaging for nixpkgs, this approach does not scale well.
53
54Exceptions to this rule are:
55
56- When you encounter one of the bugs from a Nix tool. In each of the tool specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to recreate a lock file and commit it to nixpkgs. In general `yarn2nix` has less known problems and so a simple search in nixpkgs will reveal many yarn.lock files committed.
57- Some lock files contain particular version of a package that has been pulled off npm for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs.
58- The only tool that supports workspaces (a feature of npm that helps manage sub-directories with different package.json from a single top level package.json) is `yarn2nix`. If upstream has workspaces you should try `yarn2nix`.
59
60### Try to use upstream package.json {#javascript-upstream-package-json}
61
62Exceptions to this rule are:
63
64- Sometimes the upstream repo assumes some dependencies be installed globally. In that case you can add them manually to the upstream package.json (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for CLI tools. (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases).
65- Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version by removing the `^`.
66- Sometimes the script defined in the package.json does not work as is. Some scripts for example use CLI tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into,
67
68 ```sh
69 yarn build:ui
70 yarn build:server
71 # OR
72 npm run build:ui
73 npm run build:server
74 ```
75
76 when you need to override a package.json. It's nice to use the one from the upstream source and do some explicit override. Here is an example:
77
78 ```nix
79 {
80 patchedPackageJSON = final.runCommand "package.json" { } ''
81 ${jq}/bin/jq '.version = "0.4.0" |
82 .devDependencies."@jsdoc/cli" = "^0.2.5"
83 ${sonar-src}/package.json > $out
84 '';
85 }
86 ```
87
88 You will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see.
89
90### Using node_modules directly {#javascript-using-node_modules}
91
92Each tool has an abstraction to just build the node_modules (dependencies) directory.
93You can always use the `stdenv.mkDerivation` with the node_modules to build the package (symlink the node_modules directory and then use the package build command).
94The node_modules abstraction can be also used to build some web framework frontends.
95For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. `mkYarnModules` to make the derivation containing node_modules.
96Then when building the frontend you can just symlink the node_modules directory.
97
98## Javascript packages inside nixpkgs {#javascript-packages-nixpkgs}
99
100The [pkgs/development/node-packages](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages) folder contains a generated collection of [npm packages](https://npmjs.com/) that can be installed with the Nix package manager.
101
102As a rule of thumb, the package set should only provide _end user_ software packages, such as command-line utilities.
103Libraries should only be added to the package set if there is a non-npm package that requires it.
104
105When it is desired to use npm libraries in a development project, use the `node2nix` generator directly on the `package.json` configuration file of the project.
106
107The package set provides support for the official stable Node.js versions.
108The latest stable LTS release in `nodePackages`, as well as the latest stable current release in `nodePackages_latest`.
109
110If your package uses native addons, you need to examine what kind of native build system it uses. Here are some examples:
111
112- `node-gyp`
113- `node-gyp-builder`
114- `node-pre-gyp`
115
116After you have identified the correct system, you need to override your package expression while adding in build system as a build input.
117For example, `dat` requires `node-gyp-build`, so we override its expression in [pkgs/development/node-packages/overrides.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/overrides.nix):
118
119```nix
120{
121 dat = prev.dat.override (oldAttrs: {
122 buildInputs = [
123 final.node-gyp-build
124 pkgs.libtool
125 pkgs.autoconf
126 pkgs.automake
127 ];
128 meta = oldAttrs.meta // {
129 broken = since "12";
130 };
131 });
132}
133```
134
135### Adding and Updating Javascript packages in nixpkgs {#javascript-adding-or-updating-packages}
136
137To add a package from npm to nixpkgs:
138
1391. Modify [pkgs/development/node-packages/node-packages.json](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/node-packages.json) to add, update or remove package entries to have it included in `nodePackages` and `nodePackages_latest`.
1402. Run the script:
141
142 ```sh
143 ./pkgs/development/node-packages/generate.sh
144 ```
145
1463. Build your new package to test your changes:
147
148 ```sh
149 nix-build -A nodePackages.<new-or-updated-package>
150 ```
151
152 To build against the latest stable Current Node.js version (e.g. 18.x):
153
154 ```sh
155 nix-build -A nodePackages_latest.<new-or-updated-package>
156 ```
157
158 If the package doesn't build, you may need to add an override as explained above.
1594. If the package's name doesn't match any of the executables it provides, add an entry in [pkgs/development/node-packages/main-programs.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/main-programs.nix). This will be the case for all scoped packages, e.g., `@angular/cli`.
1605. Add and commit all modified and generated files.
161
162For more information about the generation process, consult the [README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` tool.
163
164To update npm packages in nixpkgs, run the same `generate.sh` script:
165
166```sh
167./pkgs/development/node-packages/generate.sh
168```
169
170#### Git protocol error {#javascript-git-error}
171
172Some packages may have Git dependencies from GitHub specified with `git://`.
173GitHub has [disabled unencrypted Git connections](https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git), so you may see the following error when running the generate script:
174
175```
176The unauthenticated git protocol on port 9418 is no longer supported
177```
178
179Use the following Git configuration to resolve the issue:
180
181```sh
182git config --global url."https://github.com/".insteadOf git://github.com/
183```
184
185## Tool specific instructions {#javascript-tool-specific}
186
187### buildNpmPackage {#javascript-buildNpmPackage}
188
189`buildNpmPackage` allows you to package npm-based projects in Nixpkgs without the use of an auto-generated dependencies file (as used in [node2nix](#javascript-node2nix)).
190It works by utilizing npm's cache functionality -- creating a reproducible cache that contains the dependencies of a project, and pointing npm to it.
191
192Here's an example:
193
194```nix
195{
196 lib,
197 buildNpmPackage,
198 fetchFromGitHub,
199}:
200
201buildNpmPackage (finalAttrs: {
202 pname = "flood";
203 version = "4.7.0";
204
205 src = fetchFromGitHub {
206 owner = "jesec";
207 repo = "flood";
208 tag = "v${finalAttrs.version}";
209 hash = "sha256-BR+ZGkBBfd0dSQqAvujsbgsEPFYw/ThrylxUbOksYxM=";
210 };
211
212 npmDepsHash = "sha256-tuEfyePwlOy2/mOPdXbqJskO6IowvAP4DWg8xSZwbJw=";
213
214 # The prepack script runs the build script, which we'd rather do in the build phase.
215 npmPackFlags = [ "--ignore-scripts" ];
216
217 NODE_OPTIONS = "--openssl-legacy-provider";
218
219 meta = {
220 description = "Modern web UI for various torrent clients with a Node.js backend and React frontend";
221 homepage = "https://flood.js.org";
222 license = lib.licenses.gpl3Only;
223 maintainers = with lib.maintainers; [ winter ];
224 };
225})
226```
227
228In the default `installPhase` set by `buildNpmPackage`, it uses `npm pack --json --dry-run` to decide what files to install in `$out/lib/node_modules/$name/`, where `$name` is the `name` string defined in the package's `package.json`.
229Additionally, the `bin` and `man` keys in the source's `package.json` are used to decide what binaries and manpages are supposed to be installed.
230If these are not defined, `npm pack` may miss some files, and no binaries will be produced.
231
232#### Arguments {#javascript-buildNpmPackage-arguments}
233
234* `npmDepsHash`: The output hash of the dependencies for this project. Can be calculated in advance with [`prefetch-npm-deps`](#javascript-buildNpmPackage-prefetch-npm-deps).
235* `makeCacheWritable`: Whether to make the cache writable prior to installing dependencies. Don't set this unless npm tries to write to the cache directory, as it can slow down the build.
236* `npmBuildScript`: The script to run to build the project. Defaults to `"build"`.
237* `npmWorkspace`: The workspace directory within the project to build and install.
238* `dontNpmBuild`: Option to disable running the build script. Set to `true` if the package does not have a build script. Defaults to `false`. Alternatively, setting `buildPhase` explicitly also disables this.
239* `dontNpmInstall`: Option to disable running `npm install`. Defaults to `false`. Alternatively, setting `installPhase` explicitly also disables this.
240* `npmFlags`: Flags to pass to all npm commands.
241* `npmInstallFlags`: Flags to pass to `npm ci`.
242* `npmBuildFlags`: Flags to pass to `npm run ${npmBuildScript}`.
243* `npmPackFlags`: Flags to pass to `npm pack`.
244* `npmPruneFlags`: Flags to pass to `npm prune`. Defaults to the value of `npmInstallFlags`.
245* `makeWrapperArgs`: Flags to pass to `makeWrapper`, added to executable calling the generated `.js` with `node` as an interpreter. These scripts are defined in `package.json`.
246* `nodejs`: The `nodejs` package to build against, using the corresponding `npm` shipped with that version of `node`. Defaults to `pkgs.nodejs`.
247* `npmDeps`: The dependencies used to build the npm package. Especially useful to not have to recompute workspace dependencies.
248
249#### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps}
250
251`prefetch-npm-deps` is a Nixpkgs package that calculates the hash of the dependencies of an npm project ahead of time.
252
253```console
254$ ls
255package.json package-lock.json index.js
256$ prefetch-npm-deps package-lock.json
257...
258sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
259```
260
261#### fetchNpmDeps {#javascript-buildNpmPackage-fetchNpmDeps}
262
263`fetchNpmDeps` is a Nix function that requires the following mandatory arguments:
264
265- `src`: A directory / tarball with `package-lock.json` file
266- `hash`: The output hash of the node dependencies defined in `package-lock.json`.
267
268It returns a derivation with all `package-lock.json` dependencies downloaded into `$out/`, usable as an npm cache.
269
270#### importNpmLock {#javascript-buildNpmPackage-importNpmLock}
271
272This function replaces the npm dependency references in `package.json` and `package-lock.json` with paths to the Nix store.
273How each dependency is fetched can be customized with the `fetcherOpts` argument.
274
275This is a simpler and more convenient alternative to [`fetchNpmDeps`](#javascript-buildNpmPackage-fetchNpmDeps) for managing npm dependencies in Nixpkgs.
276There is no need to specify a `hash`, since it relies entirely on the integrity hashes already present in the `package-lock.json` file.
277
278##### Inputs {#javascript-buildNpmPackage-inputs}
279
280- `npmRoot`: Path to package directory containing the source tree.
281 If this is omitted, the `package` and `packageLock` arguments must be specified instead.
282- `package`: Parsed contents of `package.json`
283- `packageLock`: Parsed contents of `package-lock.json`
284- `pname`: Package name
285- `version`: Package version
286- `fetcherOpts`: An attribute set of arguments forwarded to the underlying fetcher.
287
288It returns a derivation with a patched `package.json` & `package-lock.json` with all dependencies resolved to Nix store paths.
289
290:::{.note}
291`npmHooks.npmConfigHook` cannot be used with `importNpmLock`.
292Use `importNpmLock.npmConfigHook` instead.
293:::
294
295:::{.example}
296
297##### `pkgs.importNpmLock` usage example {#javascript-buildNpmPackage-example}
298```nix
299{ buildNpmPackage, importNpmLock }:
300
301buildNpmPackage {
302 pname = "hello";
303 version = "0.1.0";
304 src = ./.;
305
306 npmDeps = importNpmLock {
307 npmRoot = ./.;
308 };
309
310 npmConfigHook = importNpmLock.npmConfigHook;
311}
312```
313:::
314
315:::{.example}
316##### `pkgs.importNpmLock` usage example with `fetcherOpts` {#javascript-buildNpmPackage-example-fetcherOpts}
317
318`importNpmLock` uses the following fetchers:
319
320- `pkgs.fetchurl` for `http(s)` dependencies
321- `builtins.fetchGit` for `git` dependencies
322
323It is possible to provide additional arguments to individual fetchers as needed:
324
325```nix
326{ buildNpmPackage, importNpmLock }:
327
328buildNpmPackage {
329 pname = "hello";
330 version = "0.1.0";
331 src = ./.;
332
333 npmDeps = importNpmLock {
334 npmRoot = ./.;
335 fetcherOpts = {
336 # Pass 'curlOptsList' to 'pkgs.fetchurl' while fetching 'axios'
337 "node_modules/axios" = {
338 curlOptsList = [ "--verbose" ];
339 };
340 };
341 };
342
343 npmConfigHook = importNpmLock.npmConfigHook;
344}
345```
346:::
347
348#### importNpmLock.buildNodeModules {#javascript-buildNpmPackage-importNpmLock.buildNodeModules}
349
350`importNpmLock.buildNodeModules` returns a derivation with a pre-built `node_modules` directory, as imported by `importNpmLock`.
351
352This is to be used together with `importNpmLock.hooks.linkNodeModulesHook` to facilitate `nix-shell`/`nix develop` based development workflows.
353
354It accepts an argument with the following attributes:
355
356`npmRoot` (Path; optional)
357: Path to package directory containing the source tree. If not specified, the `package` and `packageLock` arguments must both be specified.
358
359`package` (Attrset; optional)
360: Parsed contents of `package.json`, as returned by `lib.importJSON ./my-package.json`. If not specified, the `package.json` in `npmRoot` is used.
361
362`packageLock` (Attrset; optional)
363: Parsed contents of `package-lock.json`, as returned `lib.importJSON ./my-package-lock.json`. If not specified, the `package-lock.json` in `npmRoot` is used.
364
365`derivationArgs` (`mkDerivation` attrset; optional)
366: Arguments passed to `stdenv.mkDerivation`
367
368For example:
369
370```nix
371pkgs.mkShell {
372 packages = [
373 importNpmLock.hooks.linkNodeModulesHook
374 nodejs
375 ];
376
377 npmDeps = importNpmLock.buildNodeModules {
378 npmRoot = ./.;
379 inherit nodejs;
380 };
381}
382```
383will create a development shell where a `node_modules` directory is created & packages symlinked to the Nix store when activated.
384
385:::{.note}
386Commands like `npm install` & `npm add` that writes packages & executables needs to be used with `--package-lock-only`.
387
388This means `npm` installs dependencies by writing into `package-lock.json` without modifying the `node_modules` folder. Installation happens through reloading the devShell.
389This might be best practice since it gives the `nix shell` virtually exclusive ownership over your `node_modules` folder.
390
391It's recommended to set `package-lock-only = true` in your project-local [`.npmrc`](https://docs.npmjs.com/cli/v11/configuring-npm/npmrc).
392:::
393
394### corepack {#javascript-corepack}
395
396This package puts the corepack wrappers for pnpm and yarn in your PATH, and they will honor the `packageManager` setting in the `package.json`.
397
398### node2nix {#javascript-node2nix}
399
400#### Preparation {#javascript-node2nix-preparation}
401
402You will need to generate a Nix expression for the dependencies. Don't forget the `-l package-lock.json` if there is a lock file. Most probably you will need the `--development` to include the `devDependencies`
403
404So the command will most likely be:
405```sh
406node2nix --development -l package-lock.json
407```
408
409See `node2nix` [docs](https://github.com/svanderburg/node2nix) for more info.
410
411#### Pitfalls {#javascript-node2nix-pitfalls}
412
413- If upstream package.json does not have a "version" attribute, `node2nix` will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json).
414- `node2nix` has some [bugs](https://github.com/svanderburg/node2nix/issues/238) related to working with lock files from npm distributed with `nodejs_16`.
415- `node2nix` does not like missing packages from npm. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of npm.
416
417### pnpm {#javascript-pnpm}
418
419Pnpm is available as the top-level package `pnpm`. Additionally, there are variants pinned to certain major versions, like `pnpm_8` and `pnpm_9`, which support different sets of lock file versions.
420
421When packaging an application that includes a `pnpm-lock.yaml`, you need to fetch the pnpm store for that project using a fixed-output-derivation. The functions `pnpm_8.fetchDeps` and `pnpm_9.fetchDeps` can create this pnpm store derivation. In conjunction, the setup hooks `pnpm_8.configHook` and `pnpm_9.configHook` will prepare the build environment to install the prefetched dependencies store. Here is an example for a package that contains a `package.json` and a `pnpm-lock.yaml` files using the above `pnpm_` attributes:
422
423```nix
424{
425 stdenv,
426 nodejs,
427 # This is pinned as { pnpm = pnpm_9; }
428 pnpm,
429}:
430
431stdenv.mkDerivation (finalAttrs: {
432 pname = "foo";
433 version = "0-unstable-1980-01-01";
434
435 src = {
436 #...
437 };
438
439 nativeBuildInputs = [
440 nodejs
441 pnpm.configHook
442 ];
443
444 pnpmDeps = pnpm.fetchDeps {
445 inherit (finalAttrs) pname version src;
446 hash = "...";
447 };
448})
449```
450
451NOTE: It is highly recommended to use a pinned version of pnpm (i.e. `pnpm_8` or `pnpm_9`), to increase future reproducibility. It might also be required to use an older version, if the package needs support for a certain lock file version.
452
453In case you are patching `package.json` or `pnpm-lock.yaml`, make sure to pass `finalAttrs.patches` to the function as well (i.e. `inherit (finalAttrs) patches`.
454
455`pnpm.configHook` supports adding additional `pnpm install` flags via `pnpmInstallFlags` which can be set to a Nix string array:
456
457```nix
458{
459 pnpm,
460}:
461
462stdenv.mkDerivation (finalAttrs: {
463 pname = "foo";
464 version = "0-unstable-1980-01-01";
465
466 src = {
467 # ...
468 };
469
470 pnpmInstallFlags = [ "--shamefully-hoist" ];
471
472 pnpmDeps = pnpm.fetchDeps {
473 inherit (finalAttrs) pnpmInstallFlags;
474 };
475})
476```
477
478#### Dealing with `sourceRoot` {#javascript-pnpm-sourceRoot}
479
480If the pnpm project is in a subdirectory, you can just define `sourceRoot` or `setSourceRoot` for `fetchDeps`.
481If `sourceRoot` is different between the parent derivation and `fetchDeps`, you will have to set `pnpmRoot` to effectively be the same location as it is in `fetchDeps`.
482
483Assuming the following directory structure, we can define `sourceRoot` and `pnpmRoot` as follows:
484
485```
486.
487├── frontend
488│ ├── ...
489│ ├── package.json
490│ └── pnpm-lock.yaml
491└── ...
492```
493
494```nix
495{
496 # ...
497 pnpmDeps = pnpm.fetchDeps {
498 # ...
499 sourceRoot = "${finalAttrs.src.name}/frontend";
500 };
501
502 # by default the working directory is the extracted source
503 pnpmRoot = "frontend";
504}
505```
506
507#### PNPM Workspaces {#javascript-pnpm-workspaces}
508
509If you need to use a PNPM workspace for your project, then set `pnpmWorkspaces = [ "<workspace project name 1>" "<workspace project name 2>" ]`, etc, in your `pnpm.fetchDeps` call,
510which will make PNPM only install dependencies for those workspace packages.
511
512For example:
513
514```nix
515{
516 # ...
517 pnpmWorkspaces = [ "@astrojs/language-server" ];
518 pnpmDeps = pnpm.fetchDeps {
519 inherit (finalAttrs) pnpmWorkspaces;
520 #...
521 };
522}
523```
524
525The above would make `pnpm.fetchDeps` call only install dependencies for the `@astrojs/language-server` workspace package.
526Note that you do not need to set `sourceRoot` to make this work.
527
528Usually in such cases, you'd want to use `pnpm --filter=<pnpm workspace name> build` to build your project, as `npmHooks.npmBuildHook` probably won't work. A `buildPhase` based on the following example will probably fit most workspace projects:
529
530```nix
531{
532 buildPhase = ''
533 runHook preBuild
534
535 pnpm --filter=@astrojs/language-server build
536
537 runHook postBuild
538 '';
539}
540```
541
542#### Additional PNPM Commands and settings {#javascript-pnpm-extraCommands}
543
544If you require setting an additional PNPM configuration setting (such as `dedupe-peer-dependents` or similar),
545set `prePnpmInstall` to the right commands to run. For example:
546
547```nix
548{
549 prePnpmInstall = ''
550 pnpm config set dedupe-peer-dependants false
551 '';
552 pnpmDeps = pnpm.fetchDeps {
553 inherit (finalAttrs) prePnpmInstall;
554 # ...
555 };
556}
557```
558
559In this example, `prePnpmInstall` will be run by both `pnpm.configHook` and by the `pnpm.fetchDeps` builder.
560
561
562### Yarn {#javascript-yarn}
563
564Yarn based projects use a `yarn.lock` file instead of a `package-lock.json` to pin dependencies.
565
566To package yarn-based applications, you need to distinguish by the version pointers in the `yarn.lock` file. See the following sections.
567
568#### Yarn v1 {#javascript-yarn-v1}
569
570Yarn v1 lockfiles contain a comment `# yarn lockfile v1` at the beginning of the file.
571
572Nixpkgs provides the Nix function `fetchYarnDeps` which fetches an offline cache suitable for running `yarn install` before building the project. In addition, Nixpkgs provides the hooks:
573
574- `yarnConfigHook`: Fetches the dependencies from the offline cache and installs them into `node_modules`.
575- `yarnBuildHook`: Runs `yarn build` or a specified `yarn` command that builds the project.
576- `yarnInstallHook`: Runs `yarn install --production` to prune dependencies and installs the project into `$out`.
577
578An example usage of the above attributes is:
579
580```nix
581{
582 lib,
583 stdenv,
584 fetchFromGitHub,
585 fetchYarnDeps,
586 yarnConfigHook,
587 yarnBuildHook,
588 yarnInstallHook,
589 nodejs,
590}:
591
592stdenv.mkDerivation (finalAttrs: {
593 pname = "...";
594 version = "...";
595
596 src = fetchFromGitHub {
597 owner = "...";
598 repo = "...";
599 rev = "v${finalAttrs.version}";
600 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
601 };
602
603 yarnOfflineCache = fetchYarnDeps {
604 yarnLock = finalAttrs.src + "/yarn.lock";
605 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
606 };
607
608 nativeBuildInputs = [
609 yarnConfigHook
610 yarnBuildHook
611 yarnInstallHook
612 # Needed for executing package.json scripts
613 nodejs
614 ];
615
616 meta = {
617 # ...
618 };
619})
620```
621
622##### `yarnConfigHook` arguments {#javascript-yarnconfighook}
623
624By default, `yarnConfigHook` relies upon the attribute `${yarnOfflineCache}` (or `${offlineCache}` if the former is not set) to find the location of the offline cache produced by `fetchYarnDeps`. To disable this phase, you can set `dontYarnInstallDeps = true` or override the `configurePhase`.
625
626##### `yarnBuildHook` arguments {#javascript-yarnbuildhook}
627
628This script by default runs `yarn --offline build`, and it relies upon the project's dependencies installed at `node_modules`. Below is a list of additional `mkDerivation` arguments read by this hook:
629
630- `yarnBuildScript`: Sets a different `yarn --offline` subcommand (defaults to `build`).
631- `yarnBuildFlags`: Single string list of additional flags to pass the above command, or a Nix list of such additional flags.
632
633##### `yarnInstallHook` arguments {#javascript-yarninstallhook}
634
635To install the package `yarnInstallHook` uses both `npm` and `yarn` to cleanup project files and dependencies. To disable this phase, you can set `dontYarnInstall = true` or override the `installPhase`. Below is a list of additional `mkDerivation` arguments read by this hook:
636
637- `yarnKeepDevDeps`: Disables the removal of devDependencies from `node_modules` before installation.
638
639#### yarn2nix {#javascript-yarn2nix}
640
641WARNING: The `yarn2nix` functions have been deprecated in favor of `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook` (for Yarn v1) and `yarn-berry_*.*` tooling (Yarn v3 and v4). Documentation for `yarn2nix` functions still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
642
643##### Preparation {#javascript-yarn2nix-preparation}
644
645You will need at least a `yarn.lock` file. If upstream does not have one you need to generate it and reference it in your package definition.
646
647If the downloaded files contain the `package.json` and `yarn.lock` files they can be used like this:
648
649```nix
650{
651 offlineCache = fetchYarnDeps {
652 yarnLock = src + "/yarn.lock";
653 hash = "....";
654 };
655}
656```
657
658##### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
659
660`mkYarnPackage` will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React, WebPack, ...), you will need to explicitly override the build step with your instructions.
661
662It's important to use the `--offline` flag. For example if you script is `"build": "something"` in `package.json` use:
663
664```nix
665{
666 nativeBuildInputs = [
667 writableTmpDirAsHomeHook
668 ];
669
670 buildPhase = ''
671 runHook preBuild
672
673 yarn --offline build
674
675 runHook postBuild
676 '';
677}
678```
679
680The `distPhase` is packing the package's dependencies in a tarball using `yarn pack`. You can disable it using:
681
682```nix
683{
684 doDist = false;
685}
686```
687
688The configure phase can sometimes fail because it makes many assumptions which may not always apply. One common override is:
689
690```nix
691{
692 configurePhase = ''
693 ln -s $node_modules node_modules
694 '';
695}
696```
697
698or if you need a writeable node_modules directory:
699
700```nix
701{
702 configurePhase = ''
703 cp -r $node_modules node_modules
704 chmod +w node_modules
705 '';
706}
707```
708
709##### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
710
711This will generate a derivation including the `node_modules` directory.
712If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way.
713
714#### Overriding dependency behavior {#javascript-mkYarnPackage-overriding-dependencies}
715
716In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building.
717
718For instance, say your package is throwing errors when trying to invoke node-sass:
719
720```
721ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'
722```
723
724To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want:
725
726```nix
727mkYarnPackage rec {
728 pkgConfig = {
729 node-sass = {
730 buildInputs = with final; [
731 python
732 libsass
733 pkg-config
734 ];
735 postInstall = ''
736 LIBSASS_EXT=auto yarn --offline run build
737 rm build/config.gypi
738 '';
739 };
740 };
741}
742```
743
744##### Pitfalls {#javascript-yarn2nix-pitfalls}
745
746- If version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
747- Having trouble with `node-gyp`? Try adding these lines to the `yarnPreBuild` steps:
748
749 ```nix
750 {
751 yarnPreBuild = ''
752 mkdir -p $HOME/.node-gyp/${nodejs.version}
753 echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion
754 ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version}
755 export npm_config_nodedir=${nodejs}
756 '';
757 }
758 ```
759
760 - The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496>
761 - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
762- `offlineCache` (described [above](#javascript-yarn2nix-preparation)) must be specified to avoid [Import From Derivation](#ssec-import-from-derivation) (IFD) when used inside Nixpkgs.
763
764#### Yarn Berry v3/v4 {#javascript-yarn-v3-v4}
765Yarn Berry (v3 / v4) have similar formats, they start with blocks like these:
766
767```yaml
768__metadata:
769 version: 6
770 cacheKey: 8[cX]
771```
772
773```yaml
774__metadata:
775 version: 8
776 cacheKey: 10[cX]
777```
778
779For these packages, we have some helpers exposed under the respective `yarn-berry_3` and `yarn-berry_4` packages:
780
781- `yarn-berry-fetcher`
782- `fetchYarnBerryDeps`
783- `yarnBerryConfigHook`
784
785It's recommended to ensure you're explicitly pinning the major version used, for example by capturing the `yarn-berry_Xn` argument and then re-defining it as a `yarn-berry` `let` binding.
786
787```nix
788{
789 stdenv,
790 nodejs,
791 yarn-berry_4,
792}:
793
794let
795 yarn-berry = yarn-berry_4;
796in
797
798stdenv.mkDerivation (finalAttrs: {
799 pname = "foo";
800 version = "0-unstable-1980-01-01";
801
802 src = {
803 #...
804 };
805
806 nativeBuildInputs = [
807 nodejs
808 yarn-berry.yarnBerryConfigHook
809 ];
810
811 offlineCache = yarn-berry.fetchYarnBerryDeps {
812 inherit (finalAttrs) src;
813 hash = "...";
814 };
815})
816```
817
818##### `yarn-berry_X.fetchYarnBerryDeps` {#javascript-fetchYarnBerryDeps}
819`fetchYarnBerryDeps` runs `yarn-berry-fetcher fetch` in a fixed-output-derivation. It is a custom fetcher designed to reproducibly download all files in the `yarn.lock` file, validating their hashes in the process. For git dependencies, it creates a checkout at `${offlineCache}/checkouts/<40-character-commit-hash>` (relying on the git commit hash to describe the contents of the checkout).
820
821To produce the `hash` argument for `fetchYarnBerryDeps` function call, the `yarn-berry-fetcher prefetch` command can be used:
822
823```console
824$ yarn-berry-fetcher prefetch </path/to/yarn.lock> [/path/to/missing-hashes.json]
825```
826
827This prints the hash to stdout and can be used in update scripts to recalculate the hash for a new version of `yarn.lock`.
828
829##### `yarn-berry_X.yarnBerryConfigHook` {#javascript-yarnBerryConfigHook}
830`yarnBerryConfigHook` uses the store path `offlineCache` points to, to run a `yarn install` during the build, producing a usable `node_modules` directory from the downloaded dependencies.
831
832Internally, this uses a patched version of Yarn to ensure git dependencies are re-packed and any attempted downloads fail immediately.
833
834##### Patching upstream `package.json` or `yarn.lock` files {#javascript-yarnBerry-patching}
835In case patching the upstream `package.json` or `yarn.lock` is needed, it's important to pass `finalAttrs.patches` to `fetchYarnBerryDeps` as well, so the patched variants are picked up (i.e. `inherit (finalAttrs) patches`.
836
837##### Missing hashes in the `yarn.lock` file {#javascript-yarnBerry-missing-hashes}
838Unfortunately, `yarn.lock` files do not include hashes for optional/platform-specific dependencies. This is [by design](https://github.com/yarnpkg/berry/issues/6759).
839
840To compensate for this, the `yarn-berry-fetcher missing-hashes` subcommand can be used to produce all missing hashes. These are usually stored in a `missing-hashes.json` file, which needs to be passed to both the build itself, as well as the `fetchYarnBerryDeps` helper:
841
842```nix
843{
844 stdenv,
845 nodejs,
846 yarn-berry_4,
847}:
848
849let
850 yarn-berry = yarn-berry_4;
851in
852
853stdenv.mkDerivation (finalAttrs: {
854 pname = "foo";
855 version = "0-unstable-1980-01-01";
856
857 src = {
858 #...
859 };
860
861 nativeBuildInputs = [
862 nodejs
863 yarn-berry.yarnBerryConfigHook
864 ];
865
866 missingHashes = ./missing-hashes.json;
867 offlineCache = yarn-berry.fetchYarnBerryDeps {
868 inherit (finalAttrs) src missingHashes;
869 hash = "...";
870 };
871})
872```
873
874## Outside Nixpkgs {#javascript-outside-nixpkgs}
875
876There are some other tools available, which are written in the Nix language.
877These that can't be used inside Nixpkgs because they require [Import From Derivation](#ssec-import-from-derivation), which is not allowed in Nixpkgs.
878
879If you are packaging something outside Nixpkgs, consider the following:
880
881### npmlock2nix {#javascript-npmlock2nix}
882
883[npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might be subject to change.
884
885#### Pitfalls {#javascript-npmlock2nix-pitfalls}
886
887There are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45).
888
889### nix-npm-buildpackage {#javascript-nix-npm-buildpackage}
890
891[nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might change. It supports both `package-lock.json` and yarn.lock.
892
893#### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls}
894
895There are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33).