pyrox.dev
1# Javascript {#language-javascript}
2
3## Introduction {#javascript-introduction}
4
5This contains instructions on how to package JavaScript applications.
6
7The various tools available will be listed in the [tools-overview](#javascript-tools-overview).
8Some general principles for packaging will follow.
9Finally, some tool-specific instructions will be given.
10
11## Getting unstuck / finding code examples {#javascript-finding-examples}
12
13If you find you are lacking inspiration for packaging JavaScript applications, the links below might prove useful.
14Searching online for prior art can be helpful if you are running into solved problems.
15
16### Github {#javascript-finding-examples-github}
17
18- Searching Nix files for `yarnConfigHook`: <https://github.com/search?q=yarnConfigHook+language%3ANix&type=code>
19- Searching just `flake.nix` files for `yarnConfigHook`: <https://github.com/search?q=yarnConfigHook+path%3A**%2Fflake.nix&type=code>
20
21### Gitlab {#javascript-finding-examples-gitlab}
22
23- Searching Nix files for `yarnConfigHook`: <https://gitlab.com/search?scope=blobs&search=yarnConfigHook+extension%3Anix>
24- Searching just `flake.nix` files for `yarnConfigHook`: <https://gitlab.com/search?scope=blobs&search=yarnConfigHook+filename%3Aflake.nix>
25
26## Tools overview {#javascript-tools-overview}
27
28## General principles {#javascript-general-principles}
29
30The following principles are given in order of importance with potential exceptions.
31
32### Try to use the same node version used upstream {#javascript-upstream-node-version}
33
34It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging.
35
36This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node.
37Some cryptic errors regarding V8 may appear.
38
39### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager}
40
41A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of `node_modules` for each tool.
42
43Guidelines of package managers, recommend to commit those lock files to the repos.
44If a particular lock file is present, it is a strong indication of which package manager is used upstream.
45
46It's better to try to use a Nix tool that understands the lock file.
47Using a different tool might give you a hard-to-understand error because different packages have been installed.
48An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629).
49Upstream use npm, but this is an attempt to package it with `yarn2nix` (that uses yarn.lock).
50
51Using a different tool forces you to commit a lock file to the repository.
52These files are fairly large, so when packaging for nixpkgs, this approach does not scale well.
53
54Exceptions to this rule are:
55
56- When you encounter one of the bugs from a Nix tool. In each of the tool-specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to re-create a lock file and commit it to Nixpkgs. In general `yarn2nix` has fewer known problems, and so a simple search in Nixpkgs will reveal many `yarn.lock` files committed.
57- Some lock files contain particular version of a package that has been pulled off npm for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs.
58- The only tool that supports workspaces (a feature of npm that helps manage sub-directories with different package.json from a single top level package.json) is `yarn2nix`. If upstream has workspaces you should try `yarn2nix`.
59
60### Try to use upstream package.json {#javascript-upstream-package-json}
61
62Exceptions to this rule are:
63
64- Sometimes the upstream repo assumes some dependencies should be installed globally. In that case, you can add them manually to the upstream `package.json` (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for CLI tools (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases).
65- Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version by removing the `^`.
66- Sometimes the script defined in the package.json does not work as is. Some scripts for example use CLI tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into,
67
68 ```sh
69 yarn build:ui
70 yarn build:server
71 # OR
72 npm run build:ui
73 npm run build:server
74 ```
75
76 when you need to override a package.json. It's nice to use the one from the upstream source and do some explicit override. Here is an example:
77
78 ```nix
79 {
80 patchedPackageJSON = final.runCommand "package.json" { } ''
81 ${jq}/bin/jq '.version = "0.4.0" |
82 .devDependencies."@jsdoc/cli" = "^0.2.5"
83 ${sonar-src}/package.json > $out
84 '';
85 }
86 ```
87
88 You will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see.
89
90### Using node_modules directly {#javascript-using-node_modules}
91
92Each tool has an abstraction to just build the node_modules (dependencies) directory.
93You can always use the `stdenv.mkDerivation` with the node_modules to build the package (symlink the node_modules directory and then use the package build command).
94The node_modules abstraction can be also used to build some web framework frontends.
95For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. `mkYarnModules` to make the derivation containing node_modules.
96Then when building the frontend you can just symlink the node_modules directory.
97
98## Javascript packages inside nixpkgs {#javascript-packages-nixpkgs}
99
100The [pkgs/development/node-packages](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages) folder contains a generated collection of [npm packages](https://npmjs.com/) that can be installed with the Nix package manager.
101
102As a rule of thumb, the package set should only provide _end-user_ software packages, such as command-line utilities.
103Libraries should only be added to the package set if there is a non-npm package that requires it.
104
105When it is desired to use npm libraries in a development project, use the `node2nix` generator directly on the `package.json` configuration file of the project.
106
107The package set provides support for the official stable Node.js versions.
108The latest stable LTS release in `nodePackages`, as well as the latest stable current release in `nodePackages_latest`.
109
110If your package uses native addons, you need to examine what kind of native build system it uses. Here are some examples:
111
112- `node-gyp`
113- `node-gyp-builder`
114- `node-pre-gyp`
115
116After you have identified the correct system, you need to override your package expression while adding in build system as a build input.
117For example, `dat` requires `node-gyp-build`, so we override its expression in [pkgs/development/node-packages/overrides.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/overrides.nix):
118
119```nix
120{
121 dat = prev.dat.override (oldAttrs: {
122 buildInputs = [
123 final.node-gyp-build
124 pkgs.libtool
125 pkgs.autoconf
126 pkgs.automake
127 ];
128 meta = oldAttrs.meta // {
129 broken = since "12";
130 };
131 });
132}
133```
134
135### Adding and updating JavaScript packages in Nixpkgs {#javascript-adding-or-updating-packages}
136
137To add a package from npm to Nixpkgs:
138
1391. Modify [pkgs/development/node-packages/node-packages.json](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/node-packages.json) to add, update or remove package entries to have it included in `nodePackages` and `nodePackages_latest`.
1402. Run the script:
141
142 ```sh
143 ./pkgs/development/node-packages/generate.sh
144 ```
145
1463. Build your new package to test your changes:
147
148 ```sh
149 nix-build -A nodePackages.<new-or-updated-package>
150 ```
151
152 To build against the latest stable Current Node.js version (e.g. 18.x):
153
154 ```sh
155 nix-build -A nodePackages_latest.<new-or-updated-package>
156 ```
157
158 If the package doesn't build, you may need to add an override as explained above.
1594. If the package's name doesn't match any of the executables it provides, add an entry in [pkgs/development/node-packages/main-programs.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/main-programs.nix). This will be the case for all scoped packages, e.g., `@angular/cli`.
1605. Add and commit all modified and generated files.
161
162For more information about the generation process, consult the [README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` tool.
163
164To update npm packages in Nixpkgs, run the same `generate.sh` script:
165
166```sh
167./pkgs/development/node-packages/generate.sh
168```
169
170#### Git protocol error {#javascript-git-error}
171
172Some packages may have Git dependencies from GitHub specified with `git://`.
173GitHub has [disabled unencrypted Git connections](https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git), so you may see the following error when running the generate script:
174
175```
176The unauthenticated git protocol on port 9418 is no longer supported
177```
178
179Use the following Git configuration to resolve the issue:
180
181```sh
182git config --global url."https://github.com/".insteadOf git://github.com/
183```
184
185## Tool-specific instructions {#javascript-tool-specific}
186
187### buildNpmPackage {#javascript-buildNpmPackage}
188
189`buildNpmPackage` allows you to package npm-based projects in Nixpkgs without the use of an auto-generated dependencies file (as used in [node2nix](#javascript-node2nix)).
190It works by utilizing npm's cache functionality -- creating a reproducible cache that contains the dependencies of a project, and pointing npm to it.
191
192Here's an example:
193
194```nix
195{
196 lib,
197 buildNpmPackage,
198 fetchFromGitHub,
199}:
200
201buildNpmPackage (finalAttrs: {
202 pname = "flood";
203 version = "4.7.0";
204
205 src = fetchFromGitHub {
206 owner = "jesec";
207 repo = "flood";
208 tag = "v${finalAttrs.version}";
209 hash = "sha256-BR+ZGkBBfd0dSQqAvujsbgsEPFYw/ThrylxUbOksYxM=";
210 };
211
212 npmDepsHash = "sha256-tuEfyePwlOy2/mOPdXbqJskO6IowvAP4DWg8xSZwbJw=";
213
214 # The prepack script runs the build script, which we'd rather do in the build phase.
215 npmPackFlags = [ "--ignore-scripts" ];
216
217 NODE_OPTIONS = "--openssl-legacy-provider";
218
219 meta = {
220 description = "Modern web UI for various torrent clients with a Node.js backend and React frontend";
221 homepage = "https://flood.js.org";
222 license = lib.licenses.gpl3Only;
223 maintainers = with lib.maintainers; [ winter ];
224 };
225})
226```
227
228In the default `installPhase` set by `buildNpmPackage`, it uses `npm pack --json --dry-run` to decide what files to install in `$out/lib/node_modules/$name/`, where `$name` is the `name` string defined in the package's `package.json`.
229Additionally, the `bin` and `man` keys in the source's `package.json` are used to decide what binaries and manpages are supposed to be installed.
230If these are not defined, `npm pack` may miss some files, and no binaries will be produced.
231
232#### Arguments {#javascript-buildNpmPackage-arguments}
233
234* `npmDepsHash`: The output hash of the dependencies for this project. Can be calculated in advance with [`prefetch-npm-deps`](#javascript-buildNpmPackage-prefetch-npm-deps).
235* `makeCacheWritable`: Whether to make the cache writable prior to installing dependencies. Don't set this unless npm tries to write to the cache directory, as it can slow down the build.
236* `npmBuildScript`: The script to run to build the project. Defaults to `"build"`.
237* `npmWorkspace`: The workspace directory within the project to build and install.
238* `dontNpmBuild`: Option to disable running the build script. Set to `true` if the package does not have a build script. Defaults to `false`. Alternatively, setting `buildPhase` explicitly also disables this.
239* `dontNpmInstall`: Option to disable running `npm install`. Defaults to `false`. Alternatively, setting `installPhase` explicitly also disables this.
240* `npmFlags`: Flags to pass to all npm commands.
241* `npmInstallFlags`: Flags to pass to `npm ci`.
242* `npmBuildFlags`: Flags to pass to `npm run ${npmBuildScript}`.
243* `npmPackFlags`: Flags to pass to `npm pack`.
244* `npmPruneFlags`: Flags to pass to `npm prune`. Defaults to the value of `npmInstallFlags`.
245* `makeWrapperArgs`: Flags to pass to `makeWrapper`, added to executable calling the generated `.js` with `node` as an interpreter. These scripts are defined in `package.json`.
246* `nodejs`: The `nodejs` package to build against, using the corresponding `npm` shipped with that version of `node`. Defaults to `pkgs.nodejs`.
247* `npmDeps`: The dependencies used to build the npm package. Especially useful to not have to recompute workspace dependencies.
248
249#### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps}
250
251`prefetch-npm-deps` is a Nixpkgs package that calculates the hash of the dependencies of an npm project ahead of time.
252
253```console
254$ ls
255package.json package-lock.json index.js
256$ prefetch-npm-deps package-lock.json
257...
258sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
259```
260
261#### fetchNpmDeps {#javascript-buildNpmPackage-fetchNpmDeps}
262
263`fetchNpmDeps` is a Nix function that requires the following mandatory arguments:
264
265- `src`: A directory / tarball with `package-lock.json` file
266- `hash`: The output hash of the node dependencies defined in `package-lock.json`.
267
268It returns a derivation with all `package-lock.json` dependencies downloaded into `$out/`, usable as an npm cache.
269
270#### importNpmLock {#javascript-buildNpmPackage-importNpmLock}
271
272This function replaces the npm dependency references in `package.json` and `package-lock.json` with paths to the Nix store.
273How each dependency is fetched can be customized with the `fetcherOpts` argument.
274
275This is a simpler and more convenient alternative to [`fetchNpmDeps`](#javascript-buildNpmPackage-fetchNpmDeps) for managing npm dependencies in Nixpkgs.
276There is no need to specify a `hash`, since it relies entirely on the integrity hashes already present in the `package-lock.json` file.
277
278##### Inputs {#javascript-buildNpmPackage-inputs}
279
280- `npmRoot`: Path to package directory containing the source tree.
281 If this is omitted, the `package` and `packageLock` arguments must be specified instead.
282- `package`: Parsed contents of `package.json`
283- `packageLock`: Parsed contents of `package-lock.json`
284- `pname`: Package name
285- `version`: Package version
286- `fetcherOpts`: An attribute set of arguments forwarded to the underlying fetcher.
287
288It returns a derivation with a patched `package.json` & `package-lock.json` with all dependencies resolved to Nix store paths.
289
290:::{.note}
291`npmHooks.npmConfigHook` cannot be used with `importNpmLock`.
292Use `importNpmLock.npmConfigHook` instead.
293:::
294
295:::{.example}
296
297##### `pkgs.importNpmLock` usage example {#javascript-buildNpmPackage-example}
298```nix
299{ buildNpmPackage, importNpmLock }:
300
301buildNpmPackage {
302 pname = "hello";
303 version = "0.1.0";
304 src = ./.;
305
306 npmDeps = importNpmLock { npmRoot = ./.; };
307
308 npmConfigHook = importNpmLock.npmConfigHook;
309}
310```
311:::
312
313:::{.example}
314##### `pkgs.importNpmLock` usage example with `fetcherOpts` {#javascript-buildNpmPackage-example-fetcherOpts}
315
316`importNpmLock` uses the following fetchers:
317
318- `pkgs.fetchurl` for `http(s)` dependencies
319- `fetchGit` for `git` dependencies
320
321It is possible to provide additional arguments to individual fetchers as needed:
322
323```nix
324{ buildNpmPackage, importNpmLock }:
325
326buildNpmPackage {
327 pname = "hello";
328 version = "0.1.0";
329 src = ./.;
330
331 npmDeps = importNpmLock {
332 npmRoot = ./.;
333 fetcherOpts = {
334 # Pass 'curlOptsList' to 'pkgs.fetchurl' while fetching 'axios'
335 "node_modules/axios" = {
336 curlOptsList = [ "--verbose" ];
337 };
338 };
339 };
340
341 npmConfigHook = importNpmLock.npmConfigHook;
342}
343```
344:::
345
346#### importNpmLock.buildNodeModules {#javascript-buildNpmPackage-importNpmLock.buildNodeModules}
347
348`importNpmLock.buildNodeModules` returns a derivation with a pre-built `node_modules` directory, as imported by `importNpmLock`.
349
350This is to be used together with `importNpmLock.hooks.linkNodeModulesHook` to facilitate `nix-shell`/`nix develop` based development workflows.
351
352It accepts an argument with the following attributes:
353
354`npmRoot` (Path; optional)
355: Path to package directory containing the source tree. If not specified, the `package` and `packageLock` arguments must both be specified.
356
357`package` (Attrset; optional)
358: Parsed contents of `package.json`, as returned by `lib.importJSON ./my-package.json`. If not specified, the `package.json` in `npmRoot` is used.
359
360`packageLock` (Attrset; optional)
361: Parsed contents of `package-lock.json`, as returned `lib.importJSON ./my-package-lock.json`. If not specified, the `package-lock.json` in `npmRoot` is used.
362
363`derivationArgs` (`mkDerivation` attrset; optional)
364: Arguments passed to `stdenv.mkDerivation`
365
366For example:
367
368```nix
369pkgs.mkShell {
370 packages = [
371 importNpmLock.hooks.linkNodeModulesHook
372 nodejs
373 ];
374
375 npmDeps = importNpmLock.buildNodeModules {
376 npmRoot = ./.;
377 inherit nodejs;
378 };
379}
380```
381will create a development shell where a `node_modules` directory is created & packages symlinked to the Nix store when activated.
382
383:::{.note}
384Commands like `npm install` & `npm add` that write packages & executables need to be used with `--package-lock-only`.
385
386This means `npm` installs dependencies by writing into `package-lock.json` without modifying the `node_modules` folder. Installation happens through reloading the devShell.
387This might be best practice since it gives the `nix shell` virtually exclusive ownership over your `node_modules` folder.
388
389It's recommended to set `package-lock-only = true` in your project-local [`.npmrc`](https://docs.npmjs.com/cli/v11/configuring-npm/npmrc).
390:::
391
392### corepack {#javascript-corepack}
393
394This package puts the corepack wrappers for pnpm and yarn in your PATH, and they will honor the `packageManager` setting in the `package.json`.
395
396### node2nix {#javascript-node2nix}
397
398#### Preparation {#javascript-node2nix-preparation}
399
400You will need to generate a Nix expression for the dependencies. Don't forget the `-l package-lock.json` if there is a lock file. Most probably you will need the `--development` to include the `devDependencies`
401
402So the command will most likely be:
403```sh
404node2nix --development -l package-lock.json
405```
406
407See `node2nix` [docs](https://github.com/svanderburg/node2nix) for more info.
408
409#### Pitfalls {#javascript-node2nix-pitfalls}
410
411- If upstream package.json does not have a "version" attribute, `node2nix` will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json).
412- `node2nix` has some [bugs](https://github.com/svanderburg/node2nix/issues/238) related to working with lock files from npm distributed with `nodejs_16`.
413- `node2nix` does not like missing packages from npm. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of npm.
414
415### pnpm {#javascript-pnpm}
416
417Pnpm is available as the top-level package `pnpm`. Additionally, there are variants pinned to certain major versions, like `pnpm_8` and `pnpm_9`, which support different sets of lock file versions.
418
419When packaging an application that includes a `pnpm-lock.yaml`, you need to fetch the pnpm store for that project using a fixed-output-derivation. The functions `pnpm_8.fetchDeps` and `pnpm_9.fetchDeps` can create this pnpm store derivation. In conjunction, the setup hooks `pnpm_8.configHook` and `pnpm_9.configHook` will prepare the build environment to install the pre-fetched dependencies store. Here is an example for a package that contains `package.json` and a `pnpm-lock.yaml` files using the above `pnpm_` attributes:
420
421```nix
422{
423 stdenv,
424 nodejs,
425 # This is pinned as { pnpm = pnpm_9; }
426 pnpm,
427}:
428
429stdenv.mkDerivation (finalAttrs: {
430 pname = "foo";
431 version = "0-unstable-1980-01-01";
432
433 src = {
434 #...
435 };
436
437 nativeBuildInputs = [
438 nodejs
439 pnpm.configHook
440 ];
441
442 pnpmDeps = pnpm.fetchDeps {
443 inherit (finalAttrs) pname version src;
444 fetcherVersion = 2;
445 hash = "...";
446 };
447})
448```
449
450NOTE: It is highly recommended to use a pinned version of pnpm (i.e., `pnpm_8` or `pnpm_9`), to increase future reproducibility. It might also be required to use an older version if the package needs support for a certain lock file version.
451
452In case you are patching `package.json` or `pnpm-lock.yaml`, make sure to pass `finalAttrs.patches` to the function as well (i.e., `inherit (finalAttrs) patches`.
453
454`pnpm.configHook` supports adding additional `pnpm install` flags via `pnpmInstallFlags` which can be set to a Nix string array:
455
456```nix
457{ pnpm }:
458
459stdenv.mkDerivation (finalAttrs: {
460 pname = "foo";
461 version = "0-unstable-1980-01-01";
462
463 src = {
464 # ...
465 };
466
467 pnpmInstallFlags = [ "--shamefully-hoist" ];
468
469 pnpmDeps = pnpm.fetchDeps { inherit (finalAttrs) pnpmInstallFlags; };
470})
471```
472
473#### Dealing with `sourceRoot` {#javascript-pnpm-sourceRoot}
474
475If the pnpm project is in a subdirectory, you can just define `sourceRoot` or `setSourceRoot` for `fetchDeps`.
476If `sourceRoot` is different between the parent derivation and `fetchDeps`, you will have to set `pnpmRoot` to effectively be the same location as it is in `fetchDeps`.
477
478Assuming the following directory structure, we can define `sourceRoot` and `pnpmRoot` as follows:
479
480```
481.
482├── frontend
483│ ├── ...
484│ ├── package.json
485│ └── pnpm-lock.yaml
486└── ...
487```
488
489```nix
490{
491 # ...
492 pnpmDeps = pnpm.fetchDeps {
493 # ...
494 sourceRoot = "${finalAttrs.src.name}/frontend";
495 };
496
497 # by default the working directory is the extracted source
498 pnpmRoot = "frontend";
499}
500```
501
502#### PNPM Workspaces {#javascript-pnpm-workspaces}
503
504If you need to use a PNPM workspace for your project, then set `pnpmWorkspaces = [ "<workspace project name 1>" "<workspace project name 2>" ]`, etc, in your `pnpm.fetchDeps` call,
505which will make PNPM only install dependencies for those workspace packages.
506
507For example:
508
509```nix
510{
511 # ...
512 pnpmWorkspaces = [ "@astrojs/language-server" ];
513 pnpmDeps = pnpm.fetchDeps {
514 inherit (finalAttrs) pnpmWorkspaces;
515 #...
516 };
517}
518```
519
520The above would make `pnpm.fetchDeps` call only install dependencies for the `@astrojs/language-server` workspace package.
521Note that you do not need to set `sourceRoot` to make this work.
522
523Usually, in such cases, you'd want to use `pnpm --filter=<pnpm workspace name> build` to build your project, as `npmHooks.npmBuildHook` probably won't work. A `buildPhase` based on the following example will probably fit most workspace projects:
524
525```nix
526{
527 buildPhase = ''
528 runHook preBuild
529
530 pnpm --filter=@astrojs/language-server build
531
532 runHook postBuild
533 '';
534}
535```
536
537#### Additional PNPM Commands and settings {#javascript-pnpm-extraCommands}
538
539If you require setting an additional PNPM configuration setting (such as `dedupe-peer-dependents` or similar),
540set `prePnpmInstall` to the right commands to run. For example:
541
542```nix
543{
544 prePnpmInstall = ''
545 pnpm config set dedupe-peer-dependants false
546 '';
547 pnpmDeps = pnpm.fetchDeps {
548 inherit (finalAttrs) prePnpmInstall;
549 # ...
550 };
551}
552```
553
554In this example, `prePnpmInstall` will be run by both `pnpm.configHook` and by the `pnpm.fetchDeps` builder.
555
556#### PNPM `fetcherVersion` {#javascript-pnpm-fetcherVersion}
557
558This is the version of the output of `pnpm.fetchDeps`, if you haven't set it already, you can use `1` with your current hash:
559
560```nix
561{
562 # ...
563 pnpmDeps = pnpm.fetchDeps {
564 # ...
565 fetcherVersion = 1;
566 hash = "..."; # you can use your already set hash here
567 };
568}
569```
570
571After upgrading to a newer `fetcherVersion`, you need to regenerate the hash:
572
573```nix
574{
575 # ...
576 pnpmDeps = pnpm.fetchDeps {
577 # ...
578 fetcherVersion = 2;
579 hash = "..."; # clear this hash and generate a new one
580 };
581}
582```
583
584This variable ensures that we can make changes to the output of `pnpm.fetchDeps` without breaking existing hashes.
585Changes can include workarounds or bug fixes to existing PNPM issues.
586
587##### Version history {#javascript-pnpm-fetcherVersion-versionHistory}
588
589- 1: Initial version, nothing special
590- 2: [Ensure consistent permissions](https://github.com/NixOS/nixpkgs/pull/422975)
591
592### Yarn {#javascript-yarn}
593
594Yarn based projects use a `yarn.lock` file instead of a `package-lock.json` to pin dependencies.
595
596To package yarn-based applications, you need to distinguish by the version pointers in the `yarn.lock` file. See the following sections.
597
598#### Yarn v1 {#javascript-yarn-v1}
599
600Yarn v1 lockfiles contain a comment `# yarn lockfile v1` at the beginning of the file.
601
602Nixpkgs provides the Nix function `fetchYarnDeps` which fetches an offline cache suitable for running `yarn install` before building the project. In addition, Nixpkgs provides the hooks:
603
604- `yarnConfigHook`: Fetches the dependencies from the offline cache and installs them into `node_modules`.
605- `yarnBuildHook`: Runs `yarn build` or a specified `yarn` command that builds the project.
606- `yarnInstallHook`: Runs `yarn install --production` to prune dependencies and installs the project into `$out`.
607
608An example usage of the above attributes is:
609
610```nix
611{
612 lib,
613 stdenv,
614 fetchFromGitHub,
615 fetchYarnDeps,
616 yarnConfigHook,
617 yarnBuildHook,
618 yarnInstallHook,
619 nodejs,
620}:
621
622stdenv.mkDerivation (finalAttrs: {
623 pname = "...";
624 version = "...";
625
626 src = fetchFromGitHub {
627 owner = "...";
628 repo = "...";
629 rev = "v${finalAttrs.version}";
630 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
631 };
632
633 yarnOfflineCache = fetchYarnDeps {
634 yarnLock = finalAttrs.src + "/yarn.lock";
635 hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";
636 };
637
638 nativeBuildInputs = [
639 yarnConfigHook
640 yarnBuildHook
641 yarnInstallHook
642 # Needed for executing package.json scripts
643 nodejs
644 ];
645
646 meta = {
647 # ...
648 };
649})
650```
651
652##### `yarnConfigHook` arguments {#javascript-yarnconfighook}
653
654By default, `yarnConfigHook` relies upon the attribute `${yarnOfflineCache}` (or `${offlineCache}` if the former is not set) to find the location of the offline cache produced by `fetchYarnDeps`. To disable this phase, you can set `dontYarnInstallDeps = true` or override the `configurePhase`.
655
656##### `yarnBuildHook` arguments {#javascript-yarnbuildhook}
657
658This script by default runs `yarn --offline build`, and it relies upon the project's dependencies installed at `node_modules`. Below is a list of additional `mkDerivation` arguments read by this hook:
659
660- `yarnBuildScript`: Sets a different `yarn --offline` subcommand (defaults to `build`).
661- `yarnBuildFlags`: Single string list of additional flags to pass the above command, or a Nix list of such additional flags.
662
663##### `yarnInstallHook` arguments {#javascript-yarninstallhook}
664
665To install the package `yarnInstallHook` uses both `npm` and `yarn` to cleanup project files and dependencies. To disable this phase, you can set `dontYarnInstall = true` or override the `installPhase`. Below is a list of additional `mkDerivation` arguments read by this hook:
666
667- `yarnKeepDevDeps`: Disables the removal of devDependencies from `node_modules` before installation.
668
669#### yarn2nix {#javascript-yarn2nix}
670
671> [!WARNING]
672> The `yarn2nix` functions have been deprecated in favor of `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook` (for Yarn v1) and `yarn-berry_*.*` tooling (Yarn v3 and v4). Documentation for `yarn2nix` functions still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
673
674##### Preparation {#javascript-yarn2nix-preparation}
675
676You will need at least a `yarn.lock` file. If upstream does not have one you need to generate it and reference it in your package definition.
677
678If the downloaded files contain the `package.json` and `yarn.lock` files they can be used like this:
679
680```nix
681{
682 offlineCache = fetchYarnDeps {
683 yarnLock = src + "/yarn.lock";
684 hash = "....";
685 };
686}
687```
688
689##### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
690
691> [!WARNING]
692> The `mkYarnPackage` functions have been deprecated in favor of `yarnConfigHook`, `yarnBuildHook` and `yarnInstallHook` (for Yarn v1) and `yarn-berry_*.*` tooling (Yarn v3 and v4). Documentation for `mkYarnPackage` functions still appears here for the sake of the packages that still use them. See also a tracking issue [#324246](https://github.com/NixOS/nixpkgs/issues/324246).
693
694`mkYarnPackage` will by default try to generate a binary. For packages only generating static assets (Svelte, Vue, React, Webpack, ...), you will need to explicitly override the build step with your instructions.
695
696It's important to use the `--offline` flag. For example if you script is `"build": "something"` in `package.json` use:
697
698```nix
699{
700 nativeBuildInputs = [ writableTmpDirAsHomeHook ];
701
702 buildPhase = ''
703 runHook preBuild
704
705 yarn --offline build
706
707 runHook postBuild
708 '';
709}
710```
711
712The `distPhase` is packing the package's dependencies in a tarball using `yarn pack`. You can disable it using:
713
714```nix
715{ doDist = false; }
716```
717
718The configure phase can sometimes fail because it makes many assumptions that may not always apply. One common override is:
719
720```nix
721{
722 configurePhase = ''
723 runHook preConfigure
724
725 ln -s $node_modules node_modules
726
727 runHook postConfigure
728 '';
729}
730```
731
732or if you need a writeable node_modules directory:
733
734```nix
735{
736 configurePhase = ''
737 runHook preConfigure
738
739 cp -r $node_modules node_modules
740 chmod +w node_modules
741
742 runHook postConfigure
743 '';
744}
745```
746
747##### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
748
749This will generate a derivation including the `node_modules` directory.
750If you have to build a derivation for an integrated web framework (Rails, Phoenix, etc.), this is probably the easiest way.
751
752#### Overriding dependency behavior {#javascript-mkYarnPackage-overriding-dependencies}
753
754In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building.
755
756For instance, say your package is throwing errors when trying to invoke node-sass:
757
758```
759ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'
760```
761
762To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want:
763
764```nix
765mkYarnPackage rec {
766 pkgConfig = {
767 node-sass = {
768 buildInputs = with final; [
769 python
770 libsass
771 pkg-config
772 ];
773 postInstall = ''
774 LIBSASS_EXT=auto yarn --offline run build
775 rm build/config.gypi
776 '';
777 };
778 };
779}
780```
781
782##### Pitfalls {#javascript-yarn2nix-pitfalls}
783
784- If version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
785- Having trouble with `node-gyp`? Try adding these lines to the `yarnPreBuild` steps:
786
787 ```nix
788 {
789 yarnPreBuild = ''
790 mkdir -p $HOME/.node-gyp/${nodejs.version}
791 echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion
792 ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version}
793 export npm_config_nodedir=${nodejs}
794 '';
795 }
796 ```
797
798 - The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496>
799 - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
800- `offlineCache` (described [above](#javascript-yarn2nix-preparation)) must be specified to avoid [Import From Derivation](#ssec-import-from-derivation) (IFD) when used inside Nixpkgs.
801
802#### Yarn Berry v3/v4 {#javascript-yarn-v3-v4}
803Yarn Berry (v3 / v4) have similar formats, they start with blocks like these:
804
805```yaml
806__metadata:
807 version: 6
808 cacheKey: 8[cX]
809```
810
811```yaml
812__metadata:
813 version: 8
814 cacheKey: 10[cX]
815```
816
817For these packages, we have some helpers exposed under the respective `yarn-berry_3` and `yarn-berry_4` packages:
818
819- `yarn-berry-fetcher`
820- `fetchYarnBerryDeps`
821- `yarnBerryConfigHook`
822
823It's recommended to ensure you're explicitly pinning the major version used, for example by capturing the `yarn-berry_Xn` argument and then re-defining it as a `yarn-berry` `let` binding.
824
825```nix
826{
827 stdenv,
828 nodejs,
829 yarn-berry_4,
830}:
831
832let
833 yarn-berry = yarn-berry_4;
834
835in
836stdenv.mkDerivation (finalAttrs: {
837 pname = "foo";
838 version = "0-unstable-1980-01-01";
839
840 src = {
841 #...
842 };
843
844 nativeBuildInputs = [
845 nodejs
846 yarn-berry.yarnBerryConfigHook
847 ];
848
849 offlineCache = yarn-berry.fetchYarnBerryDeps {
850 inherit (finalAttrs) src;
851 hash = "...";
852 };
853})
854```
855
856##### `yarn-berry_X.fetchYarnBerryDeps` {#javascript-fetchYarnBerryDeps}
857`fetchYarnBerryDeps` runs `yarn-berry-fetcher fetch` in a fixed-output-derivation. It is a custom fetcher designed to reproducibly download all files in the `yarn.lock` file, validating their hashes in the process. For git dependencies, it creates a checkout at `${offlineCache}/checkouts/<40-character-commit-hash>` (relying on the git commit hash to describe the contents of the checkout).
858
859To produce the `hash` argument for `fetchYarnBerryDeps` function call, the `yarn-berry-fetcher prefetch` command can be used:
860
861```console
862$ yarn-berry-fetcher prefetch </path/to/yarn.lock> [/path/to/missing-hashes.json]
863```
864
865This prints the hash to stdout and can be used in update scripts to recalculate the hash for a new version of `yarn.lock`.
866
867##### `yarn-berry_X.yarnBerryConfigHook` {#javascript-yarnBerryConfigHook}
868`yarnBerryConfigHook` uses the store path `offlineCache` points to, to run a `yarn install` during the build, producing a usable `node_modules` directory from the downloaded dependencies.
869
870Internally, this uses a patched version of Yarn to ensure git dependencies are re-packed and any attempted downloads fail immediately.
871
872##### Patching upstream `package.json` or `yarn.lock` files {#javascript-yarnBerry-patching}
873In case patching the upstream `package.json` or `yarn.lock` is needed, it's important to pass `finalAttrs.patches` to `fetchYarnBerryDeps` as well, so the patched variants are picked up (i.e., `inherit (finalAttrs) patches`.
874
875##### Missing hashes in the `yarn.lock` file {#javascript-yarnBerry-missing-hashes}
876Unfortunately, `yarn.lock` files do not include hashes for optional/platform-specific dependencies. This is [by design](https://github.com/yarnpkg/berry/issues/6759).
877
878To compensate for this, the `yarn-berry-fetcher missing-hashes` subcommand can be used to produce all missing hashes. These are usually stored in a `missing-hashes.json` file, which needs to be passed to both the build itself, as well as the `fetchYarnBerryDeps` helper:
879
880```nix
881{
882 stdenv,
883 nodejs,
884 yarn-berry_4,
885}:
886
887let
888 yarn-berry = yarn-berry_4;
889
890in
891stdenv.mkDerivation (finalAttrs: {
892 pname = "foo";
893 version = "0-unstable-1980-01-01";
894
895 src = {
896 #...
897 };
898
899 nativeBuildInputs = [
900 nodejs
901 yarn-berry.yarnBerryConfigHook
902 ];
903
904 missingHashes = ./missing-hashes.json;
905 offlineCache = yarn-berry.fetchYarnBerryDeps {
906 inherit (finalAttrs) src missingHashes;
907 hash = "...";
908 };
909})
910```
911
912## Outside Nixpkgs {#javascript-outside-nixpkgs}
913
914There are some other tools available, which are written in the Nix language.
915These can't be used inside Nixpkgs because they require [Import From Derivation](#ssec-import-from-derivation), which is not allowed in Nixpkgs.
916
917If you are packaging something outside Nixpkgs, consider the following:
918
919### npmlock2nix {#javascript-npmlock2nix}
920
921[npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might be subject to change.
922
923#### Pitfalls {#javascript-npmlock2nix-pitfalls}
924
925There are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45).
926
927### nix-npm-buildpackage {#javascript-nix-npm-buildpackage}
928
929[nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might change. It supports both `package-lock.json` and yarn.lock.
930
931#### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls}
932
933There are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33).