pyrox.dev
1# Javascript {#language-javascript}
2
3## Introduction {#javascript-introduction}
4
5This contains instructions on how to package javascript applications.
6
7The various tools available will be listed in the [tools-overview](#javascript-tools-overview). Some general principles for packaging will follow. Finally some tool specific instructions will be given.
8
9## Getting unstuck / finding code examples {#javascript-finding-examples}
10
11If you find you are lacking inspiration for packing javascript applications, the links below might prove useful. Searching online for prior art can be helpful if you are running into solved problems.
12
13### Github {#javascript-finding-examples-github}
14
15- Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code>
16- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+filename%3Aflake.nix&type=code>
17
18### Gitlab {#javascript-finding-examples-gitlab}
19
20- Searching Nix files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix>
21- Searching just `flake.nix` files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix>
22
23## Tools overview {#javascript-tools-overview}
24
25## General principles {#javascript-general-principles}
26
27The following principles are given in order of importance with potential exceptions.
28
29### Try to use the same node version used upstream {#javascript-upstream-node-version}
30
31It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging.
32
33This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node. Some cryptic errors regarding V8 may appear.
34
35### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager}
36
37A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of node_modules for each tool.
38
39Guidelines of package managers, recommend to commit those lock files to the repos. If a particular lock file is present, it is a strong indication of which package manager is used upstream.
40
41It's better to try to use a Nix tool that understand the lock file. Using a different tool might give you hard to understand error because different packages have been installed. An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629). Upstream use NPM, but this is an attempt to package it with `yarn2nix` (that uses yarn.lock).
42
43Using a different tool forces to commit a lock file to the repository. Those files are fairly large, so when packaging for nixpkgs, this approach does not scale well.
44
45Exceptions to this rule are:
46
47- When you encounter one of the bugs from a Nix tool. In each of the tool specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to recreate a lock file and commit it to nixpkgs. In general `yarn2nix` has less known problems and so a simple search in nixpkgs will reveal many yarn.lock files committed.
48- Some lock files contain particular version of a package that has been pulled off NPM for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs.
49- The only tool that supports workspaces (a feature of NPM that helps manage sub-directories with different package.json from a single top level package.json) is `yarn2nix`. If upstream has workspaces you should try `yarn2nix`.
50
51### Try to use upstream package.json {#javascript-upstream-package-json}
52
53Exceptions to this rule are:
54
55- Sometimes the upstream repo assumes some dependencies be installed globally. In that case you can add them manually to the upstream package.json (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for CLI tools. (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases).
56- Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version by removing the `^`.
57- Sometimes the script defined in the package.json does not work as is. Some scripts for example use CLI tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into,
58
59 ```sh
60 yarn build:ui
61 yarn build:server
62 # OR
63 npm run build:ui
64 npm run build:server
65 ```
66
67 when you need to override a package.json. It's nice to use the one from the upstream source and do some explicit override. Here is an example:
68
69 ```nix
70 patchedPackageJSON = final.runCommand "package.json" { } ''
71 ${jq}/bin/jq '.version = "0.4.0" |
72 .devDependencies."@jsdoc/cli" = "^0.2.5"
73 ${sonar-src}/package.json > $out
74 '';
75 ```
76
77 You will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see.
78
79### Using node_modules directly {#javascript-using-node_modules}
80
81Each tool has an abstraction to just build the node_modules (dependencies) directory. You can always use the `stdenv.mkDerivation` with the node_modules to build the package (symlink the node_modules directory and then use the package build command). The node_modules abstraction can be also used to build some web framework frontends. For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. `mkYarnModules` to make the derivation containing node_modules. Then when building the frontend you can just symlink the node_modules directory.
82
83## Javascript packages inside nixpkgs {#javascript-packages-nixpkgs}
84
85The [pkgs/development/node-packages](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages) folder contains a generated collection of [NPM packages](https://npmjs.com/) that can be installed with the Nix package manager.
86
87As a rule of thumb, the package set should only provide _end user_ software packages, such as command-line utilities. Libraries should only be added to the package set if there is a non-NPM package that requires it.
88
89When it is desired to use NPM libraries in a development project, use the `node2nix` generator directly on the `package.json` configuration file of the project.
90
91The package set provides support for the official stable Node.js versions. The latest stable LTS release in `nodePackages`, as well as the latest stable current release in `nodePackages_latest`.
92
93If your package uses native addons, you need to examine what kind of native build system it uses. Here are some examples:
94
95- `node-gyp`
96- `node-gyp-builder`
97- `node-pre-gyp`
98
99After you have identified the correct system, you need to override your package expression while adding in build system as a build input. For example, `dat` requires `node-gyp-build`, so we override its expression in [pkgs/development/node-packages/overrides.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/overrides.nix):
100
101```nix
102 dat = prev.dat.override (oldAttrs: {
103 buildInputs = [ final.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ];
104 meta = oldAttrs.meta // { broken = since "12"; };
105 });
106```
107
108### Adding and Updating Javascript packages in nixpkgs {#javascript-adding-or-updating-packages}
109
110To add a package from NPM to nixpkgs:
111
1121. Modify [pkgs/development/node-packages/node-packages.json](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/node-packages.json) to add, update or remove package entries to have it included in `nodePackages` and `nodePackages_latest`.
1132. Run the script:
114
115 ```sh
116 ./pkgs/development/node-packages/generate.sh
117 ```
118
1193. Build your new package to test your changes:
120
121 ```sh
122 nix-build -A nodePackages.<new-or-updated-package>
123 ```
124
125 To build against the latest stable Current Node.js version (e.g. 18.x):
126
127 ```sh
128 nix-build -A nodePackages_latest.<new-or-updated-package>
129 ```
130
131 If the package doesn't build, you may need to add an override as explained above.
1324. If the package's name doesn't match any of the executables it provides, add an entry in [pkgs/development/node-packages/main-programs.nix](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/main-programs.nix). This will be the case for all scoped packages, e.g., `@angular/cli`.
1335. Add and commit all modified and generated files.
134
135For more information about the generation process, consult the [README.md](https://github.com/svanderburg/node2nix) file of the `node2nix` tool.
136
137To update NPM packages in nixpkgs, run the same `generate.sh` script:
138
139```sh
140./pkgs/development/node-packages/generate.sh
141```
142
143#### Git protocol error {#javascript-git-error}
144
145Some packages may have Git dependencies from GitHub specified with `git://`.
146GitHub has [disabled unecrypted Git connections](https://github.blog/2021-09-01-improving-git-protocol-security-github/#no-more-unauthenticated-git), so you may see the following error when running the generate script:
147
148```
149The unauthenticated git protocol on port 9418 is no longer supported
150```
151
152Use the following Git configuration to resolve the issue:
153
154```sh
155git config --global url."https://github.com/".insteadOf git://github.com/
156```
157
158## Tool specific instructions {#javascript-tool-specific}
159
160### buildNpmPackage {#javascript-buildNpmPackage}
161
162`buildNpmPackage` allows you to package npm-based projects in Nixpkgs without the use of an auto-generated dependencies file (as used in [node2nix](#javascript-node2nix)). It works by utilizing npm's cache functionality -- creating a reproducible cache that contains the dependencies of a project, and pointing npm to it.
163
164```nix
165{ lib, buildNpmPackage, fetchFromGitHub }:
166
167buildNpmPackage rec {
168 pname = "flood";
169 version = "4.7.0";
170
171 src = fetchFromGitHub {
172 owner = "jesec";
173 repo = pname;
174 rev = "v${version}";
175 hash = "sha256-BR+ZGkBBfd0dSQqAvujsbgsEPFYw/ThrylxUbOksYxM=";
176 };
177
178 npmDepsHash = "sha256-tuEfyePwlOy2/mOPdXbqJskO6IowvAP4DWg8xSZwbJw=";
179
180 # The prepack script runs the build script, which we'd rather do in the build phase.
181 npmPackFlags = [ "--ignore-scripts" ];
182
183 NODE_OPTIONS = "--openssl-legacy-provider";
184
185 meta = with lib; {
186 description = "A modern web UI for various torrent clients with a Node.js backend and React frontend";
187 homepage = "https://flood.js.org";
188 license = licenses.gpl3Only;
189 maintainers = with maintainers; [ winter ];
190 };
191}
192```
193
194#### Arguments {#javascript-buildNpmPackage-arguments}
195
196* `npmDepsHash`: The output hash of the dependencies for this project. Can be calculated in advance with [`prefetch-npm-deps`](#javascript-buildNpmPackage-prefetch-npm-deps).
197* `makeCacheWritable`: Whether to make the cache writable prior to installing dependencies. Don't set this unless npm tries to write to the cache directory, as it can slow down the build.
198* `npmBuildScript`: The script to run to build the project. Defaults to `"build"`.
199* `npmFlags`: Flags to pass to all npm commands.
200* `npmInstallFlags`: Flags to pass to `npm ci` and `npm prune`.
201* `npmBuildFlags`: Flags to pass to `npm run ${npmBuildScript}`.
202* `npmPackFlags`: Flags to pass to `npm pack`.
203
204#### prefetch-npm-deps {#javascript-buildNpmPackage-prefetch-npm-deps}
205
206`prefetch-npm-deps` can calculate the hash of the dependencies of an npm project ahead of time.
207
208```console
209$ ls
210package.json package-lock.json index.js
211$ prefetch-npm-deps package-lock.json
212...
213sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=
214```
215
216### node2nix {#javascript-node2nix}
217
218#### Preparation {#javascript-node2nix-preparation}
219
220You will need to generate a Nix expression for the dependencies. Don't forget the `-l package-lock.json` if there is a lock file. Most probably you will need the `--development` to include the `devDependencies`
221
222So the command will most likely be:
223```sh
224node2nix --development -l package-lock.json
225```
226
227See `node2nix` [docs](https://github.com/svanderburg/node2nix) for more info.
228
229#### Pitfalls {#javascript-node2nix-pitfalls}
230
231- If upstream package.json does not have a "version" attribute, `node2nix` will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json).
232- `node2nix` has some [bugs](https://github.com/svanderburg/node2nix/issues/238) related to working with lock files from NPM distributed with `nodejs_16`.
233- `node2nix` does not like missing packages from NPM. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of NPM.
234
235### yarn2nix {#javascript-yarn2nix}
236
237#### Preparation {#javascript-yarn2nix-preparation}
238
239You will need at least a `yarn.lock` file. If upstream does not have one you need to generate it and reference it in your package definition.
240
241If the downloaded files contain the `package.json` and `yarn.lock` files they can be used like this:
242
243```nix
244offlineCache = fetchYarnDeps {
245 yarnLock = src + "/yarn.lock";
246 hash = "....";
247};
248```
249
250#### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
251
252`mkYarnPackage` will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React, WebPack, ...), you will need to explicitly override the build step with your instructions.
253
254It's important to use the `--offline` flag. For example if you script is `"build": "something"` in `package.json` use:
255
256```nix
257buildPhase = ''
258 export HOME=$(mktemp -d)
259 yarn --offline build
260'';
261```
262
263The dist phase is also trying to build a binary, the only way to override it is with:
264
265```nix
266distPhase = "true";
267```
268
269The configure phase can sometimes fail because it makes many assumptions which may not always apply. One common override is:
270
271```nix
272configurePhase = ''
273 ln -s $node_modules node_modules
274'';
275```
276
277or if you need a writeable node_modules directory:
278
279```nix
280configurePhase = ''
281 cp -r $node_modules node_modules
282 chmod +w node_modules
283'';
284```
285
286#### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
287
288This will generate a derivation including the `node_modules` directory.
289If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way.
290
291#### Overriding dependency behavior {#javascript-mkYarnPackage-overriding-dependencies}
292
293In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building.
294
295For instance, say your package is throwing errors when trying to invoke node-sass:
296
297```
298ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'
299```
300
301To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want:
302
303```nix
304mkYarnPackage rec {
305 pkgConfig = {
306 node-sass = {
307 buildInputs = with final;[ python libsass pkg-config ];
308 postInstall = ''
309 LIBSASS_EXT=auto yarn --offline run build
310 rm build/config.gypi
311 '';
312 };
313 };
314}
315```
316
317#### Pitfalls {#javascript-yarn2nix-pitfalls}
318
319- If version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
320- Having trouble with `node-gyp`? Try adding these lines to the `yarnPreBuild` steps:
321
322 ```nix
323 yarnPreBuild = ''
324 mkdir -p $HOME/.node-gyp/${nodejs.version}
325 echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion
326 ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version}
327 export npm_config_nodedir=${nodejs}
328 '';
329 ```
330
331 - The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496>
332 - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
333
334## Outside Nixpkgs {#javascript-outside-nixpkgs}
335
336There are some other tools available, which are written in the Nix language.
337These that can't be used inside Nixpkgs because they require [Import From Derivation](#ssec-import-from-derivation), which is not allowed in Nixpkgs.
338
339If you are packaging something outside Nixpkgs, consider the following:
340
341### npmlock2nix {#javascript-npmlock2nix}
342
343[npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might be subject to change.
344
345#### Pitfalls {#javascript-npmlock2nix-pitfalls}
346
347There are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45).
348
349### nix-npm-buildpackage {#javascript-nix-npm-buildpackage}
350
351[nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building `node_modules` without code generation. It hasn't reached v1 yet, the API might change. It supports both `package-lock.json` and yarn.lock.
352
353#### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls}
354
355There are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33).