pyrox.dev
1# Javascript {#language-javascript}
2
3## Introduction {#javascript-introduction}
4
5This contains instructions on how to package javascript applications.
6
7The various tools available will be listed in the [tools-overview](#javascript-tools-overview). Some general principles for packaging will follow. Finally some tool specific instructions will be given.
8
9## Getting unstuck / finding code examples
10
11If you find you are lacking inspiration for packing javascript applications, the links below might prove useful.
12Searching online for prior art can be helpful if you are running into solved problems.
13
14### Github
15
16- Searching Nix files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+language%3ANix&type=code>
17
18- Searching just `flake.nix` files for `mkYarnPackage`: <https://github.com/search?q=mkYarnPackage+filename%3Aflake.nix&type=code>
19
20### Gitlab
21
22- Searching Nix files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix>
23
24- Searching just `flake.nix` files for `mkYarnPackage`: <https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix>
25
26## Tools overview {#javascript-tools-overview}
27
28## General principles {#javascript-general-principles}
29
30The following principles are given in order of importance with potential exceptions.
31
32### Try to use the same node version used upstream {#javascript-upstream-node-version}
33
34It is often not documented which node version is used upstream, but if it is, try to use the same version when packaging.
35
36This can be a problem if upstream is using the latest and greatest and you are trying to use an earlier version of node. Some cryptic errors regarding V8 may appear.
37
38An exception to this:
39
40### Try to respect the package manager originally used by upstream (and use the upstream lock file) {#javascript-upstream-package-manager}
41
42A lock file (package-lock.json, yarn.lock...) is supposed to make reproducible installations of node_modules for each tool.
43
44Guidelines of package managers, recommend to commit those lock files to the repos. If a particular lock file is present, it is a strong indication of which package manager is used upstream.
45
46It's better to try to use a nix tool that understand the lock file. Using a different tool might give you hard to understand error because different packages have been installed. An example of problems that could arise can be found [here](https://github.com/NixOS/nixpkgs/pull/126629). Upstream uses npm, but this is an attempt to package it with yarn2nix (that uses yarn.lock)
47
48Using a different tool forces to commit a lock file to the repository. Those files are fairly large, so when packaging for nixpkgs, this approach does not scale well.
49
50Exceptions to this rule are:
51
52- when you encounter one of the bugs from a nix tool. In each of the tool specific instructions, known problems will be detailed. If you have a problem with a particular tool, then it's best to try another tool, even if this means you will have to recreate a lock file and commit it to nixpkgs. In general yarn2nix has less known problems and so a simple search in nixpkgs will reveal many yarn.lock files committed
53- Some lock files contain particular version of a package that has been pulled off npm for some reason. In that case, you can recreate upstream lock (by removing the original and `npm install`, `yarn`, ...) and commit this to nixpkgs.
54- The only tool that supports workspaces (a feature of npm that helps manage sub-directories with different package.json from a single top level package.json) is yarn2nix. If upstream has workspaces you should try yarn2nix.
55
56### Try to use upstream package.json {#javascript-upstream-package-json}
57
58Exceptions to this rule are
59
60- Sometimes the upstream repo assumes some dependencies be installed globally. In that case you can add them manually to the upstream package.json (`yarn add xxx` or `npm install xxx`, ...). Dependencies that are installed locally can be executed with `npx` for cli tools. (e.g. `npx postcss ...`, this is how you can call those dependencies in the phases).
61- Sometimes there is a version conflict between some dependency requirements. In that case you can fix a version (by removing the `^`).
62- Sometimes the script defined in the package.json does not work as is. Some scripts for example use cli tools that might not be available, or cd in directory with a different package.json (for workspaces notably). In that case, it's perfectly fine to look at what the particular script is doing and break this down in the phases. In the build script you can see `build:*` calling in turns several other build scripts like `build:ui` or `build:server`. If one of those fails, you can try to separate those into:
63
64```Shell
65yarn build:ui
66yarn build:server
67# OR
68npm run build:ui
69npm run build:server
70```
71
72when you need to override a package.json. It's nice to use the one from the upstream src and do some explicit override. Here is an example.
73
74```nix
75patchedPackageJSON = final.runCommand "package.json" { } ''
76 ${jq}/bin/jq '.version = "0.4.0" |
77 .devDependencies."@jsdoc/cli" = "^0.2.5"
78 ${sonar-src}/package.json > $out
79'';
80```
81
82you will still need to commit the modified version of the lock files, but at least the overrides are explicit for everyone to see.
83
84### Using node_modules directly {#javascript-using-node_modules}
85
86each tool has an abstraction to just build the node_modules (dependencies) directory. you can always use the stdenv.mkDerivation with the node_modules to build the package (symlink the node_modules directory and then use the package build command). the node_modules abstraction can be also used to build some web framework frontends. For an example of this see how [plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix) is built. mkYarnModules to make the derivation containing node_modules. Then when building the frontend you can just symlink the node_modules directory
87
88## javascript packages inside nixpkgs {#javascript-packages-nixpkgs}
89
90The `pkgs/development/node-packages` folder contains a generated collection of
91[NPM packages](https://npmjs.com/) that can be installed with the Nix package
92manager.
93
94As a rule of thumb, the package set should only provide _end user_ software
95packages, such as command-line utilities. Libraries should only be added to the
96package set if there is a non-NPM package that requires it.
97
98When it is desired to use NPM libraries in a development project, use the
99`node2nix` generator directly on the `package.json` configuration file of the
100project.
101
102The package set provides support for the official stable Node.js versions.
103The latest stable LTS release in `nodePackages`, as well as the latest stable
104Current release in `nodePackages_latest`.
105
106If your package uses native addons, you need to examine what kind of native
107build system it uses. Here are some examples:
108
109- `node-gyp`
110- `node-gyp-builder`
111- `node-pre-gyp`
112
113After you have identified the correct system, you need to override your package
114expression while adding in build system as a build input. For example, `dat`
115requires `node-gyp-build`, so [we override](https://github.com/NixOS/nixpkgs/blob/32f5e5da4a1b3f0595527f5195ac3a91451e9b56/pkgs/development/node-packages/default.nix#L37-L40) its expression in [`default.nix`](https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/node-packages/default.nix):
116
117```nix
118 dat = super.dat.override {
119 buildInputs = [ self.node-gyp-build pkgs.libtool pkgs.autoconf pkgs.automake ];
120 meta.broken = since "12";
121 };
122```
123
124To add a package from NPM to nixpkgs:
125
1261. Modify `pkgs/development/node-packages/node-packages.json` to add, update
127 or remove package entries to have it included in `nodePackages` and
128 `nodePackages_latest`.
1292. Run the script: `cd pkgs/development/node-packages && ./generate.sh`.
1303. Build your new package to test your changes:
131 `cd /path/to/nixpkgs && nix-build -A nodePackages.<new-or-updated-package>`.
132 To build against the latest stable Current Node.js version (e.g. 14.x):
133 `nix-build -A nodePackages_latest.<new-or-updated-package>`
1344. Add and commit all modified and generated files.
135
136For more information about the generation process, consult the
137[README.md](https://github.com/svanderburg/node2nix) file of the `node2nix`
138tool.
139
140## Tool specific instructions {#javascript-tool-specific}
141
142### node2nix {#javascript-node2nix}
143
144#### Preparation {#javascript-node2nix-preparation}
145
146you will need to generate a nix expression for the dependencies
147
148- don't forget the `-l package-lock.json` if there is a lock file
149- Most probably you will need the `--development` to include the `devDependencies`
150
151so the command will most likely be
152`node2nix --development -l package-lock.json`
153
154[link to the doc in the repo](https://github.com/svanderburg/node2nix)
155
156#### Pitfalls {#javascript-node2nix-pitfalls}
157
158- if upstream package.json does not have a "version" attribute, node2nix will crash. You will need to add it like shown in [the package.json section](#javascript-upstream-package-json)
159- node2nix has some [bugs](https://github.com/svanderburg/node2nix/issues/238). related to working with lock files from npm distributed with nodejs-16_x
160- node2nix does not like missing packages from npm. If you see something like `Cannot resolve version: vue-loader-v16@undefined` then you might want to try another tool. The package might have been pulled off of npm.
161
162### yarn2nix {#javascript-yarn2nix}
163
164#### Preparation {#javascript-yarn2nix-preparation}
165
166you will need at least a yarn.lock and yarn.nix file
167
168- generate a yarn.lock in upstream if it is not already there
169- `yarn2nix > yarn.nix` will generate the dependencies in a nix format
170
171#### mkYarnPackage {#javascript-yarn2nix-mkYarnPackage}
172
173this will by default try to generate a binary. For package only generating static assets (Svelte, Vue, React...), you will need to explicitly override the build step with your instructions. It's important to use the `--offline` flag. For example if you script is `"build": "something"` in package.json use
174
175```nix
176buildPhase = ''
177 yarn build --offline
178'';
179```
180
181The dist phase is also trying to build a binary, the only way to override it is with
182
183```nix
184distPhase = "true";
185```
186
187the configure phase can sometimes fail because it tries to be too clever.
188One common override is
189
190```nix
191configurePhase = "ln -s $node_modules node_modules";
192```
193
194#### mkYarnModules {#javascript-yarn2nix-mkYarnModules}
195
196this will generate a derivation including the node_modules. If you have to build a derivation for an integrated web framework (rails, phoenix..), this is probably the easiest way. [Plausible](https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/web-apps/plausible/default.nix#L39) offers a good example of how to do this.
197
198#### Overriding dependency behavior
199
200In the `mkYarnPackage` record the property `pkgConfig` can be used to override packages when you encounter problems building.
201
202For instance, say your package is throwing errors when trying to invoke node-sass: `ENOENT: no such file or directory, scandir '/build/source/node_modules/node-sass/vendor'`
203
204To fix this we will specify different versions of build inputs to use, as well as some post install steps to get the software built the way we want:
205
206```nix
207mkYarnPackage rec {
208 pkgConfig = {
209 node-sass = {
210 buildInputs = with final;[ python libsass pkg-config ];
211 postInstall = ''
212 LIBSASS_EXT=auto yarn --offline run build
213 rm build/config.gypi
214 '';
215 };
216 };
217}
218```
219
220#### Pitfalls {#javascript-yarn2nix-pitfalls}
221
222- if version is missing from upstream package.json, yarn will silently install nothing. In that case, you will need to override package.json as shown in the [package.json section](#javascript-upstream-package-json)
223
224- having trouble with node-gyp? Try adding these lines to the `yarnPreBuild` steps:
225
226 ```nix
227 yarnPreBuild = ''
228 mkdir -p $HOME/.node-gyp/${nodejs.version}
229 echo 9 > $HOME/.node-gyp/${nodejs.version}/installVersion
230 ln -sfv ${nodejs}/include $HOME/.node-gyp/${nodejs.version}
231 export npm_config_nodedir=${nodejs}
232 '';
233 ```
234
235 - The `echo 9` steps comes from this answer: <https://stackoverflow.com/a/49139496>
236 - Exporting the headers in `npm_config_nodedir` comes from this issue: <https://github.com/nodejs/node-gyp/issues/1191#issuecomment-301243919>
237
238## Outside of nixpkgs {#javascript-outside-nixpkgs}
239
240There are some other options available that can't be used inside nixpkgs. Those other options are written in nix. Importing them in nixpkgs will require moving the source code into nixpkgs. Using [Import From Derivation](https://nixos.wiki/wiki/Import_From_Derivation) is not allowed in hydra at present. If you are packaging something outside nixpkgs, those can be considered
241
242### npmlock2nix {#javascript-npmlock2nix}
243
244[npmlock2nix](https://github.com/nix-community/npmlock2nix) aims at building node_modules without code generation. It hasn't reached v1 yet, the api might be subject to change.
245
246#### Pitfalls {#javascript-npmlock2nix-pitfalls}
247
248- there are some [problems with npm v7](https://github.com/tweag/npmlock2nix/issues/45).
249
250### nix-npm-buildpackage {#javascript-nix-npm-buildpackage}
251
252[nix-npm-buildpackage](https://github.com/serokell/nix-npm-buildpackage) aims at building node_modules without code generation. It hasn't reached v1 yet, the api might change. It supports both package-lock.json and yarn.lock.
253
254#### Pitfalls {#javascript-nix-npm-buildpackage-pitfalls}
255
256- there are some [problems with npm v7](https://github.com/serokell/nix-npm-buildpackage/issues/33).