pyrox.dev
1# lisp-modules {#lisp}
2
3This document describes the Nixpkgs infrastructure for building Common Lisp
4systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System
5Definition Facility). It lives in `pkgs/development/lisp-modules`.
6
7## Overview {#lisp-overview}
8
9The main entry point of the API are the Common Lisp implementation packages
10themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`,
11`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used
12to discover available packages and to build wrappers, respectively.
13
14The `pkgs` attribute set contains packages that were automatically
15[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any
16other [manually defined](#lisp-defining-packages-inside) ones. Not every package
17works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).
18
19The `withPackages` function is of primary utility. It is used to build
20[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built
21[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable,
22and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to
23[find the desired systems at runtime](#lisp-loading-systems).
24
25In addition, Lisps have the `withOverrides` function, which can be used to
26[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of
27their `pkgs`. This will also be useful together with `overrideLispAttrs` when
28[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they
29should stay in the main package and be built by specifying the `systems`
30argument to `build-asdf-system`.
31
32## The 90% use case example {#lisp-use-case-example}
33
34The most common way to use the library is to run ad-hoc wrappers like this:
35
36`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'`
37
38Then, in a shell:
39
40```
41$ sbcl
42* (load (sb-ext:posix-getenv "ASDF"))
43* (asdf:load-system 'alexandria)
44```
45
46Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`:
47
48```nix
49let
50 sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
51in
52mkShell { packages = [ sbcl' ]; }
53```
54
55Such a Lisp can be now used e.g. to compile your sources:
56
57```nix
58{
59 buildPhase = ''
60 runHook preBuild
61
62 ${sbcl'}/bin/sbcl --load my-build-file.lisp
63
64 runHook postBuild
65 '';
66}
67```
68
69## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}
70
71To save some work of writing Nix expressions, there is a script that imports all
72the packages distributed by Quicklisp into `imported.nix`. This works by parsing
73its `releases.txt` and `systems.txt` files, which are published every couple of
74months on [quicklisp.org](https://beta.quicklisp.org/dist/quicklisp.txt).
75
76The import process is implemented in the `import` directory as Common Lisp
77code in the `org.lispbuilds.nix` ASDF system. To run the script, one can
78execute `ql-import.lisp`:
79
80```
81cd pkgs/development/lisp-modules
82nix-shell --run 'sbcl --script ql-import.lisp'
83```
84
85The script will:
86
871. Download the latest Quicklisp `systems.txt` and `releases.txt` files
882. Generate a temporary SQLite database of all QL systems in `packages.sqlite`
893. Generate an `imported.nix` file from the database
90
91(The `packages.sqlite` file can be deleted at will, because it is regenerated
92each time the script runs.)
93
94The maintainer's job is to:
95
961. Re-run the `ql-import.lisp` script when there is a new Quicklisp release
972. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix`
983. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix`
99
100Also, the `imported.nix` file **must not be edited manually**! It should only be
101generated as described in this section (by running `ql-import.lisp`).
102
103### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}
104
105The Quicklisp files contain ASDF dependency data, but don't include native
106library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.
107
108The `ql.nix` file contains a long list of overrides, where these dependencies
109can be added.
110
111Packages defined in `packages.nix` contain these dependencies naturally.
112
113### Trusting `systems.txt` and `releases.txt` {#lisp-quicklisp-trusting}
114
115The previous implementation of `lisp-modules` didn't fully trust the Quicklisp
116data, because there were times where the dependencies specified were not
117complete and caused broken builds. It instead used a `nix-shell` environment to
118discover real dependencies by using the ASDF APIs.
119
120The current implementation has chosen to trust this data, because it's faster to
121parse a text file than to build each system to generate its Nix file, and
122because that way packages can be mass-imported. Because of that, there may come
123a day where some packages will break, due to bugs in Quicklisp. In that case,
124the fix could be a manual override in `packages.nix` and `ql.nix`.
125
126A known fact is that Quicklisp doesn't include dependencies on slashy systems in
127its data. This is an example of a situation where such fixes were used, e.g. to
128replace the `systems` attribute of the affected packages. (See the definition of
129`iolib`).
130
131### Quirks {#lisp-quicklisp-quirks}
132
133During Quicklisp import:
134
135- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
136- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base`
137- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)
138- `_` in names is converted to `__` for reversibility
139
140## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}
141
142Packages that for some reason are not in Quicklisp, and so cannot be
143auto-imported, or don't work straight from the import, are defined in the
144`packages.nix` file.
145
146In that file, use the `build-asdf-system` function, which is a wrapper around
147`mkDerivation` for building ASDF systems. Various other hacks are present, such
148as `build-with-compile-into-pwd` for systems which create files during
149compilation (such as cl-unicode).
150
151The `build-asdf-system` function is documented
152[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of
153examples of how to use it.
154
155## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}
156
157Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`
158function, which is similar to `build-asdf-system` from `packages.nix`, but is
159part of the public API.
160
161It takes the following arguments:
162
163- `pname`: the package name
164- `version`: the package version
165- `src`: the package source
166- `patches`: patches to apply to the source before build
167- `nativeLibs`: native libraries used by CFFI and grovelling
168- `javaLibs`: Java libraries for ABCL
169- `lispLibs`: dependencies on other packages build with `buildASDFSystem`
170- `systems`: list of systems to build
171
172It can be used to define packages outside Nixpkgs, and, for example, add them
173into the package scope with `withOverrides`.
174
175### Including an external package in scope {#lisp-including-external-pkg-in-scope}
176
177A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the
178Nixpkgs-provided scope like this:
179
180```nix
181let
182 alexandria = sbcl.buildASDFSystem rec {
183 pname = "alexandria";
184 version = "1.4";
185 src = fetchFromGitLab {
186 domain = "gitlab.common-lisp.net";
187 owner = "alexandria";
188 repo = "alexandria";
189 tag = "v${version}";
190 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
191 };
192 };
193 sbcl' = sbcl.withOverrides (self: super: { inherit alexandria; });
194in
195sbcl'.pkgs.alexandria
196```
197
198## Overriding package attributes {#lisp-overriding-package-attributes}
199
200Packages export the `overrideLispAttrs` function, which can be used to build a
201new package with different parameters.
202
203Example of overriding `alexandria`:
204
205```nix
206sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
207 version = "1.4";
208 src = fetchFromGitLab {
209 domain = "gitlab.common-lisp.net";
210 owner = "alexandria";
211 repo = "alexandria";
212 tag = "v${version}";
213 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
214 };
215})
216```
217
218### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}
219
220Slashy (secondary) systems should not exist in their own packages! Instead, they
221should be included in the parent package as an extra entry in the `systems`
222argument to the `build-asdf-system`/`buildASDFSystem` functions.
223
224The reason is that ASDF searches for a secondary system in the `.asd` of the
225parent package. Thus, having them separate would cause either one of them not to
226load cleanly, because one will contain FASLs of itself but not the other, and
227vice versa.
228
229To package slashy systems, use `overrideLispAttrs`, like so:
230
231```nix
232ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
233 systems = oldAttrs.systems ++ [ "alexandria/tests" ];
234 lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
235})
236```
237
238See the [respective section](#lisp-including-external-pkg-in-scope) on using
239`withOverrides` for how to weave it back into `ecl.pkgs`.
240
241Note that sometimes the slashy systems might not only have more dependencies
242than the main one, but create a circular dependency between `.asd`
243files. Unfortunately, in this case an ad-hoc solution becomes necessary.
244
245## Building Wrappers {#lisp-building-wrappers}
246
247Wrappers can be built using the `withPackages` function of Common Lisp
248implementations (`abcl`, `ecl`, `sbcl` etc.):
249
250```
251nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])'
252```
253
254Such a wrapper can then be used like this:
255
256```
257$ sbcl
258* (load (sb-ext:posix-getenv "ASDF"))
259* (asdf:load-system 'alexandria)
260* (asdf:load-system 'bordeaux-threads)
261```
262
263### Loading ASDF {#lisp-loading-asdf}
264
265For best results, avoid calling `(require 'asdf)` when using the
266library-generated wrappers.
267
268Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of
269getting an environment variable for `ext:getenv`. This will load the
270(pre-compiled to FASL) Nixpkgs-provided version of ASDF.
271
272### Loading systems {#lisp-loading-systems}
273
274There, you can use `asdf:load-system`. This works by setting the right
275values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment
276variables, so that systems are found in the Nix store and pre-compiled FASLs are
277loaded.
278
279## Adding a new Lisp {#lisp-adding-a-new-lisp}
280
281The function `wrapLisp` is used to wrap Common Lisp implementations. It adds the
282`pkgs`, `withPackages`, `withOverrides` and `buildASDFSystem` attributes to the
283derivation.
284
285`wrapLisp` takes these arguments:
286
287- `pkg`: the Lisp package
288- `faslExt`: Implementation-specific extension for FASL files
289- `program`: The name of executable file in `${pkg}/bin/` (Default: `pkg.pname`)
290- `flags`: A list of flags to always pass to `program` (Default: `[]`)
291- `asdf`: The ASDF version to use (Default: `pkgs.asdf_3_3`)
292- `packageOverrides`: Package overrides config (Default: `(self: super: {})`)
293
294This example wraps CLISP:
295
296```nix
297wrapLisp {
298 pkg = clisp;
299 faslExt = "fas";
300 flags = [
301 "-E"
302 "UTF8"
303 ];
304}
305```