pyrox.dev
1# lisp-modules {#lisp}
2
3This document describes the Nixpkgs infrastructure for building Common Lisp
4systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System
5Definition Facility). It lives in `pkgs/development/lisp-modules`.
6
7## Overview {#lisp-overview}
8
9The main entry point of the API are the Common Lisp implementation packages
10themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`,
11`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used
12to discover available packages and to build wrappers, respectively.
13
14The `pkgs` attribute set contains packages that were automatically
15[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any
16other [manually defined](#lisp-defining-packages-inside) ones. Not every package
17works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).
18
19The `withPackages` function is of primary utility. It is used to build
20[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built
21[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable,
22and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to
23[find the desired systems on runtime](#lisp-loading-systems).
24
25In addition, Lisps have the `withOverrides` function, which can be used to
26[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of
27their `pkgs`. This will also be useful together with `overrideLispAttrs` when
28[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they
29should stay in the main package and be built by specifying the `systems`
30argument to `build-asdf-system`.
31
32## The 90% use case example {#lisp-use-case-example}
33
34The most common way to use the library is to run ad-hoc wrappers like this:
35
36`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'`
37
38Then, in a shell:
39
40```
41$ sbcl
42* (load (sb-ext:posix-getenv "ASDF"))
43* (asdf:load-system 'alexandria)
44```
45
46Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`:
47
48```nix
49let
50 sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
51in mkShell {
52 packages = [ sbcl' ];
53}
54```
55
56Such a Lisp can be now used e.g. to compile your sources:
57
58```nix
59{
60 buildPhase = ''
61 ${sbcl'}/bin/sbcl --load my-build-file.lisp
62 '';
63}
64```
65
66## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}
67
68To save some work of writing Nix expressions, there is a script that imports all
69the packages distributed by Quicklisp into `imported.nix`. This works by parsing
70its `releases.txt` and `systems.txt` files, which are published every couple of
71months on [quicklisp.org](https://beta.quicklisp.org/dist/quicklisp.txt).
72
73The import process is implemented in the `import` directory as Common Lisp
74code in the `org.lispbuilds.nix` ASDF system. To run the script, one can
75execute `ql-import.lisp`:
76
77```
78cd pkgs/development/lisp-modules
79nix-shell --run 'sbcl --script ql-import.lisp'
80```
81
82The script will:
83
841. Download the latest Quicklisp `systems.txt` and `releases.txt` files
852. Generate a temporary SQLite database of all QL systems in `packages.sqlite`
863. Generate an `imported.nix` file from the database
87
88(The `packages.sqlite` file can be deleted at will, because it is regenerated
89each time the script runs.)
90
91The maintainer's job is to:
92
931. Re-run the `ql-import.lisp` script when there is a new Quicklisp release
942. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix`
953. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix`
96
97Also, the `imported.nix` file **must not be edited manually**! It should only be
98generated as described in this section (by running `ql-import.lisp`).
99
100### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}
101
102The Quicklisp files contain ASDF dependency data, but don't include native
103library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.
104
105The `ql.nix` file contains a long list of overrides, where these dependencies
106can be added.
107
108Packages defined in `packages.nix` contain these dependencies naturally.
109
110### Trusting `systems.txt` and `releases.txt` {#lisp-quicklisp-trusting}
111
112The previous implementation of `lisp-modules` didn't fully trust the Quicklisp
113data, because there were times where the dependencies specified were not
114complete and caused broken builds. It instead used a `nix-shell` environment to
115discover real dependencies by using the ASDF APIs.
116
117The current implementation has chosen to trust this data, because it's faster to
118parse a text file than to build each system to generate its Nix file, and
119because that way packages can be mass-imported. Because of that, there may come
120a day where some packages will break, due to bugs in Quicklisp. In that case,
121the fix could be a manual override in `packages.nix` and `ql.nix`.
122
123A known fact is that Quicklisp doesn't include dependencies on slashy systems in
124its data. This is an example of a situation where such fixes were used, e.g. to
125replace the `systems` attribute of the affected packages. (See the definition of
126`iolib`).
127
128### Quirks {#lisp-quicklisp-quirks}
129
130During Quicklisp import:
131
132- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
133- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base`
134- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)
135- `_` in names is converted to `__` for reversibility
136
137
138## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}
139
140Packages that for some reason are not in Quicklisp, and so cannot be
141auto-imported, or don't work straight from the import, are defined in the
142`packages.nix` file.
143
144In that file, use the `build-asdf-system` function, which is a wrapper around
145`mkDerivation` for building ASDF systems. Various other hacks are present, such
146as `build-with-compile-into-pwd` for systems which create files during
147compilation (such as cl-unicode).
148
149The `build-asdf-system` function is documented
150[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of
151examples of how to use it.
152
153## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}
154
155Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`
156function, which is similar to `build-asdf-system` from `packages.nix`, but is
157part of the public API.
158
159It takes the following arguments:
160
161- `pname`: the package name
162- `version`: the package version
163- `src`: the package source
164- `patches`: patches to apply to the source before build
165- `nativeLibs`: native libraries used by CFFI and grovelling
166- `javaLibs`: Java libraries for ABCL
167- `lispLibs`: dependencies on other packages build with `buildASDFSystem`
168- `systems`: list of systems to build
169
170It can be used to define packages outside Nixpkgs, and, for example, add them
171into the package scope with `withOverrides`.
172
173### Including an external package in scope {#lisp-including-external-pkg-in-scope}
174
175A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the
176Nixpkgs-provided scope like this:
177
178```nix
179let
180 alexandria = sbcl.buildASDFSystem rec {
181 pname = "alexandria";
182 version = "1.4";
183 src = fetchFromGitLab {
184 domain = "gitlab.common-lisp.net";
185 owner = "alexandria";
186 repo = "alexandria";
187 rev = "v${version}";
188 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
189 };
190 };
191 sbcl' = sbcl.withOverrides (self: super: {
192 inherit alexandria;
193 });
194in sbcl'.pkgs.alexandria
195```
196
197## Overriding package attributes {#lisp-overriding-package-attributes}
198
199Packages export the `overrideLispAttrs` function, which can be used to build a
200new package with different parameters.
201
202Example of overriding `alexandria`:
203
204```nix
205sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
206 version = "1.4";
207 src = fetchFromGitLab {
208 domain = "gitlab.common-lisp.net";
209 owner = "alexandria";
210 repo = "alexandria";
211 rev = "v${version}";
212 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
213 };
214})
215```
216
217### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}
218
219Slashy (secondary) systems should not exist in their own packages! Instead, they
220should be included in the parent package as an extra entry in the `systems`
221argument to the `build-asdf-system`/`buildASDFSystem` functions.
222
223The reason is that ASDF searches for a secondary system in the `.asd` of the
224parent package. Thus, having them separate would cause either one of them not to
225load cleanly, because one will contains FASLs of itself but not the other, and
226vice versa.
227
228To package slashy systems, use `overrideLispAttrs`, like so:
229
230```nix
231ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
232 systems = oldAttrs.systems ++ [ "alexandria/tests" ];
233 lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
234})
235```
236
237See the [respective section](#lisp-including-external-pkg-in-scope) on using
238`withOverrides` for how to weave it back into `ecl.pkgs`.
239
240Note that sometimes the slashy systems might not only have more dependencies
241than the main one, but create a circular dependency between `.asd`
242files. Unfortunately, in this case an adhoc solution becomes necessary.
243
244## Building Wrappers {#lisp-building-wrappers}
245
246Wrappers can be built using the `withPackages` function of Common Lisp
247implementations (`abcl`, `ecl`, `sbcl` etc.):
248
249```
250nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])'
251```
252
253Such a wrapper can then be used like this:
254
255```
256$ sbcl
257* (load (sb-ext:posix-getenv "ASDF"))
258* (asdf:load-system 'alexandria)
259* (asdf:load-system 'bordeaux-threads)
260```
261
262### Loading ASDF {#lisp-loading-asdf}
263
264For best results, avoid calling `(require 'asdf)` When using the
265library-generated wrappers.
266
267Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of
268getting an environment variable for `ext:getenv`. This will load the
269(pre-compiled to FASL) Nixpkgs-provided version of ASDF.
270
271### Loading systems {#lisp-loading-systems}
272
273There, you can use `asdf:load-system`. This works by setting the right
274values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment
275variables, so that systems are found in the Nix store and pre-compiled FASLs are
276loaded.
277
278## Adding a new Lisp {#lisp-adding-a-new-lisp}
279
280The function `wrapLisp` is used to wrap Common Lisp implementations. It adds the
281`pkgs`, `withPackages`, `withOverrides` and `buildASDFSystem` attributes to the
282derivation.
283
284`wrapLisp` takes these arguments:
285
286- `pkg`: the Lisp package
287- `faslExt`: Implementation-specific extension for FASL files
288- `program`: The name of executable file in `${pkg}/bin/` (Default: `pkg.pname`)
289- `flags`: A list of flags to always pass to `program` (Default: `[]`)
290- `asdf`: The ASDF version to use (Default: `pkgs.asdf_3_3`)
291- `packageOverrides`: Package overrides config (Default: `(self: super: {})`)
292
293This example wraps CLISP:
294
295```nix
296wrapLisp {
297 pkg = clisp;
298 faslExt = "fas";
299 flags = ["-E" "UTF8"];
300}
301```