pyrox.dev
1# lisp-modules {#lisp}
2
3This document describes the Nixpkgs infrastructure for building Common Lisp
4systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System
5Definition Facility). It lives in `pkgs/development/lisp-modules`.
6
7## Overview {#lisp-overview}
8
9The main entry point of the API are the Common Lisp implementation packages
10themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`,
11`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used
12to discover available packages and to build wrappers, respectively.
13
14The `pkgs` attribute set contains packages that were automatically
15[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any
16other [manually defined](#lisp-defining-packages-inside) ones. Not every package
17works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).
18
19The `withPackages` function is of primary utility. It is used to build
20[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built
21[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable,
22and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to
23[find the desired systems on runtime](#lisp-loading-systems).
24
25In addition, Lisps have the `withOverrides` function, which can be used to
26[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of
27their `pkgs`. This will also be useful together with `overrideLispAttrs` when
28[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they
29should stay in the main package and be built by specifying the `systems`
30argument to `build-asdf-system`.
31
32## The 90% use case example {#lisp-use-case-example}
33
34The most common way to use the library is to run ad-hoc wrappers like this:
35
36`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'`
37
38Then, in a shell:
39
40```
41$ sbcl
42* (load (sb-ext:posix-getenv "ASDF"))
43* (asdf:load-system 'alexandria)
44```
45
46Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`:
47
48```nix
49let
50 sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
51in
52mkShell {
53 packages = [ sbcl' ];
54}
55```
56
57Such a Lisp can be now used e.g. to compile your sources:
58
59```nix
60{
61 buildPhase = ''
62 ${sbcl'}/bin/sbcl --load my-build-file.lisp
63 '';
64}
65```
66
67## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}
68
69To save some work of writing Nix expressions, there is a script that imports all
70the packages distributed by Quicklisp into `imported.nix`. This works by parsing
71its `releases.txt` and `systems.txt` files, which are published every couple of
72months on [quicklisp.org](https://beta.quicklisp.org/dist/quicklisp.txt).
73
74The import process is implemented in the `import` directory as Common Lisp
75code in the `org.lispbuilds.nix` ASDF system. To run the script, one can
76execute `ql-import.lisp`:
77
78```
79cd pkgs/development/lisp-modules
80nix-shell --run 'sbcl --script ql-import.lisp'
81```
82
83The script will:
84
851. Download the latest Quicklisp `systems.txt` and `releases.txt` files
862. Generate a temporary SQLite database of all QL systems in `packages.sqlite`
873. Generate an `imported.nix` file from the database
88
89(The `packages.sqlite` file can be deleted at will, because it is regenerated
90each time the script runs.)
91
92The maintainer's job is to:
93
941. Re-run the `ql-import.lisp` script when there is a new Quicklisp release
952. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix`
963. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix`
97
98Also, the `imported.nix` file **must not be edited manually**! It should only be
99generated as described in this section (by running `ql-import.lisp`).
100
101### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}
102
103The Quicklisp files contain ASDF dependency data, but don't include native
104library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.
105
106The `ql.nix` file contains a long list of overrides, where these dependencies
107can be added.
108
109Packages defined in `packages.nix` contain these dependencies naturally.
110
111### Trusting `systems.txt` and `releases.txt` {#lisp-quicklisp-trusting}
112
113The previous implementation of `lisp-modules` didn't fully trust the Quicklisp
114data, because there were times where the dependencies specified were not
115complete and caused broken builds. It instead used a `nix-shell` environment to
116discover real dependencies by using the ASDF APIs.
117
118The current implementation has chosen to trust this data, because it's faster to
119parse a text file than to build each system to generate its Nix file, and
120because that way packages can be mass-imported. Because of that, there may come
121a day where some packages will break, due to bugs in Quicklisp. In that case,
122the fix could be a manual override in `packages.nix` and `ql.nix`.
123
124A known fact is that Quicklisp doesn't include dependencies on slashy systems in
125its data. This is an example of a situation where such fixes were used, e.g. to
126replace the `systems` attribute of the affected packages. (See the definition of
127`iolib`).
128
129### Quirks {#lisp-quicklisp-quirks}
130
131During Quicklisp import:
132
133- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
134- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base`
135- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)
136- `_` in names is converted to `__` for reversibility
137
138## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}
139
140Packages that for some reason are not in Quicklisp, and so cannot be
141auto-imported, or don't work straight from the import, are defined in the
142`packages.nix` file.
143
144In that file, use the `build-asdf-system` function, which is a wrapper around
145`mkDerivation` for building ASDF systems. Various other hacks are present, such
146as `build-with-compile-into-pwd` for systems which create files during
147compilation (such as cl-unicode).
148
149The `build-asdf-system` function is documented
150[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of
151examples of how to use it.
152
153## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}
154
155Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`
156function, which is similar to `build-asdf-system` from `packages.nix`, but is
157part of the public API.
158
159It takes the following arguments:
160
161- `pname`: the package name
162- `version`: the package version
163- `src`: the package source
164- `patches`: patches to apply to the source before build
165- `nativeLibs`: native libraries used by CFFI and grovelling
166- `javaLibs`: Java libraries for ABCL
167- `lispLibs`: dependencies on other packages build with `buildASDFSystem`
168- `systems`: list of systems to build
169
170It can be used to define packages outside Nixpkgs, and, for example, add them
171into the package scope with `withOverrides`.
172
173### Including an external package in scope {#lisp-including-external-pkg-in-scope}
174
175A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the
176Nixpkgs-provided scope like this:
177
178```nix
179let
180 alexandria = sbcl.buildASDFSystem rec {
181 pname = "alexandria";
182 version = "1.4";
183 src = fetchFromGitLab {
184 domain = "gitlab.common-lisp.net";
185 owner = "alexandria";
186 repo = "alexandria";
187 tag = "v${version}";
188 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
189 };
190 };
191 sbcl' = sbcl.withOverrides (
192 self: super: {
193 inherit alexandria;
194 }
195 );
196in
197sbcl'.pkgs.alexandria
198```
199
200## Overriding package attributes {#lisp-overriding-package-attributes}
201
202Packages export the `overrideLispAttrs` function, which can be used to build a
203new package with different parameters.
204
205Example of overriding `alexandria`:
206
207```nix
208sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
209 version = "1.4";
210 src = fetchFromGitLab {
211 domain = "gitlab.common-lisp.net";
212 owner = "alexandria";
213 repo = "alexandria";
214 tag = "v${version}";
215 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
216 };
217})
218```
219
220### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}
221
222Slashy (secondary) systems should not exist in their own packages! Instead, they
223should be included in the parent package as an extra entry in the `systems`
224argument to the `build-asdf-system`/`buildASDFSystem` functions.
225
226The reason is that ASDF searches for a secondary system in the `.asd` of the
227parent package. Thus, having them separate would cause either one of them not to
228load cleanly, because one will contains FASLs of itself but not the other, and
229vice versa.
230
231To package slashy systems, use `overrideLispAttrs`, like so:
232
233```nix
234ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
235 systems = oldAttrs.systems ++ [ "alexandria/tests" ];
236 lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
237})
238```
239
240See the [respective section](#lisp-including-external-pkg-in-scope) on using
241`withOverrides` for how to weave it back into `ecl.pkgs`.
242
243Note that sometimes the slashy systems might not only have more dependencies
244than the main one, but create a circular dependency between `.asd`
245files. Unfortunately, in this case an adhoc solution becomes necessary.
246
247## Building Wrappers {#lisp-building-wrappers}
248
249Wrappers can be built using the `withPackages` function of Common Lisp
250implementations (`abcl`, `ecl`, `sbcl` etc.):
251
252```
253nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])'
254```
255
256Such a wrapper can then be used like this:
257
258```
259$ sbcl
260* (load (sb-ext:posix-getenv "ASDF"))
261* (asdf:load-system 'alexandria)
262* (asdf:load-system 'bordeaux-threads)
263```
264
265### Loading ASDF {#lisp-loading-asdf}
266
267For best results, avoid calling `(require 'asdf)` When using the
268library-generated wrappers.
269
270Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of
271getting an environment variable for `ext:getenv`. This will load the
272(pre-compiled to FASL) Nixpkgs-provided version of ASDF.
273
274### Loading systems {#lisp-loading-systems}
275
276There, you can use `asdf:load-system`. This works by setting the right
277values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment
278variables, so that systems are found in the Nix store and pre-compiled FASLs are
279loaded.
280
281## Adding a new Lisp {#lisp-adding-a-new-lisp}
282
283The function `wrapLisp` is used to wrap Common Lisp implementations. It adds the
284`pkgs`, `withPackages`, `withOverrides` and `buildASDFSystem` attributes to the
285derivation.
286
287`wrapLisp` takes these arguments:
288
289- `pkg`: the Lisp package
290- `faslExt`: Implementation-specific extension for FASL files
291- `program`: The name of executable file in `${pkg}/bin/` (Default: `pkg.pname`)
292- `flags`: A list of flags to always pass to `program` (Default: `[]`)
293- `asdf`: The ASDF version to use (Default: `pkgs.asdf_3_3`)
294- `packageOverrides`: Package overrides config (Default: `(self: super: {})`)
295
296This example wraps CLISP:
297
298```nix
299wrapLisp {
300 pkg = clisp;
301 faslExt = "fas";
302 flags = [
303 "-E"
304 "UTF8"
305 ];
306}
307```