pyrox.dev
1# lisp-modules {#lisp}
2
3This document describes the Nixpkgs infrastructure for building Common Lisp
4systems that use [ASDF](https://asdf.common-lisp.dev/) (Another System
5Definition Facility). It lives in `pkgs/development/lisp-modules`.
6
7## Overview {#lisp-overview}
8
9The main entry point of the API are the Common Lisp implementation packages
10themselves (e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp`, `ecl`,
11`sbcl`). They have the `pkgs` and `withPackages` attributes, which can be used
12to discover available packages and to build wrappers, respectively.
13
14The `pkgs` attribute set contains packages that were automatically
15[imported](#lisp-importing-packages-from-quicklisp) from Quicklisp, and any
16other [manually defined](#lisp-defining-packages-inside) ones. Not every package
17works for all the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).
18
19The `withPackages` function is of primary utility. It is used to build
20[runnable wrappers](#lisp-building-wrappers), with a pinned and pre-built
21[ASDF FASL](#lisp-loading-asdf) available in the `ASDF` environment variable,
22and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` configured to
23[find the desired systems on runtime](#lisp-loading-systems).
24
25In addition, Lisps have the `withOverrides` function, which can be used to
26[substitute](#lisp-including-external-pkg-in-scope) any package in the scope of
27their `pkgs`. This will also be useful together with `overrideLispAttrs` when
28[dealing with slashy systems](#lisp-dealing-with-slashy-systems), because they
29should stay in the main package and be built by specifying the `systems`
30argument to `build-asdf-system`.
31
32## The 90% use case example {#lisp-use-case-example}
33
34The most common way to use the library is to run ad-hoc wrappers like this:
35
36`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'`
37
38Then, in a shell:
39
40```
41$ sbcl
42* (load (sb-ext:posix-getenv "ASDF"))
43* (asdf:load-system 'alexandria)
44```
45
46Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`:
47
48```
49let
50 sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
51in mkShell {
52 packages = [ sbcl' ];
53}
54```
55
56Such a Lisp can be now used e.g. to compile your sources:
57
58```
59buildPhase = ''
60 ${sbcl'}/bin/sbcl --load my-build-file.lisp
61''
62```
63
64## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}
65
66To save some work of writing Nix expressions, there is a script that imports all
67the packages distributed by Quicklisp into `imported.nix`. This works by parsing
68its `releases.txt` and `systems.txt` files, which are published every couple of
69months on [quicklisp.org](https://beta.quicklisp.org/dist/quicklisp.txt).
70
71The import process is implemented in the `import` directory as Common Lisp
72code in the `org.lispbuilds.nix` ASDF system. To run the script, one can
73execute `ql-import.lisp`:
74
75```
76cd pkgs/development/lisp-modules
77nix-shell --run 'sbcl --script ql-import.lisp'
78```
79
80The script will:
81
821. Download the latest Quicklisp `systems.txt` and `releases.txt` files
832. Generate a temporary SQLite database of all QL systems in `packages.sqlite`
843. Generate an `imported.nix` file from the database
85
86(The `packages.sqlite` file can be deleted at will, because it is regenerated
87each time the script runs.)
88
89The maintainer's job is to:
90
911. Re-run the `ql-import.lisp` script when there is a new Quicklisp release
922. [Add any missing native dependencies](#lisp-quicklisp-adding-native-dependencies) in `ql.nix`
933. For packages that still don't build, [package them manually](#lisp-defining-packages-inside) in `packages.nix`
94
95Also, the `imported.nix` file **must not be edited manually**! It should only be
96generated as described in this section (by running `ql-import.lisp`).
97
98### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}
99
100The Quicklisp files contain ASDF dependency data, but don't include native
101library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.
102
103The `ql.nix` file contains a long list of overrides, where these dependencies
104can be added.
105
106Packages defined in `packages.nix` contain these dependencies naturally.
107
108### Trusting `systems.txt` and `releases.txt` {#lisp-quicklisp-trusting}
109
110The previous implementation of `lisp-modules` didn't fully trust the Quicklisp
111data, because there were times where the dependencies specified were not
112complete and caused broken builds. It instead used a `nix-shell` environment to
113discover real dependencies by using the ASDF APIs.
114
115The current implementation has chosen to trust this data, because it's faster to
116parse a text file than to build each system to generate its Nix file, and
117because that way packages can be mass-imported. Because of that, there may come
118a day where some packages will break, due to bugs in Quicklisp. In that case,
119the fix could be a manual override in `packages.nix` and `ql.nix`.
120
121A known fact is that Quicklisp doesn't include dependencies on slashy systems in
122its data. This is an example of a situation where such fixes were used, e.g. to
123replace the `systems` attribute of the affected packages. (See the definition of
124`iolib`).
125
126### Quirks {#lisp-quicklisp-quirks}
127
128During Quicklisp import:
129
130- `+` in names is converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
131- `.` in names is converted to `_dot_`: `iolib.base`->`iolib_dot_base`
132- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)
133- `_` in names is converted to `__` for reversibility
134
135
136## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}
137
138Packages that for some reason are not in Quicklisp, and so cannot be
139auto-imported, or don't work straight from the import, are defined in the
140`packages.nix` file.
141
142In that file, use the `build-asdf-system` function, which is a wrapper around
143`mkDerivation` for building ASDF systems. Various other hacks are present, such
144as `build-with-compile-into-pwd` for systems which create files during
145compilation (such as cl-unicode).
146
147The `build-asdf-system` function is documented
148[here](#lisp-defining-packages-outside). Also, `packages.nix` is full of
149examples of how to use it.
150
151## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}
152
153Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`
154function, which is similar to `build-asdf-system` from `packages.nix`, but is
155part of the public API.
156
157It takes the following arguments:
158
159- `pname`: the package name
160- `version`: the package version
161- `src`: the package source
162- `patches`: patches to apply to the source before build
163- `nativeLibs`: native libraries used by CFFI and grovelling
164- `javaLibs`: Java libraries for ABCL
165- `lispLibs`: dependencies on other packages build with `buildASDFSystem`
166- `systems`: list of systems to build
167
168It can be used to define packages outside Nixpkgs, and, for example, add them
169into the package scope with `withOverrides`.
170
171### Including an external package in scope {#lisp-including-external-pkg-in-scope}
172
173A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the
174Nixpkgs-provided scope like this:
175
176```
177let
178 alexandria = sbcl.buildASDFSystem rec {
179 pname = "alexandria";
180 version = "1.4";
181 src = fetchFromGitLab {
182 domain = "gitlab.common-lisp.net";
183 owner = "alexandria";
184 repo = "alexandria";
185 rev = "v${version}";
186 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
187 };
188 };
189 sbcl' = sbcl.withOverrides (self: super: {
190 inherit alexandria;
191 });
192in sbcl'.pkgs.alexandria
193```
194
195## Overriding package attributes {#lisp-overriding-package-attributes}
196
197Packages export the `overrideLispAttrs` function, which can be used to build a
198new package with different parameters.
199
200Example of overriding `alexandria`:
201
202```
203sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
204 version = "1.4";
205 src = fetchFromGitLab {
206 domain = "gitlab.common-lisp.net";
207 owner = "alexandria";
208 repo = "alexandria";
209 rev = "v${version}";
210 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
211 };
212})
213```
214
215### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}
216
217Slashy (secondary) systems should not exist in their own packages! Instead, they
218should be included in the parent package as an extra entry in the `systems`
219argument to the `build-asdf-system`/`buildASDFSystem` functions.
220
221The reason is that ASDF searches for a secondary system in the `.asd` of the
222parent package. Thus, having them separate would cause either one of them not to
223load cleanly, because one will contains FASLs of itself but not the other, and
224vice versa.
225
226To package slashy systems, use `overrideLispAttrs`, like so:
227
228```
229ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
230 systems = oldAttrs.systems ++ [ "alexandria/tests" ];
231 lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
232})
233```
234
235See the [respective section](#lisp-including-external-pkg-in-scope) on using
236`withOverrides` for how to weave it back into `ecl.pkgs`.
237
238Note that sometimes the slashy systems might not only have more dependencies
239than the main one, but create a circular dependency between `.asd`
240files. Unfortunately, in this case an adhoc solution becomes necessary.
241
242## Building Wrappers {#lisp-building-wrappers}
243
244Wrappers can be built using the `withPackages` function of Common Lisp
245implementations (`abcl`, `ecl`, `sbcl` etc.):
246
247```
248nix-shell -p 'sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])'
249```
250
251Such a wrapper can then be used like this:
252
253```
254$ sbcl
255* (load (sb-ext:posix-getenv "ASDF"))
256* (asdf:load-system 'alexandria)
257* (asdf:load-system 'bordeaux-threads)
258```
259
260### Loading ASDF {#lisp-loading-asdf}
261
262For best results, avoid calling `(require 'asdf)` When using the
263library-generated wrappers.
264
265Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of
266getting an environment variable for `ext:getenv`. This will load the
267(pre-compiled to FASL) Nixpkgs-provided version of ASDF.
268
269### Loading systems {#lisp-loading-systems}
270
271There, you can use `asdf:load-system`. This works by setting the right
272values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment
273variables, so that systems are found in the Nix store and pre-compiled FASLs are
274loaded.
275
276## Adding a new Lisp {#lisp-adding-a-new-lisp}
277
278The function `wrapLisp` is used to wrap Common Lisp implementations. It adds the
279`pkgs`, `withPackages`, `withOverrides` and `buildASDFSystem` attributes to the
280derivation.
281
282`wrapLisp` takes these arguments:
283
284- `pkg`: the Lisp package
285- `faslExt`: Implementation-specific extension for FASL files
286- `program`: The name of executable file in `${pkg}/bin/` (Default: `pkg.pname`)
287- `flags`: A list of flags to always pass to `program` (Default: `[]`)
288- `asdf`: The ASDF version to use (Default: `pkgs.asdf_3_3`)
289- `packageOverrides`: Package overrides config (Default: `(self: super: {})`)
290
291This example wraps CLISP:
292
293```
294wrapLisp {
295 pkg = clisp;
296 faslExt = "fas";
297 flags = ["-E" "UTF8"];
298}
299```