pyrox.dev
1# lisp-modules {#lisp}
2
3This document describes the Nixpkgs infrastructure for building Common Lisp
4libraries that use ASDF (Another System Definition Facility). It lives in
5`pkgs/development/lisp-modules`.
6
7## Overview {#lisp-overview}
8
9The main entry point of the API are the Common Lisp implementation packages
10(e.g. `abcl`, `ccl`, `clasp-common-lisp`, `clisp` `ecl`, `sbcl`)
11themselves. They have the `pkgs` and `withPackages` attributes, which can be
12used to discover available packages and to build wrappers, respectively.
13
14The `pkgs` attribute set contains packages that were automatically imported from
15Quicklisp, and any other manually defined ones. Not every package works for all
16the CL implementations (e.g. `nyxt` only makes sense for `sbcl`).
17
18The `withPackages` function is of primary utility. It is used to build runnable
19wrappers, with a pinned and pre-built ASDF FASL available in the `ASDF`
20environment variable, and `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS`
21configured to find the desired systems on runtime.
22
23With a few exceptions, the primary thing that the infrastructure does is to run
24`asdf:load-system` for each system specified in the `systems` argument to
25`build-asdf-system`, and save the FASLs to the Nix store. Then, it makes these
26FASLs available to wrappers. Any other use-cases, such as producing SBCL
27executables with `sb-ext:save-lisp-and-die`, are achieved via overriding the
28`buildPhase` etc.
29
30In addition, Lisps have the `withOverrides` function, which can be used to
31substitute any package in the scope of their `pkgs`. This will be useful
32together with `overrideLispAttrs` when dealing with slashy ASDF systems, because
33they should stay in the main package and be build by specifying the `systems`
34argument to `build-asdf-system`.
35
36## The 90% use case example {#lisp-use-case-example}
37
38The most common way to use the library is to run ad-hoc wrappers like this:
39
40`nix-shell -p 'sbcl.withPackages (ps: with ps; [ alexandria ])'`
41
42Then, in a shell:
43
44```
45$ result/bin/sbcl
46* (load (sb-ext:posix-getenv "ASDF"))
47* (asdf:load-system 'alexandria)
48```
49
50Also one can create a `pkgs.mkShell` environment in `shell.nix`/`flake.nix`:
51
52```
53let
54 sbcl' = sbcl.withPackages (ps: [ ps.alexandria ]);
55in mkShell {
56 buildInputs = [ sbcl' ];
57}
58```
59
60Such a Lisp can be now used e.g. to compile your sources:
61
62```
63buildPhase = ''
64 ${sbcl'}/bin/sbcl --load my-build-file.lisp
65''
66```
67
68## Importing packages from Quicklisp {#lisp-importing-packages-from-quicklisp}
69
70The library is able to very quickly import all the packages distributed by
71Quicklisp by parsing its `releases.txt` and `systems.txt` files. These files are
72available from [http://beta.quicklisp.org/dist/quicklisp.txt].
73
74The import process is implemented in the `import` directory as Common Lisp
75functions in the `org.lispbuilds.nix` ASDF system. To run the script, one can
76execute `ql-import.lisp`:
77
78```
79nix-shell --run 'sbcl --script ql-import.lisp'
80```
81
82The script will:
83
841. Download the latest Quicklisp `systems.txt` and `releases.txt` files
852. Generate an SQLite database of all QL systems in `packages.sqlite`
863. Generate an `imported.nix` file from the database
87
88The maintainer's job there is to:
89
901. Re-run the `ql-import.lisp` script
912. Add missing native dependencies in `ql.nix`
923. For packages that still don't build, package them manually in `packages.nix`
93
94Also, the `imported.nix` file **must not be edited manually**! It should only be
95generated as described in this section.
96
97### Adding native dependencies {#lisp-quicklisp-adding-native-dependencies}
98
99The Quicklisp files contain ASDF dependency data, but don't include native
100library (CFFI) dependencies, and, in the case of ABCL, Java dependencies.
101
102The `ql.nix` file contains a long list of overrides, where these dependencies
103can be added.
104
105Packages defined in `packages.nix` contain these dependencies naturally.
106
107### Trusting `systems.txt` and `releases.txt` {#lisp-quicklisp-trusting}
108
109The previous implementation of `lisp-modules` didn't fully trust the Quicklisp
110data, because there were times where the dependencies specified were not
111complete, and caused broken builds. It instead used a `nix-shell` environment to
112discover real dependencies by using the ASDF APIs.
113
114The current implementation has chosen to trust this data, because it's faster to
115parse a text file than to build each system to generate its Nix file, and
116because that way packages can be mass-imported. Because of that, there may come
117a day where some packages will break, due to bugs in Quicklisp. In that case,
118the fix could be a manual override in `packages.nix` and `ql.nix`.
119
120A known fact is that Quicklisp doesn't include dependencies on slashy systems in
121its data. This is an example of a situation where such fixes were used, e.g. to
122replace the `systems` attribute of the affected packages. (See the definition of
123`iolib`).
124
125### Quirks {#lisp-quicklisp-quirks}
126
127During Quicklisp import:
128
129- `+` in names are converted to `_plus{_,}`: `cl+ssl`->`cl_plus_ssl`, `alexandria+`->`alexandria_plus`
130- `.` to `_dot_`: `iolib.base`->`iolib_dot_base`
131- names starting with a number have a `_` prepended (`3d-vectors`->`_3d-vectors`)
132- `_` in names is converted to `__` for reversibility
133
134
135## Defining packages manually inside Nixpkgs {#lisp-defining-packages-inside}
136
137New packages, that for some reason are not in Quicklisp, and so cannot be
138auto-imported, can be written in the `packages.nix` file.
139
140In that file, use the `build-asdf-system` function, which is a wrapper around
141`mkDerivation` for building ASDF systems. Various other hacks are present, such
142as `build-with-compile-into-pwd` for systems which create files during
143compilation.
144
145The `build-asdf-system` function is documented with comments in
146`nix-cl.nix`. Also, `packages.nix` is full of examples of how to use it.
147
148## Defining packages manually outside Nixpkgs {#lisp-defining-packages-outside}
149
150Lisp derivations (`abcl`, `sbcl` etc.) also export the `buildASDFSystem`
151function, which is the same as `build-asdf-system`, except for the `lisp`
152argument which is set to the given CL implementation.
153
154It can be used to define packages outside Nixpkgs, and, for example, add them
155into the package scope with `withOverrides` which will be discussed later on.
156
157### Including an external package in scope {#lisp-including-external-pkg-in-scope}
158
159A package defined outside Nixpkgs using `buildASDFSystem` can be woven into the
160Nixpkgs-provided scope like this:
161
162```
163let
164 alexandria = sbcl.buildASDFSystem rec {
165 pname = "alexandria";
166 version = "1.4";
167 src = fetchFromGitLab {
168 domain = "gitlab.common-lisp.net";
169 owner = "alexandria";
170 repo = "alexandria";
171 rev = "v${version}";
172 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
173 };
174 };
175 sbcl' = sbcl.withOverrides (self: super: {
176 inherit alexandria;
177 });
178in sbcl'.pkgs.alexandria
179```
180
181## Overriding package attributes {#lisp-overriding-package-attributes}
182
183Packages export the `overrideLispAttrs` function, which can be used to build a
184new package with different parameters.
185
186Example of overriding `alexandria`:
187
188```
189sbcl.pkgs.alexandria.overrideLispAttrs (oldAttrs: rec {
190 version = "1.4";
191 src = fetchFromGitLab {
192 domain = "gitlab.common-lisp.net";
193 owner = "alexandria";
194 repo = "alexandria";
195 rev = "v${version}";
196 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
197 };
198})
199```
200
201## Overriding packages in scope {#lisp-overriding-packages-in-scope}
202
203Packages can be woven into a new scope by using `withOverrides`:
204
205```
206let
207 sbcl' = sbcl.withOverrides (self: super: {
208 alexandria = super.alexandria.overrideLispAttrs (oldAttrs: rec {
209 pname = "alexandria";
210 version = "1.4";
211 src = fetchFromGitLab {
212 domain = "gitlab.common-lisp.net";
213 owner = "alexandria";
214 repo = "alexandria";
215 rev = "v${version}";
216 hash = "sha256-1Hzxt65dZvgOFIljjjlSGgKYkj+YBLwJCACi5DZsKmQ=";
217 };
218 });
219 });
220in builtins.elemAt sbcl'.pkgs.bordeaux-threads.lispLibs 0
221```
222
223### Dealing with slashy systems {#lisp-dealing-with-slashy-systems}
224
225Slashy (secondary) systems should not exist in their own packages! Instead, they
226should be included in the parent package as an extra entry in the `systems`
227argument to the `build-asdf-system`/`buildASDFSystem` functions.
228
229The reason is that ASDF searches for a secondary system in the `.asd` of the
230parent package. Thus, having them separate would cause either one of them not to
231load cleanly, because one will contains FASLs of itself but not the other, and
232vice versa.
233
234To package slashy systems, use `overrideLispAttrs`, like so:
235
236```
237ecl.pkgs.alexandria.overrideLispAttrs (oldAttrs: {
238 systems = oldAttrs.systems ++ [ "alexandria/tests" ];
239 lispLibs = oldAttrs.lispLibs ++ [ ecl.pkgs.rt ];
240})
241```
242
243See the respective section on using `withOverrides` for how to weave it back
244into `ecl.pkgs`.
245
246Note that sometimes the slashy systems might not only have more dependencies
247than the main one, but create a circular dependency between `.asd`
248files. Unfortunately, in this case an adhoc solution becomes necessary.
249
250## Building Wrappers {#lisp-building-wrappers}
251
252Wrappers can be built using the `withPackages` function of Common Lisp
253implementations (`abcl`, `ecl`, `sbcl` etc.):
254
255```
256sbcl.withPackages (ps: [ ps.alexandria ps.bordeaux-threads ])
257```
258
259Such a wrapper can then be executed like this:
260
261```
262result/bin/sbcl
263```
264
265### Loading ASDF {#lisp-loading-asdf}
266
267For best results, avoid calling `(require 'asdf)` When using the
268library-generated wrappers.
269
270Use `(load (ext:getenv "ASDF"))` instead, supplying your implementation's way of
271getting an environment variable for `ext:getenv`. This will load the
272(pre-compiled to FASL) Nixpkgs-provided version of ASDF.
273
274### Loading systems {#lisp-loading-systems}
275
276There, you can simply use `asdf:load-system`. This works by setting the right
277values for the `CL_SOURCE_REGISTRY`/`ASDF_OUTPUT_TRANSLATIONS` environment
278variables, so that systems are found in the Nix store and pre-compiled FASLs are
279loaded.
280
281## Adding a new Lisp {#lisp-adding-a-new-lisp}
282
283The function `wrapLisp` is used to wrap Common Lisp implementations. It adds the
284`pkgs`, `withPackages`, `withOverrides` and `buildASDFSystem` attributes to the
285derivation.
286
287`wrapLisp` takes these arguments:
288
289- `pkg`: the Lisp package
290- `faslExt`: Implementation-specific extension for FASL files
291- `program`: The name of executable file in `${pkg}/bin/` (Default: `pkg.pname`)
292- `flags`: A list of flags to always pass to `program` (Default: `[]`)
293- `asdf`: The ASDF version to use (Default: `pkgs.asdf_3_3`)
294- `packageOverrides`: Package overrides config (Default: `(self: super: {})`)
295
296This example wraps CLISP:
297
298```
299wrapLisp {
300 pkg = clisp;
301 faslExt = "fas";
302 flags = ["-E" "UTF8"];
303}
304```